This is a review of J. Paul Reed’s’ session at DevOps Connect: DevSecOps at RSAC 2017

Vacuums: Good for cleaning carpets.  Not great for teams who need to collaborate.

DevOps without collaboration cannot succeed.  I’ll guarantee that.  In fact, collaboration is fundamental to the cultural changes required for successful DevOps evolutions.  Gone are the days of development developing in a vacuum and then handing off to operations and testing, who were operating in their own vacuum. While no organization is ever perfect, we always need to strive for improvement.

 That being said, a couple of groups often forgotten in the successful DevOps transitions and evolutions are release engineers and security operators. Looking at how these often-forgotten functions intersect was the subject of a talk by J. Paul Reed (@jpaulreed) at the RSAC DevOps Connect 2017 conference. This isn’t the first time Paul, a release engineer, has discussed this at RSAC, and it probably isn’t the last. Each year, though, it has a different take because we are improving on the whole as an industry.

Paul laid out some similarities between release engineering and security operations when it comes being folded into a DevOps practice.  He reflected:

• We both look “a little off” to developers and the business
• We both shovel DevOps unicorn poop because now we are in DevOps
• We are often an afterthought in project plans/scoping/requirements
• When “it’ breaks, suddenly all eyes are on us
• We have well-deserved reputations for “no”
• We are both seeing greater automation in what we do

 The reality, though, is that we are in it together.  When DevOps works, there is no “us” vs. “them”.  It is just “us” -- and what we do intersects in very important ways.

To emphasize the year over year differences in the DevOps community, Paul outlined three things he have learned since his talk at RSAC a year ago:

 Software supply chains

First, the ways in which we consume software continue to be problematic. Our software is stitched together from more places and more languages than ever. This means your software may have more actors than you think.  Think about where you binaries originated, who created that container image, and what dependencies found their way into that build artifact.

He also addressed the move away from version numbers with the ever-growing adoption of continuous delivery - a move he believes ignores the human factor because version numbers allow us to communicate which version we are running.

To address these problems, he said we need to, “actively manage your software supply chain to make sure someone else doesn’t own it.”

Production Issues

Second, the ways in which we produce software continue to be problematic. Paul picked on Docker a bit, but pointed out that the issue isn’t with Docker.  The issue is with the speed at which it was adopted across the industry.  For many organizations he works with, Docker was integrated into the production process without sufficient or careful planning. Consequently, they end up introducing problems (e.g., where people use Docker images containing an entire operating system). We cannot simply adopt technology without fully assessing the impact on our production methods -- especially if short-term benefits overshadow long-term risks.

 DevOps House Cleaning

Third, we ignore heuristics that can help us. That is, are we ignoring practical signals that may solve our problems. Paul names a few: 

• Build processes are taking a lot of time.  This is often because builds are pulling artifacts in from somewhere else at build time, making them more difficult to audit.
• Build processes can’t be done on a train.  Paul suggests that if you have to be connected, it means you are relying on something external and it probably isn’t reproducible.
• You shipped artifacts as part of a build, but can’t find them later.  This means you can’t do forensics on them or track them down when immediate repairs/replacements are needed. 

For this third lesson, Paul offers a few words of wisdom he heard as an undergrad, “Think of it as house cleaning. Software bugs are like cockroaches.  They hide in the darkest, messiest parts of your code. To get rid of cockroaches, you wouldn’t hunt them down one-by-one. Instead, you clean up the house and get rid of their hiding places. Do the same in your code.”

Taking care of the items he noted below can help us clean the house.  The bug can go somewhere else.

For the upcoming year - before his next talk - Paul challenged us to:

1. Introduce our release engineers and security operators into our DevOps practices
2. After doing this, he suggests asking the two groups to research your software supply chain
3. Then he said we should ask, “How can we engage and help each other more?” 

To close, he reminded us all that we, alone, own our software supply chain.

You can view Paul’s entire talk here for more insight, wisdom, and examples.