Striking the Proper Balance When Dealing with the Inevitable Move to DevOps & Meeting Regulatory and Compliance Requirements

By Dan Cornell 

In addition to my RSAC-TV presentation on Effective Application Security Testing for DevOps, I had the opportunity to run a Peer2Peer session at RSAC 2017 on Implementing SecDevOps in Regulated Industries. I proposed this session topic because, although there is a lot of public information about how organizations are managing their transition to DevOps, most of it relates to organizations that aren’t under heavy regulatory burdens. The information that has been published is fantastic and provides excellent ideas for aspiration. However, in our consulting practice we work far more frequently with organizations that are moving toward DevOps, but whose journeys are constrained by compliance and regulatory requirements. Many of them are concerned that the move to DevOps will erode the progress they have already made in their application security programs, and others view the move to DevOps as an opportunity to “start things off right” as new processes are crafted and new tools are adopted. We’ve had success in helping those organizations shape their transitions, and I thought the Peer2Peer session would be a great opportunity to have a conversation with more organizations dealing with those challenges. 

Attendees

Attendees hailed from a variety of industries – financial services, critical infrastructure, insurance, healthcare, and retail, to name a few. We also had visitors from a regulating body I will leave unnamed. They were classy and identified themselves up front, expressing their desire to use the session as a learning experience. 

Why the Move to DevOps?

The first topics we discussed were the impetus for organizations moving to DevOps. Here we saw a spectrum of reasons such as deficiencies in legacy IT practices, IT not perceived as being responsive and unable to deal with how code change velocity had increased. Application teams were forced to focus on the automation required to meet the increased release tempo necessary for the business. Other teams were being driven by the need to decrease costs. Organizations were adopting cloud and automation for cost-control reasons, and DevOps was included with that transition. Finally, one or two organization’s DevOps initiatives were driven by dubious management directives – a desire to be the “cool guys” doing the “new thing.” Regardless, all attendees were faced with a situation where DevOps was a mandate for at least some of their systems and they were grappling with what needed to happen to support that transition. 

First Systems to Make the Move

Moving on, we discussed the initial teams and systems making the transition to DevOps. This group contained newly developed systems adopting DevOps practices - mobile application supporting marketing functions and new functions where microservices architectures were being deployed. One organization was faced with evolving mainframe development and operations practices supporting DevOps characteristics (and I certainly don’t envy that team…). 

Tools in Use and Challenges with Tooling

We had a brief discussion about the toolsets organizations were adopting. There weren’t a lot of surprises here – standard stack of Git, Jenkins, and various cloud providers. One organization was incorporating hardware security modules (HSMs), which was interesting. But for the most part, the tools used were to be expected. The interesting thing was looking at the challenges these organizations had in adapting many standard security tools to meet their needs. There was a strong desire to automate security checks in the DevOps continuous integration/continuous delivery (CI/CD) pipelines, but often the tools took too long to run and required asynchronous handling of results. In addition, insufficient APIs hindered automation attempts. Finally, there were a lot of licensing challenges where traditional licensing models were not adapting well to the way these tools needed to be deployed in DevOps environments. 

One interesting observation was that embedding security and compliance checks into DevOps CI/CD pipelines created potential separation of duties concerns. If audit findings go to the line of business and their development team, that could allow them to cut corners. We’ll see some more about how organizations aspired to address this shortly. 

Serious Challenges

It shouldn’t be a surprise, but all attendees had identified challenges associated with moving to DevOps practices in regulated environments. Chief among these was the technology and process framework that needed to be crafted from the outset to address compliance issues or else trouble ensued. If you have a well-defined technology architecture, you can map technical controls to compliance concerns. The trick is keeping up with ever-evolving technology architectures. One specific example was data handling and EU requirements, but others were evident as well. Also, many organizations with an entrenched culture of IT “shared services” were reeling with the demands of DevOps, creating palpable tension between a desire for the legacy IT organization to evolve the existing shared services approach and the desire for DevOps teams to forge their own path. An excellent example of an implementation challenge that organizations faced was answering the question of “where does incident response fit?”. Many of the DevOps teams’ impulse upon discovery of a breach was “delete the instance and reprovision” However that makes it impossible to collect forensic data. This was a concrete example of how getting the framework right up front is critical - these are the sorts of decisions that need to be made before an incident occurs if you want to see predictable and correct outcomes. 

Value Derived

At the end of the day, attendees gained value from the adoption of DevOps within their organizations. To begin, vulnerabilities are getting fixed faster. Tighter integration with security testing tooling in CI/CD pipelines led to serious issues identified early and addressed quickly without much fuss. This was progress in many environments. Also, DevOps provided an opportunity to apply good security patterns to different lines of business. A common one was to have an embedded security contact/representative/advocate to be the “voice of security” within every DevOps team. There were challenges involved in putting these individuals in a position to stop a build and make them feel empowered to do so. We encouraged them to notify security/compliance teams, so they can be the bad guy. Generally, attendees felt that compliance could be a good stick to use, but that it had to be used sparingly. Substituting “security” for “compliance” is probably good advice for any security team struggling to come to grips with the DevOps revolution. 

Conclusions

I thoroughly enjoyed the Peer2Peer session and hopefully the attendees felt the same. My main takeaway from the session was that regulated environments feel the same challenges that non-regulated environments have when dealing with DevOps transitions, but they feel certain pains a bit more acutely while simultaneously having a slightly larger stick they can use to beat back against the tide of changes.