One of the big trends sweeping the IT world is the embracing of DevOps. But, as attendees at the recent DevOps Enterprise Summit in San Francisco were reminded, a DevOps strategy ultimately is only as strong as the security wrapped around it.
Look at it this way: The point of DevOps is to get an organization's software development and operations teams on the same page to speed up the development process. But when you removing the obstacles that typically slow development teams and free them to innovate without fear, you also open yourself up to potential security holes. What if a buggy piece of software manages to get out? What if a development process causes a failure in some other business process, opening a vulnerability?
Such concerns are the reason that, for many organizations, DevOps has become Secure DevOps, or SecDevOps. The idea is to integrate secure development practices into your DevOps program, thereby ensuring that application security is a priority starting with the first lines of code, not just after deployment.
Four years ago, this became clear to Gnani Dathathreya, director of enterprise architecture for Capital One, once the financial giant headed down the DevOps path with a goal of adding speed and scale to its software development process. Dathathreya saw that more speed and more scale also equated to more failure, and the last thing he wanted to do was discourage failure.
"We need to embrace failure as part of our development," he told a packed room at the summit. "That is part of the culture change we are embracing."
In order to embrace that failure, Capital One set out to design "anti-fragility" into its applications. It wanted its developers to feel free to write code without worrying about whether it would cause the application or some other asset to fail. And it went out about this in a very interesting way: By embracing a chaos engineering automation solution called a "cloud detour."
The idea is to reflect the reality that cloud provider outages are inevitable. Cloud detour services create mini-storms on live systems, forcing a level of remediation that boosts the safety and resiliency of applications.
"Cloud detour addresses the need for a chaos engineer automation tool by providing failure-as-a-service for applications," said Sathiya Shunmugasundaram, lead software engineer for Capital One's technology operations. "You discover so many things."
Having such insurance policies in place can save a company a lot of heartache. Jennifer Brady, technology governance director for Capital One, said she once saw an extra comma cause a half-billion-dollar error, one that could have been avoided with some simple governance steps. Beyond secure procedures such as a cloud detour, Brady strongly recommends that organizations embracing DevOps also ensure they have the proper governance in place to ensure constant awareness of how development activities can introduce risk.
"When I joined Capital One, we were federating everything," said Brady. "That concerned me from a governance and risk assessment perspective. You want to make sure the things you're federating aren't putting the company at risk."
Such concerns helped spur Capital One to take another step by adopting a "clean room" approach to software development. That means that everything is under source control, and that every change is peer reviewed. Production changes only occur via code changes, and nobody gets unfettered access to production servers. Every code change goes through multiple levels of testing and scanning, and every piece of evidence that's captured is analyzed for discrepancies.
The results of this approach have been impressive. Capital One has seen the average number of code deployments increase from one a day to four, and the number of products that deploy multiple times per day has risen from 20 in 2016 to 300 this year.
In other words, it is possible to speed up development efforts without sacrificing security controls.
One astute summit attendee asked during a question and answer session with speakers if this amounted to enabling and allowing shadow IT practices to reign. The answers implied that everyone knows shadow IT is happening, that it's important to know why it exists, and to recognize that sometimes business priorities dictate that it be allowed.
Sometimes, it even needs to be encouraged.
"We have institutionalized it," said Jason Cox, director of systems engineering for The Walt Disney Company. "Part of the problem with shadow IT is that it assumes we need to control it. You want to power the edge. We want people to take risks."
At the end of the day, security is a critical part of software development processes, and with SecDevOps, organizations can enjoy both parts of the equation: Innovative, no-fear software development combined with a strong, well-thought-out security strategy.
It's the best of both worlds.