As a security architect, one of the most significant and challenging threats I've encountered in recent years is the rise of attacks targeting the software supply chain. These attacks, which exploit vulnerabilities in the complex web of dependencies, libraries, and tools used to develop and deploy software, have the potential to cause widespread damage across multiple organizations simultaneously. They keep me up at night because they undermine many of our traditional security assumptions and require a fundamental rethinking of how we approach software security.

Real World Examples

The SolarWinds breach, discovered in late 2020, was a stark wake-up call. Attackers managed to inject malicious code into SolarWinds' Orion software, a widely used IT management tool. This compromised version was then unwittingly distributed by SolarWinds to thousands of customers, including Fortune 500 companies and government agencies. The attackers used this foothold to conduct espionage and further their access into the networks of high-value targets.

I experienced the fallout from this breach firsthand while working with a large technology client. The company used Orion for network monitoring and had to assume that they were compromised. The incident response effort was massive, involving a complete audit of their network, the rebuilding of multiple systems, and months of heightened monitoring. It was a painful reminder that trust in our software vendors can be exploited, and that a single compromised component can unravel an entire security strategy.

The SolarWinds attack was not an isolated incident. The Codecov breach, disclosed in 2021, showed similar tactics. Attackers gained access to Codecov's Bash Uploader script, used by thousands of customers in their CI/CD pipelines, and modified it to export sensitive environment variables. This gave them access to a trove of tokens, keys, and credentials used by Codecov's customers.

These incidents highlight several key challenges in securing the software supply chain:

1. Complexity and Opacity

Modern software is built using a vast array of components, often pulled from open-source repositories or third-party vendors. This complex web of dependencies is difficult to map and monitor. It's often unclear what components are being used, what their security posture is, and what risks they introduce.

2. Trust and Verification 

We implicitly trust software vendors and tools, often giving them high levels of access and privileges, but as the SolarWinds and Codecov breaches showed, this trust can be exploited. We need better ways to verify the integrity and security of the software we use.

3. Security vs. Speed

The pressure to deliver software quickly often trumps security considerations. Security checks, code reviews, and testing are seen as impediments to agility. This trade-off is false and dangerous. As the saying goes, "slow is smooth, and smooth is fast." By integrating security into the development process, we can catch issues early and avoid costly incidents down the line.

So, what can we do to start addressing these challenges?

Here are some emerging best practices:

1. Implement a Software Bill of Materials (SBOM)

An SBOM is a comprehensive inventory of all components in a software product, including open-source dependencies. By maintaining and scrutinizing SBOMs, we can better understand our software supply chain and react quickly when a component is found to be vulnerable. Tools like Syft and Anchore can help automate SBOM generation and monitoring.

2. Adopt DevSecOps Practices

DevSecOps is about integrating security into every phase of the software development lifecycle, from design and coding to testing and deployment. This includes practices like security training for developers, static and dynamic code analysis, automated security testing, and continuous monitoring. By making security a shared responsibility and an integral part of the development process, we can catch issues early and build more secure software from the ground up.

3. Implement Least Privilege and Segregation

The principle of least privilege dictates that each component or process should have only the permissions it needs to function. Segregating build environments, using ephemeral credentials, and tightly controlling access to sensitive resources can help limit the blast radius of a breach. Techniques like multi-factor authentication, role-based access control, and network segmentation are crucial.

4. Establish a Software Supply Chain Risk Management Program

Managing risk in the software supply chain requires a proactive and holistic approach. This includes vetting and continuously monitoring suppliers, establishing security requirements in contracts, and having an incident response plan tailored to supply chain scenarios. Organizations like the Linux Foundation and the Cloud Native Computing Foundation (CNCF) offer frameworks and best practices for software supply chain risk management.

5. Leverage Blockchain and Distributed Ledger Technology

Emerging technologies like blockchain offer promising solutions for supply chain security. By creating an immutable, decentralized record of every component and  change in the software development lifecycle, blockchain can provide end-to-end transparency and traceability. Projects like Hyperledger and Tekton Chains are exploring the use of blockchain for secure software supply chains.

Conclusion

Securing the software supply chain is a complex and ever-evolving challenge. There is no silver bullet solution, and the journey will undoubtedly involve trial and error. But by learning from recent incidents, adopting a proactive and integrated approach to security, and leveraging emerging best practices and technologies, we can start to build more resilient and trustworthy software.

As security architects, we have a key role to play in driving this change. We need to educate our organizations about the risks, advocate for secure practices, and architect solutions that prioritize software supply chain security. It's a daunting task, but also an exciting opportunity to fundamentally reshape how we build and deploy software.

In the words of the famous computer scientist Edsger W. Dijkstra, "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." Similarly, the question of whether our software is secure is less important than the question of how we can make it secure. By focusing on the latter, by continually learning and adapting, we can navigate the uncharted waters of software supply chain security and emerge stronger and more resilient.

Useful Resources:

• NIST's Guidance on Software Supply Chain Security: https://www.nist.gov/itl/executive-order-improving-nations-cybersecurity/software-supply-chain-security-guidance
• The Linux Foundation's Software Supply Chain Best Practices: https://www.linuxfoundation.org/en/blog/software-supply-chain-best-practices/
• Cloud Native Computing Foundation's Software Supply Chain Special Interest Group: https://github.com/cncf/sig-security/tree/main/supply-chain-security
• Syft - A tool for SBOM generation: https://github.com/anchore/syft
• Hyperledger - Blockchain solutions for supply chain: https://www.hyperledger.org/use/supply-chain