While working with Public Key Infrastructure (PKI) systems, I learned that managing signing keys is like walking a tightrope. Regular rotation is essential for security, but each rotation risks breaking compatibility with legacy systems—a complex challenge to resolve. After plenty of trial and error, I found an approach that works: rooting signing keys to a Certificate Authority (CA). It’s not a silver bullet, but it’s reliable and flexible, and it prevents last-minute scrambles during key rotations.
The Key Rotation Conundrum: Why This Is So Hard in Practice
On paper, rotating signing keys sounds simple, but in practice, it’s anything but. During my time on Uber Advanced Technology Group’s (ATG) Autonomous Vehicle Security (AVSec) team (2018-2021), we managed signing keys for software artifacts deployed to vehicles—maps, software, manifests, models, and constraint files. We weren’t just signing code; we were ensuring the integrity of every system and artifact that kept vehicles operational.
Initially, we used a single signing key across all artifacts. It seemed efficient but created a massive risk: if that key were ever compromised, all artifacts would be vulnerable. So, we split different artifact types into separate keys. Generating these keys was easy and deploying them to the vehicles’ base operating systems (OS) wasn’t complex either. The real challenge came with revocation.
Not all vehicles ran the latest OS at any given time. Some remained on older versions for weeks, meaning they’d still trust old keys while missing updated ones. If a key were compromised, coordinating a fleet-wide OS update to roll out new keys and revoke old ones required days of planning to ensure the fleet stayed in sync. Through it all, security couldn’t block vehicles from being on the road, whether for training or actual rides. Each rotation became a balancing act to keep the fleet secure while ensuring operations continued smoothly.
Rooting Keys to a CA: A Foundation for Seamless Verification
After dealing with key distribution, we needed a way to rotate keys without risking downtime. Rooting all signing keys to a CA was the answer, and here’s why it worked:
- The CA as a Constant: Rooting keys to a CA simplified thing. We issued an X.509 certificate for each signing key and linked it back to the CA. Legacy systems only needed to validate the CA itself, not each individual key, making life easier.
- Certificate-Based Verification: Verification libraries could now extract the certificate, check it against the CA, and verify the data’s signature. We moved away from a “one key for all” model, and older systems didn’t need to recognize each new key.
- Flexible Key Rotation: Rooting to a CA allowed us to rotate signing keys without breaking legacy systems. As long as the CA remained constant, every associated key could be verified, keeping our PKI resilient.
Certificate Expiration: Enforcing Key Rotation by Design
Another benefit of this approach? Certificate expiration. Certificates naturally come with expiration dates, which means signatures with expired certificates won’t verify. At Uber ATG, this helped us cut compliance checks by 30% since expired certificates naturally pushed systems to update. Compliance wasn’t a burden anymore; it was built into the PKI.
I’d love to say everything went seamlessly, but we hit bumps, especially with the old-versus-new key dilemma. Some deployments didn’t go as planned, and we had to scramble to fix compatibility issues. Thankfully, these glitches rarely affected the fleet directly, as vehicles tended to run one or two OS versions behind our latest deployments. Over time, we ironed out the process, but it took some trial and error.
How to Model Your Verification Libraries
Here’s what my experience has taught me about building verification libraries that work in the real world:
- Start Simple, Scale Smart: Don’t try to solve every scenario immediately. Get the basics right first.
- Focus on Key Management: Get signing key rotation working smoothly before tackling complex CA rotation. Trying to do both at once was like juggling while learning to ride a bike.
Here’s a simple cheat sheet to evaluate your needs:
| Key Rotation Approach | Pros | Cons |
| Pre-Configured CA | Great for stable, smaller systems | Breaks down if a hard rotation of the signing key or CA is required |
| Remote Attestation | Perfect for dynamic environments | Failed us during network issues |
| Remote Attestation with Fallback to Pre-Configured CA | Ensured continuous operations even if attestation failed | Higher complexity, requires robust fallback logic |
Lessons Learned: Avoiding Cryptographic Sprawl
One major takeaway from Uber ATG was the risk of cryptographic sprawl. As we introduced more keys and CA structures, we realized how quickly things could spiral out of control. It’s like piling up cables until you’re left with a tangled mess. If you’re adding cryptographic elements to your system, plan carefully. Without a clear rotation strategy, compliance, updates, and verification can quickly turn into a nightmare.
Building a Future-Ready PKI
No PKI setup is flawless, but this strategy kept us secure and adaptable. For your own PKI needs, assess your environment and plan accordingly—key rotation can be smooth with the right foundation.