Secure Software Update Distribution

by Nathan Regola

Recent news has brought software supply chains to the forefront of cyber security discussions. While this post isn’t attempting to speculate on the cause of this most recent breach, there are some broad lessons to take away from any breach, namely that robust cybersecurity always requires a diligent approach. 

Companies push software updates to customers on a regular basis, but how do the customers know the updates are genuine?  The solutions to increase the assurance a customer has in the authenticity of a software update have been “known” for some time, and the theoretical solutions don’t require a moonshot program to implement. As an architect, I’m often called upon to build a solution from existing components. If we think about the toolbox available to support software update processes, we have an enormous set of tools to draw upon:

  1. Cryptography; specifically, hash algorithms, and public key cryptography
  2. Ready access to a multitude of cloud providers (this will come in handy later on)
  3. Email. For example, update cadence can be proactively communicated to customers (for example, a cryptographically signed email might be sent ahead of an out-of-cycle update to inform customers that they should expect an out of cycle update)
  4. Pre-arranged “wait” policy to ensure that the update isn’t being recalled 20 minutes after release because the process was hijacked.
  5. Well defined processes, with sophistication based on the importance or criticality of the product.
  6. Source code and artifact repositories.
  7. Infrastructure as code approaches to manage configuration and continuously redeploy public facing assets from source code and artifact repositories (this would lessen persistence if a public facing asset is compromised).

Let’s spell out a simple conceptual example to detect tampering of software in the distribution pipeline (e.g. update servers).  First, internal CI/CD pipelines generate software update, cryptographically sign the release (details dependent on the type of release—RPM, Windows EXE, etc. and where this happens in the process), and generate a hash of the signed release file.  Then, developer(s) independently generate a hash of signed release file.  This allows for cross-checking of the CI/CD pipeline.  A team member then loads the hash to a website with hashes available for customer validation.  The external check team updates the website integrity check to monitor for the new hash.  Finally, registered customers can readily access the integrity check site to determine if the organization’s website containing the hashes has been compromised.  If the site is secure, customers will see a green checkmark with “hash website verified.”  This process, while tedious, is not difficult to implement and can prevent customers from installing files that have been tampered with because there are numerous checks built in to the process.

If the process above is in place, we’ve made it much more challenging for an adversary to compromise our update distribution process.  Mitigations of attacks during the development process, such as a compromise of a developer account, a developer laptop, etc. will be topics of future blog posts.  Let’s step through what would have to happen to compromise an update in the update distribution process and distribute it broadly if the features above have been implemented:

  1. Adversary breaches update servers before release is signed
  2. Adversary breaches website containing hashes (hint: make this really hard; store it as Amazon Web Services static content or on another platform with significant cyber resources behind the service; create a completely separate set of accounts, and other infrastructure so that a single attack can’t result in the breach of both #1 and #2, like a common Active Directory account that can Single Sign On to both accounts).
  3. Adversary breaches integrity check system to update the hash so that a user sees “hash website verified” with a different hash than the distributor intended.

If you need help designing software deployment, software release, or hardening any of these processes, our team at Regola Consulting, Inc. has extensive experience with continuous development and continuous integration pipelines, including in regulated environments, such as the United States Federal Government.  Please reach out to us for a consultation.

Nathan Regola is the president of Regola Consulting, Inc.