Nobody likes having vulnerabilities in production and potentially being exploited by attackers – but let’s be honest we all have them. Just look at the number of SSL related vulnerabilities announced in the last few months that were in production for years. If we can identify those issues as early as possible and fix them as fast as possible then the window of opportunity for an attacker will be as small as possible. That’s a big win for security! But how do we achieve this without impacting our delivery teams?
At Betfair we like a challenge, so when our development teams started working on improving our Continuous Delivery approach our Security team saw the opportunity to also embrace a more automated approach for the following reasons:
- reduced time to market for new product features
- reduced time to apply routine patches
- reduced effort required to apply patches
- clear lineage and audit trail of the security testing and state for every build
Additionally, our developers now get feedback as their build moves along the pipeline, for every build, with metrics and trending ensuring complete transparency of the tests performed by security and their results. If a security check kills a build, or breaks a pipeline it won’t be a surprise to the development teams.
Risk avoidance is achieved by fixing vulnerabilities before problematic code reaches production. Risk reduction is achieved by reducing the amount of time uncaught vulnerabilities are exploitable in production. Residual risk is tracked and visible to the product owner empowering them to make sensible business decisions.
At Betfair, we created a process combining application security and infrastructure security to ensure that every asset we deploy is of an acceptable secure state. This is achieved by a combination of:
- Creating hardening guidelines for teams to implement when building configuration files; stuff like removing unnecessary services, changing default ports, changing default credentials, etc.
- Performing a vulnerability scan of the OS gold build with the aim to create a baseline. Any identified vulnerabilities must be remediated before moving to the next stage of the process.
- The OS baseline image is then marked as immutable and made available to teams for inclusion in their application stacks.
- Vulnerability scans take place again as part of the QA process and check for any newly introduced vulnerabilities that may have crept in as part of the provisioning stage. By running this second scan in parallel to QA we do not impact the time the change takes to traverse the pipeline to production.
In parallel to the infrastructure security scan, we have developed the “Application Security Risk Calculator (TM)*”. It ensures the risk is within the appetite set by product owner, taking into consideration the following information:
- Static code analysis results.
- Type of information processed and the type of component.
- Inclusion of standard security controls/libraries (like input white-listing).
- Team performance metrics (e.g. previous time to fix metrics, number of low impact vulnerabilities).
- Development process attributes (e.g. user story threat assessment, pair programming, peer review, security unit tests).
- The deployable (server and application) follows the standard process of CD through to production.
The last step is ongoing vulnerability scans occurring in production to constantly monitor and verify the security status of our production assets. This is the final feedback loop assuring the entire estate’s security health is known. Any positive test results here trigger changes which travel along the entire pipeline again before being delivered into production.
The approach of assuring our OS builds and allowing the inherent and dynamic risks associated with a code change to interact with a pipeline has allowed us to provide a real risk mitigation approach without impacting delivery.
We presented this solution to the Pipeline conference and gathered some amazing feedback to continue improving our processes, so please reach out to us for any further information. We’ll post the video of this presentation here soon.
/Francois is a Senior Manager in Betfair’s Security team.
*The “AppSecRiskCalc” will soon be available as an opensource component – we’ll add details here when it is launched. We are actively trying to rename it to something more sexy 🙂