Don't neutralize CI/CD business gains by failing to account for risk. Here are best practices to ensure your software development pipeline is secure.
As the adoption of continuous integration/continuous delivery (CI/CD) approaches to software development keeps snowballing, the benefits are stacking up for many organizations. CI/CD helps organizations deploy software more quickly and more often, making code changes faster in response to changing business conditions.
But the automation and complexity of CI/CD pipelines and processes can introduce significant security risks into the development process if organizations don't plan carefully. Not only do organizations need to ensure that security checks are built into the fast-paced workflow of CI/CD processes, but the tools and integrations of the CI/CD pipeline itself must be protected.
With attackers increasingly turning their eye to development environments, organizations must take measures to prevent attackers from infiltrating CI/CD infrastructure to keep them from no only breaching the organization but also introducing dangerous flaws into their applications.
Here are eight key practices to help shore up CI/CD security on all of these fronts.
1. Harden your CI/CD infrastructure
Step one to great CI/CD security is to ensure that the infrastructure that backs the entire pipeline is properly configured, updated and isolated to prevent lateral movement.
Liav Caspi, CTO and co-founder of Legit Security, said that means hardening your CI/CD infrastructure.
"For example, if you're running a Jenkins server, be sure to harden the server itself, its configuration, its authentication and so on so that nobody can hack it, or move laterally if they do hack it."
—Liav Caspi
Viktor Gazdag, managing security consultant for NCC Group, explains that "basic things like password policy, network segmentation, least privilege principal for identities, and updating will help prevent a lot of attacks."
"All the essential blocking and tackling practices of establishing security hygiene are at play here."
—Viktor Gazdag
Brian Price, founder of Interplay, a security-focused managed service provider, agreed that software updates on CI/CD systems should be an especially big priority.
"Obsessive software updates and patching on every device that touches the development chain are critical. Turning on automatic updates isn't enough; a system that manages, reports on, and can roll back patches is vital."
—Brian Price
2. Establish developer guidelines and guardrails
Another fundamental starting point for establishing CI/CD security is to set up workable security guidelines and guardrails for developers in how they code and operate within the environment on a day-to-day basis, says René Milzarek, managing director for Crashtest Security.
"For a less security-aware company, I would suggest starting with code reviews and the creation of developer guidelines addressing cyber security. Define how you handle static secrets, encryption algorithms, and updates of third-party components. Afterward, I would go into automating some of these steps with specific tools but keep in mind there is no clear answer to this as it depends on the organization and application at hand. You need to make this decision based on the risks at hand."
—René Milzarek
Sometimes establishing guardrails starts simply with putting controls in place to ensure developers play by the same rules in the CI/CD environment as other privileged users would in some other sensitive platform. One example: Enforcement of password hygiene and two-factor authentication.
"Developers may know that they need strong, unique passwords across all toolsets, but they may not actually practice this. A good password manager can dramatically improve password hygiene. Two-factor authentication is also essential everywhere it is available and a password manager can act as an authenticator in most cases."
—Brian Price
3. Make permissions and secrets a priority
Speaking of passwords, getting control over secrets management and identity management stands as one of the highest value practices that organizations can engage in to take their CI/CD security to the next level. This includes managing permissions to development systems and sensitive data stores, as well as securely storing and managing everything from passwords to tokens, encryption keys to APIs.
As Gazdag explains, even though the industry has for years been awash with horror stories about hardcoded credentials and secrets leaked from development environments, weak to non-existent secrets management is still rampant.
"When we look at lack of CI/CD best practices, my main concerns are secret management and identity management. Identity management can be very complex and often over-permissive permissions are assigned. The scope and storage of the secrets are also often overlooked and misconfigured, allowing everyone or a wider audience to access them."
—Viktor Gazdag
Ultimately, the goal should be to enforce the rule of least privilege — both for developers and other people who touch these systems, as well as machines and automated processes that integrate into the stack.
Stavros Zavrakas, managing director of Orthogonality, a software development firm, said it was about ensuring that not everyone has access to the network and pipelines, "so that it remains well in control."
"Give access to only those people who are needed for the job. It is important to keep a regular check on the inventory of systems and remove any component who is no longer needed for the task."
—Stavros Zavrakas
4. Eliminate the single power of authority
As organizations utilize a kaleidoscope of tooling to create an automated CI/CD pipeline, some of the biggest risks can crop up when automated actions are unqualifying trusted to trigger code commits or deployments.
Zavrakas said that it's crucial for security to eliminate single power of authority in the pipeline. "Establish control mechanisms to ensure that no one authority has power to ship sensitive code through the pipeline without verification or validation from another source externally," he says.
On the developer end, organizations can minimize the attack surface by reducing permissions around source code, says Nir Valtman, founder of Arnica, a software supply chain security firm.
"If the company culture is to provide access to push code for all developers, then apply branch protection policies to require pull request reviews by the right owners, and review the CI/CD permissions and triggers."
—Nir Valtman
5. Carefully plan when and how security tests are done
Some of the most important practices in CI/CD have to do with security testing. As organizations plan their CI/CD workflows, they must carefully plan when and how security tests are done.
Keatron Evans, principal security researcher for Infosec Institute, said that all too often organizations rely too heavily on static methods like code audits and code reviews. "It completely overlooks the fact that many of the most critical vulnerabilities only exist in runtime when outside libraries and other code infrastructure is pulled," Evans says.
Evans said he believes one of the most important CI/CD best practices is to weave in regular dynamic testing into the production of software. "This has the most immediate impact and can identify the most serious vulnerabilities."
Larry Maccherone, a longtime DevSecOps practitioner and DevSecOps transformation architect for Contrast Security, goes one step further, stating that organizations should be thinking big picture with security testing as a part of high-coverage automated functional testing — a.k.a. end-to-end, integration, or smoke testing — rather than focusing on unit tests.
"Having a high-coverage automated functional suite is my hidden best practice for security aware organizations. It’s hidden because most security people think that focusing on a test suite that targets 'quality' as opposed to directly targeting security is out of their swim lane but it’s not.”
—Larry Maccherone
Maccherone said he also believes that testing should be done prior to the pull-request merge decision, which requires a container-based CI system that stands up the resources needed in the pipeline to run.
"This is not trivial engineering work but is worth it. Once you do this, you run all of your tests in as parallel a fashion as possible so it finishes fast enough to inform a pull-request merge decision."
—Larry Maccherone
Milzarek says getting testing right is a matter of pulling together a combination of tools specific to purpose, including a balance of software composition analysis, static code analysis, and dynamic code analysis. "You need to understand that the different tools have different strengths in addressing threats and that you need to build a security culture."
"If your developers do not share the awareness, they will find ways around automatic CI/CD checks or get demotivated by breaking builds."
—René Milzarek
6. Verify security of build software
Caspi says that when development teams utilize open source components and platforms to write CI/CD automations, that they don't blindly trust the content of what they're using to build it. This is a microcosm of the broader software supply chain security issue, but narrowly focused on the software that underpins the CI/CD pipeline.
"The factory line that creates software is itself software. And it can be targeted. For example, when you look at SolarWinds, the build server was hacked in SolarWinds to produce the malicious software. So, verify the build components you're using, the plugins, the GitHub Actions you're relying on. Make sure you're in control of the other components you're using and make sure you use verified components and not something very shady that you don't know what's running in it."
—Liav Caspi
7. Make the CI/CD pipeline more visible
Ideally organizations should be monitoring and tracking activity across the CI/CD pipeline to detect potential malicious behavior, unauthorized access to sensitive systems and code repositories, and potential risks latent in the CI/CD pipeline.
The most important priority organizations can engage in is to make sure tolook at the whole pipeline, and the pipeline flows themselves. Todd Weber, operating partner at Ten Eleven Ventures and a longtime security-focused CTO before that, said that several existing cybersecurity technologies look at one or two pieces, like the repositories or the code, but not all the components together.
"My biggest concern is that the ecosystem we have established for this process has become so complex, with so many third-party dependencies, that no single party can monitor or even enumerate everything, particularly now that we see such extensions to the CI/CD pipeline with categories like Data-as-a-Service and AI-as-a-Service."
—Todd Weber
As a VC, Weber says this is where the biggest opportunity is for future innovators, but at the moment organizations must do some hard work to get full visibility into how the pipeline works and which third-parties impact the security posture of that software factory.
"Keep digging deeper. There are a lot more third parties involved than you think. Identifying all the components of your pipeline seems deceptively simple — it actually takes a lot of work."
—Todd Weber
8. Use threat modeling
Threat modeling can also be a valuable next step for organizations that have already covered many of the basics of secrets management and security hygiene in CI/CD environments. Not only should organizations be consider developing threat models for the software components of the software product they're producing, but also for critical systems in the build environment, explains Gazdag.
"Continuous threat modeling will show connection boundaries, connecting components and data flows. This will allow an organization to see the big picture with gaps and blind spots, so they can put in place the correct preventative and security controls. For example, developers can see where a credential is stored, used, where it came from, how it’s stored and who can access it."
—Viktor Gazdag
Article Link: 8 CI/CD security best practices: Protect your software pipeline