In the previous article, we covered the build and test process and why it’s important to use automated scanning tools for security scanning and remediation. The build pipeline compiles the software and packages into an artifact. The artifact is then stored in a repository (called a registry) where it can be retrieved by the release pipeline during the release process. The release process prepares the different components, including the packaged software (or artifacts) to be deployed into an environment. This article will cover the contents of a release and features within a release pipeline that prepare a release for deployment into an environment.
Artifact registry
Artifacts are stored in a registry (separate system from the code repository) and accessible by DevOps tools like release pipelines and the IT systems that the application will be deployed on to. Registries should be managed as an IT system and provided to the Development and DevOps teams as a service. The IT systems that support the registry should be hardened using the corporate security policies. The registry should be private and only accessible within the company if it is not intended to be a public source for artifacts. Password protection and IP whitelisting are also advised to ensure that packages can only be retrieved by approved systems. Logging information needs to be sent to a Security Operations Center (SOC) for monitoring. Encryption of the packages at rest and in transit is also required.
Contents of a release
A release is created by the release pipeline (Azure DevOps, Jenkins, Team City) and uses the artifacts created by a build pipeline. The release pipeline is triggered by the build pipeline and it knows attributes like the latest software version that was created and the name and location of the artifacts. The release pipeline is highly configurable depending on when the release should be scheduled to deploy, what variables and secrets (passwords, certificates, and keys) should be used, which version of the compiled code needs to be deployed, and approval processes to protect environments from having a release replaced without approvals.
Releases are capable of being automatically deployed onto IT systems when a new artifact version is built. DevSecOps best practice encourages automated builds but advises manual approval instead of automated releases to most environments. However, it may be appropriate for release pipelines to automatically deploy into a development environment that is under the development team control. Environments controlled by different teams like Quality Assurance (QA), User Acceptability Testing (UAT), Production, and Disaster Recovery (DR) typically do not have automated release pipelines after every new artifact version is built.
Variables and secrets are how running applications can be adapted to the different environments (development, QA, UAT, Production and DR). Variables are created in the pipeline tools and can be applied during the release pipeline. DevSecOps recommends storing secrets in a separate “key” vault that can be connected to the pipeline. Separate variables and secrets allow the software artifacts to remain the same no matter which environment they are deployed into. When the software is deployed, it looks for variables and secrets that were configured in the build and release processes and uses them to properly set the system up. Variables and secrets are a big reason why releases can be deployed so quickly and multiple times per day.
Version control is mandatory for knowing which version of software is being deployed, having a rollback plan to recover if there is a bug or issue, and keeping track of features and additions to the application as developers work on the code. Every time a build creates an artifact, a version number is applied. The version number is used by the release pipeline so that it knows which artifact to retrieve and deploy. DevSecOps recommends using the semantic versioning system for all artifacts.
Release pipelines have approval features that control the software release. At a minimum, approvals should be setup for each environment, or where separation of duties is required between the development team and other groups in the organization. For example, the QA group should be the only team who can grant approval to release and deploy into the QA environment. This is because QA teams may still be working through their test plans on an older release, and they need to finish before testing the new release.
Build and release agents
The two types of servers (IT) for build and release activities are vendor-hosted and self-hosted. Vendor-hosted agents are managed by the vendor so upgrades and maintenance or taken care of. Also, a new agent is provided every time the build or release pipelines are run. This makes resource management easy for the company but may not be an option for unique build and deploy dependencies. While extremely secure, builds and releases performed by vendor-hosted servers are not in the company’s control.
Self-hosted agents are managed by the company and require the systems to be upgraded, maintained and any dependencies be installed before the agents can be used in build and release activities. Self-hosted agents work well when the DevOps platforms are internally hosted and not using any cloud-based servers. For example, self-hosted Jenkins pipelines use self-hosted servers and remain completely private and in the control of IT.
Next steps
There are many moving parts and components to the release process that need to be architected and designed with security in mind. The parts and components overlay with multiple vendors, all of the different environment owners, security, and IT. This requires the different members of a DevOps team, spread across all the different organizations, to work together and deliver changes to business systems safely and securely. The next article covers how a release is deployed and operated securely
Article Link: DevOps release process | AT&T Cybersecurity