Container technologies are rapidly transforming application development and deployment practices at many organizations. But they also present a minefield of security risks for the growing number of organizations using the technology to package and deploy modern, micro-services based application architectures.
Primary risks include those related to vulnerable container images, poor access controls and overly permissive privileges, exposed attack surfaces and misconfigured runtimes. Over the last couple of years, software supply chain attacks have increased container security risks and heightened the need for organizations to deploy controls for managing and mitigating container security risks.
"As containers have become fundamental to modern software development and deployment, it is vital to secure them," says Patrick Tiquet, vice president, security and architecture at Keeper Security. "Securing containers involves implementing a combination of best practices, tools and processes to protect containerized applications and their environments."
Here are five of them:.
Implement controls for containers accessing sensitive data
Users, applications, and devices should only be allowed to communicate and access resources that are required within their role. The best way to ensure that is to follow the principles of zero trust at the user, application and network layers, says Anthony Tam, manager, security engineering at Tigera. In addition, ensure that all sensitive data is encrypted at rest and in transit regardless of an internal or external destination. Ensure you use the latest industry standard crypto algorithms, he advocates.
"Data encryption is a mandatory requirement for customer PII data to meet compliance requirements such as GDPR," says Kong Yew Chan, director of product management, container security at Qualys. Key encryption should be secured with external key management, so that only authorized individuals can retrieve the encryption key to encrypt or decrypt the sensitive data.
"Use network segmentation to limit only authorized containers to access sensitive data volumes and use the principle of least privilege to limit the number of authorized users who have access to sensitive data," Chan says.
KC Berg, chief architect at API security testing firm StackHawk advocates that administrators treat all containers like they contain PII. For containers that need extra security controls implement a separate namespace with role-based access control (RBAC) and even more limited access.
Leverage the Init Containers feature in Kubernetes to unmount/unset configuration files and/or secrets once the primary process has loaded them, Berg says. "Run a side-car, or use a service, such as a reverse proxy in front of the container's exposed network service port that monitors for PII data patterns and generates alerts if unexpected access is detected."
Use runtime security tools to detect anomalous behaviors
Continuously monitor and analyze application behavior during runtime to identify and respond to threats in real time. Runtime tools can help reduce false positives by helping security teams detect attacks, risks and anomalous behaviors that are specific to an organization's infrastructure says Anthony Tam, manager security engineering at Tigera. "They can also help security teams respond to incidents in real time through actions like blocking network traffic to the affected container, quarantining the container, or other methods of preventing the spread of an attack," Tam says. Runtime security products also leverage automation for rapid investigation and response, and many can take containment actions like killing processes or isolating workloads with a single click, he says.
Kong Yew Chan, director of product management, container security at Qualys recommends that organizations use eBPF-based (extended Berkeley Packet Filter) technology to monitor and protect containers during runtime. With eBPF–based technology, runtime behaviors including file, process and networking events can be monitored and protected. "For example, any unintended or unauthorized file access to specific system files such as 'etc/hosts' can be considered malicious," Chan says. "Users want to prevent malicious file access with container runtime, so they can prevent potential threat risks and meet compliance requirements such as PCI-DSS."
Create eBPF-based rules to protect against known threats. For example, treat any processes that are created from in-memory data as fileless malware. Alternatively, administrators can create whitelist rules that allow specific known behaviors while blocking unknown ones "This approach ensures that all other events besides the well-known behaviors will not trigger potential attacks," he says.
Mitigate dependency related risks
Maintaining visibility into dependencies for container environments can be challenging. But there are certain measures organizations can take to mitigate risk from open source and other third-party code dependencies.
Kong Yew Chan, director of product management, container security at Qualys advocates that organizations use minimal base images or distroless images to build their container images. "The minimal base images will have limited dependencies, reducing the number of potential vulnerabilities," Chan says. "This approach will simplify base image dependency management and reduce the complexity of managing vulnerabilities of the base image." It's also a good idea to create a software bill of materials (SBOM) for container images. Open source tools such as Dependency Tracker, allow admins to maintain visibility into the software package dependencies within container images, he says.
In addition, use Kubernetes inventories to maintain visibility for container images in use. Running containers have specific images with specific dependencies. "Therefore, users can identify vulnerabilities within those running containers and prioritize fixes related to exploitable vulnerabilities with critical severity."
KC Berg, chief architect at API security testing firm StackHawk, lists other measures organizations can take to maintain visibility into dependencies in container environments. These include enabling container scanning in the Docker registry, running a container registry as a cache and scanning third party containers, scanning containers for vulnerable packages on every pull request and using tools such as Jira Security Center and Snyk to look for vulnerabilities. "Engineering teams should review vulnerabilities as part of a regular cadence," Berg says. "Once a good state has been reached, switch this to be Pull Request check-driven and treat vulnerable packages like any other software defect."
Have controls for mitigating software supply chain risks within container ecosystems
Vulnerabilities in the software supply chain pose a major security risk in container environments. But there are several measures organizations can take to mitigate these risks says Chan from Qualys. These include only using base images from trusted sources so images can be reproduced and verified and scanning images with Software Composition Analysis (SCA) and SBOM tools to identify known vulnerabilities and potential license. Chan also recommends that organizations scan images with secret detection tools to prevent any secrets from being leaked within the container images.
"Sign container images to ensure the authenticity of the image source," Chan recommends as another measure for mitigating risks related to the software supply chain. "Signed container images will prevent tampering with the immutable images, preventing malicious packages from being injected into the image." It's also a good idea to deploy container runtime security tools to detect configuration drifts and malicious threats in the runtime environments, he says.
"Deploy container runtime security tools to detect configuration drifts, malicious threats in the runtime environments," Chan notes. "This approach can detect and prevent threats in the runtime environment, thereby maintaining software supply chain integrity."
Implement controls for protecting against lateral movement
Have workload access policies that limit pod-to-pod communication, says Anthony Tam, manager security engineering at Tigera. He points to a recent survey that Tigera conducted of more than 1,200 users who are actively using Calico open-source networking and security tools in their container and Kubernetes environments, The survey showed 61% of Calico users using workload access policies to limit pod-to-pod communication, 41% using secure egress policies and 24% using microsegmentation to limit pod-to-pod communication. "An overwhelming 85% of users said they needed to achieve network segmentation and protect east-west traffic," he says. "IT leaders need enhanced security controls at the workload level to reduce the risk of lateral movement of threats.
KC Berg, chief architect at API security testing firm StackHawk lists four measures organizations can take to ensure isolation, prevent lateral movement and detect anomalous behavior in container environments.
The first is ensuring containers run with the least amount of privileges, file access and network exposure as possible by, for instance, running the primary process as a non-root user with limited access to container assets and mounting all configurations as read only. The second measure is to ensure containers run one primary process as far as possible.
The third measure he recommends is making sure that a container's network access is limited to only network resources required by the service. "Use your cloud providers IAM and SDN capabilities in combination with Kubernetes' RBAC to run containers with least privilege access to assets like databases and other services," Berg says. "Use service mesh and networking tools like istio or traefik to control K8s service-to-service connections."
Fourthly, Berg recommends that organizations monitor containers via node/host services that can expose telemetry to visibility tools like Prometheus and Grafana.
Organizations need to understand that just because an application runs in a container does not necessarily mean it is automatically secure, says Tiquet from Keeper Security. "The same discipline and best-practices for scanning and managing vulnerabilities in on-prem and cloud server instances can and should be applied to containers."
For the most part, the risks that containers pose — and the methods for addressing them — are not fundamentally different from risks associated with other application environments. Even so container security remains somewhat of a black box for the many organizations that are trying to harness the technology to speed up application deployment and efficiencies.
Containers are highly portable because they include everything that an application needs to run, such as dependences and configuration files. So, for development teams, containers offer a way to build an application once and deploy it across on-premises, multi-cloud, and virtualized environments. Container orchestration software such as Docker Swarm and Kubernetes have made it relatively easy for development team to centrally deploy and manage containerized apps.
In development environments, containers have made it easier for developers to automate the pipeline and more efficiently move applications from testing to production. Many organizations are taking advantage of container technologies to migrate internally developed applications to the cloud or to build cloud-native applications from scratch. Server consolidation and multi-cloud adoption have also contributed to container adoption.
The growing adoption of containers has also heightened the need for more security controls around their use. Increasingly, threat actors have begun targeting container environments with DDoS attacks, exploits targeting kernel and container orchestration technologies and other attacks, putting enterprise cloud applications and assets at risk.
Here's what your team needs to know about the state of container security so that you can release your software with full confidence that it is secure.
Container security risks
There's considerable concern particularly about attackers targeting private and public container registries to distribute malware and poisoned software updates to organizations using these repositories. Last year for example, a security vendor discovered five poisoned container images on Docker Hub that were designed to deploy cryptocurrency miners on the networks of organizations that downloaded these application containers.
Some of the malicious containers had misleading titles that suggested they were legitimate and were therefore downloaded thousands of times before being removed from the registry. More recently, an elevation-of-privilege vulnerability in the CRI-O runtime for Kubernetes raised similar supply chain attack concerns because the flaw gave adversaries a way to escape from a Kubernetes container and execute a variety of malicious actions on the host — including poisoning other containers.
Containers are nothing more than a stack of operating system and application packages bundled together that are vulnerable to security issues like any software environment, said Chris Romeo, CEO of Security Journey.
"The attacker's focus on the software supply chain is the cause of the bulk of threats against container security in recent years."
—Chris Romeo
Attackers can exploit these vulnerabilities to compromise a build pipeline, for example by poisoning an app container that is used by others, or by planting a poisoned container in a repository.
"Attackers focus on the build pipeline as a source of weakness. [They] are going after the pipeline, and the pipeline is where container images are built."
—Chris Romeo
With the build pipeline a target of attackers, software teams need visibility into compromises or tampering with heir pipelines, which can then poison the container images being produced.
Palo Alto Networks and others have identified multiple security issues tied to container environments. These include issues tied to vulnerabilities, malware, software signing, and secrets in container images, hosts, runtimes, registries, and orchestration technologies, all of which organizations need to be aware of — and actively securing them. These issues are a reality with container images just as they are with other software development practices.
Best practices like code integrity and maintaining a software bill of materials (SBOM) are critical in container environments. However, such practices can be highly challenging to implement given the sheer number of containers in typical application environments, as well as with how frequently containers are updated, Palo Alto Networks says.
Similarly, registries while critical to bringing a semblance of order for distributing containerized applications, make an attractive target for adversaries seeking to compromise entire environments that depend on them. To mitigate the threat, organizations need to consider implementing controls for continuously monitoring container registries, and for locking down the server and other infrastructure hosting the registry.
Organizations need to be similarly cognizant about and prepared for each of the other container security threats such as those tied to CRI-O runtime, container orchestration and host operating systems and misconfigured container images. In addition, because containers are constantly spun up and down it's difficult for organizations to apply the same security controls to containers as the do to other software.
Challenges to container security
Organizations face multiple challenges when it comes to implementing a container security program that addresses these threats. Chief among them is implementing a trusted image repository and enforcing its usage, said Romeo.
"Too many developers pull Docker images from any source as a starting point."
—Chris Romeo
Choose the right security tools
Choosing the right tool to run within the container to monitor for compromise and evaluate the current security posture is critical. So too is education and awareness for developers about the inherent risks in using containers, orchestration, and images for application development and deployment, Romeo said.
He recommends that development and security teams take the time to learn about tools that are currently available to help organizations secure container environments. They need to understand the capabilities of these tools and recognize the differences in the risks that container environments pose compared to typical app development and deployment pipelines.
"Answer the question are we scanning our containers for vulnerabilities? Are we monitoring running containers for compromise? Scope the depth of the problem, understand it, and then implement solutions."
—Chris Romeo
Lisa Azevedo, CEO of container security vendor Containn, said one big limitation with many current container security products and services is that they are reactive, and designed to detect after-the-fact security vulnerabilities. Many container security products allow organizations to scan for and detect known security issues, but do little to prevent them from happening in the first place. Most tools, at best, allow organizations to get a point-in-time assessment of security vulnerabilities in the container environment, she says.
Currently available container security tools generally are good at detecting existing vulnerabilities, providing a remediation report, and pushing the work of fixing the issues back to the development team. A growing number of tools have also started becoming available that harness machine learning, to predict vulnerabilities in software under development. But they don’t allow security teams an opportunity to stay ahead of the curve, because by the time organizations have a chance to remediate the detected issues, new issues likely have already surfaced, Azevedo said.
Push security further left
The key is to ensure container security by pushing it further left during the build process, Azevedo said. Organizations should be thinking about how to implement container security at scale from the beginning and finding ways to maintain control of container deployments and state. The focus should be on shrinking the attack surface while maintaining control of deployments and container state.
Such capabilities are critical because many organizations are on the cusp of moving away from manual tools to intelligent tools for container development and deployment, Azevedo notes. The goal is to be able to spin up containers that are standardized for specific environments and integrate security and compliance features like those required under different industry regulations and national data security and privacy mandates.
Romeo said tools are now available to help organizations tackle container security issues, but that there are challenges to adoption to be aware of.
"There are solid options both from commercial providers and open-source. Functionally, they provide the capabilities that the enterprise requires. The catch is getting implemented into the program, with developers taking action upon the results of the container security tools."
—Chris Romeo
Rethink your container security regimen
Build pipeline attacks are on the rise, and software supply chain security is front and center. With the potential for attackers to inject malware, tamper or compromise signing, the focus for security teams needs to shift beyond vulnerabilities alone. To ensure container security, you need to know if someone has changed or introduced malware in your container images — just like your code.
Article Link: https://www.reversinglabs.com/blog/state-of-container-security-release-confidently