Adrift in the Cloud: A Forensic Dive into Container Drift

Adding container drift detection to Google’s Container Explorer

Containers drifting in the ocean

I’ve been meaning to blog about container forensics for quite some time now. As you may know, working in IR isn’t just about dealing with incidents; it’s also about tons of research and constant, unending upskilling — does it ever end!? So, I make the most of what little downtime I get, and it’s super difficult to convince myself to build tools or blog about stuff during R&R. Over the past few months, when I wasn’t busy with IR or forensics or automating something, I was prepping for the GIAC GRID exam (nailed it BTW). After taking some time off, I started thinking about finally sitting down to write about detection and forensics again.

Introduction

In this discussion, I’ll be diving into container drift detection, specifically, analyzing container drift from a forensics perspective, with a focus on OverlayFS.

Tools like Docker Forensics Toolkit, Kube Forensics (live-response), Dissects Docker Plugin, and Docker & Container Explorer by Google do a great job making container forensics doable. I absolutely love Container Explorer, as it can handle Docker and containerd-managed systems. My only gripe is that there’s no way to identify container drift quickly & easily.

Container Drift

So, what exactly is Container Drift? In layman terms, it means that there’s changes to the containers filesystem, and the container has drifted from its known original state, undermining the principle of immutability. Immutability is a core principle of cloud-native design. Any unintended or unauthorized changes to its filesystem should be investigated, and this is often the first step. If a container has been compromised or a host running a container is compromised, performing container forensics will help you scope the incident and determine the root cause & blast radius of the incident.

Container Filesystem

In containerized environments, filesystems are designed with a layered architecture that efficiently manages files for each container instance. The base layer holds the read-only image shared across containers, and a writable layer manages the changes unique to a particular container.
I’ve not seen anything other than OverlayFS — a union file system- in the incidents I have responded to. I’d recommend reading the Docker storage drivers manual & this article on Docker Forensics by Didactic Security. My container forensic workflow usually includes checking the container config, image analysis, looking at the mounted volumes, scanning the container FS for secrets with something like trufflehog, and checking the diff/upper directories to identify new files created while the container was running, to name a few. I’d also review the container logs to identify any suspicious activities.

Forensics — Container Drift

Suppose you are responding to an incident involving a Docker-managed container and can perform live response actions. In that case, I’d recommend executing docker diff, or nerdctl diff (containerd) to see container drift.

docker diff <container ID>
nerdctl diff <container ID>

Docker

I could discuss how to manually find the writable layer of a container filesystem on the disk or show you. I wrote a Python script to identify the writable layer and its files. Let’s examine the tool’s output below.

Identifying the upper (writable layer) directory

For a given container with container ID ending with e139, we can see the associated mount-id file under:

/var/lib/docker/image/overlay2/layerdb/mounts/<container ID …e139>/mount-id

Notice the container ID highlighted in yellow? Read the mount ID from the aforementioned mount-id file — value is highlighted in blue in the screenshot. The diff layer of the container can then be found by constructing the path with the mount ID value we just obtained (notice the blue highlight in the above screenshot):

/var/lib/docker/overlay2/<mount ID>/diff

Looking at the tree view of the diff layer, we can see files that were created and deleted. Deleted files are created as whiteout files in the upper directory. File /etc/host.conf was deleted from within the container, and the same is highlighted. Inspecting the host.conf file shows it’s a zero-size character device with a 0/0 device number.

tree view showing added & deleted (whiteout) files

Comparing the output of the Python script with docker diff for the container shows the same output as noted from below. A — added, D — deleted, & C — changed.

Script vs. docker diff

Containerd

In containerd, each layer of a container’s image is a snapshot with OverlayFS being used to manage it. Containerd uses a key-value store (Bolt DB) located in the below path to store the metadata associated with containers, namespaces, labels etc.

/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

I found a CLI Bolt DB browser here, so I will use that to find where the upper directory for a given container can be found on disk. Let’s open the meta.db file with boltbrowser.

For a given container with a container ID ending with ***898f42, note down the value of the key name in the below path under the associated namespace. Refer to the attached screenshot as well.

snapshots -> overlayfs -> <container ID>
Finding the name key

The value of the key name is shown below. This value maps to the snapshot key in another bolt DB file — metadata.db.

k8s.io/175/7ee0c6c2ac27dcf810737b6426611e42bd91d963edcad5aa3fd12db20d898f42

To find the path on disk a containerd snapshot key maps to, let’s use boltbrowser to open metadata.db file located in the below path.

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db

With metadata.db open, let’s find the aforementioned name keys’ value (k8s.io/175/….) under snapshots. Notice the key id and its value z highlighted in red in the screenshot below? ASCII value of z is 122 in decimal.

Finding the upper directory

The corresponding directory under snapshots/122/fs shows the files in the upper directory of the container in question. And of course, we can see an obviously suspiciously named executable script l33t.sh under /opt/pinot.

Listing the upper directory of the container we were investigating

Wasn’t that fun and time well spent?

Adding drift detection to Google’s Container Explorer tool

Knowing how to find container drift manually is important, but let’s automate it. I’m looking forward to the Forensic Container Checkpointing feature (live-response) becoming GA in Kubernetes. I wanted to create something that emulates the filesystem changes output generated when checkpointing a container using this feature.

I could write another tool from scratch or add this nifty little feature to an existing one. Considering the fact that Container Explorer supports both Docker & containerd, I decided to add the feature to it. Using the debug switch on Container Explorer while mounting the container filesystems shows you the lower & upper directories. But this involves mounting the container’s filesystem. There are scenarios where mounting the container’s entire filesystem can be useful, but as incident responders, we are always looking for those quick wins, and analyzing a container’s entire FS is something I’m going to only do if absolutely necessary. Not to mention that sometimes there will be more than one container on a host, & don’t even get me started on EKS/GKE nodes! You can find my fork of Container Explorer here in my GitHub repo.

My PR has been merged into the project.

Demo

For a quick show-and-tell, I created an EKS cluster and deployed a vulnerable Pinot instance. I did some Groovy RCE, took a snapshot of the node, created a volume, and attached it to my forensic EC2. I mounted the volume as read-only to /mnt/case. I also mounted the volume from another compromised EC2 (running a Docker-managed container) to /mnt/case-1.

The newly added container drift detection feature can be invoked from the command line by using drift or the alias diff. It also identifies whether the file is an executable. In order to view the drift for a particular container, provide the container ID after the drift command, i.e.:

sudo ce -i /mnt/case/ --output json --support-container-data supportcontainer.yaml drift <container ID here>

The tool’s output shows container drift from the EKS node.

EKS node — container drift

For containers managed by Docker, pointing the tool at the mount point /mnt/case-1 generates the following output.

EC2 — Docker managed container drift

At this point, I’d point ClamAV & YARA scans at those files and see what turns up. You also could copy the files & analyze them. I’d look at the log files for references of these files or a large number of errors/exceptions generated by the containerized application in the logs to see how the file was downloaded/written to the filesystem.

That’s pretty much it folks! It’s easy to think of container drift detection as a silver bullet in container forensics. Container drift detection does provide an easy win for Digital Forensics & Incident Response teams, as it helps quickly identify changes in the container. However, it must be noted that focusing solely on container drift does not provide a comprehensive view of container based incidents. From a dead box forensics perspective, there are several other critical areas to examine, like image analysis, log files, orchestration data, environment variables & secrets, volume & bind mounts, & host-level interactions. By expanding to these additional aspects of CFIR, DFIR teams can gain a more holistic understanding of containerized environments & better respond to incidents.

References:

  1. https://docs.docker.com/engine/storage/drivers/select-storage-driver/
  2. https://github.com/docker-forensics-toolkit/toolkit
  3. https://github.com/fox-it/dissect.target/blob/main/dissect/target/plugins/apps/container/docker.py
  4. https://github.com/google/docker-explorer
  5. https://github.com/keikoproj/kube-forensics
  6. https://github.com/google/container-explorer
  7. https://sysdig.com/blog/guide-kubernetes-forensics-dfir/
  8. https://www.didactic-security.com/resources/docker-forensics.pdf
  9. https://docs.docker.com/engine/storage/drivers/overlayfs-driver/#deleting-files-and-directories
  10. https://povilasv.me/how-to-monitor-containerd/
  11. https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/
  12. https://kubernetes.io/blog/2023/03/10/forensic-container-analysis/#file-system-changes-rootfs-diff-tar

Adrift in the Cloud: A Forensic Dive into Container Drift was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction to Malware Binary Triage (IMBT) Course

Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.

Enroll Now and Save 10%: Coupon Code MWNEWS10

Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.

Article Link: Adrift in the Cloud: A Forensic Dive into Container Drift | by Alex John | Dec, 2024 | Detect FYI