The recent SCARLETEEL incident highlights the importance of detecting security threats early in the development cycle. With Terraform state files, attackers can easily access sensitive information and gain unauthorized access to your cloud infrastructure.
In this case, the attackers exploited a containerized workload and used it to perform privilege escalation into an AWS account, stealing software and credentials. They also attempted to expand their reach by using a Terraform state file to access other connected AWS accounts within the organization.
In this blog post, we’ll explore how to use Sysdig Secure to detect SCARLETEEL and other similar security threats.
Sysdig Secure is a comprehensive cloud security platform that provides continuous monitoring, threat detection, and compliance enforcement across your cloud infrastructure. It integrates with popular DevOps tools such as Terraform, Kubernetes, and Docker to provide full-stack security visibility.
Initial Access
This attack was particularly advanced as it started from a compromised Kubernetes container and spread to the victim’s AWS account.
The attackers demonstrated knowledge of AWS cloud mechanics such as EC2 roles, Lambda serverless functions, and Terraform. Their primary motive was not cryptojacking, but rather the theft of proprietary software. However, it’s important to secure the public-facing web application from potential hijacking.
To this purpose, we need real-time intrusion detection of those Kubernetes apps.
Exploit public-facing web application
In the case of the SCARLETEEL attack, the adversary exploited an internet-exposed service deployed in a Kubernetes cluster. Once they accessed the container, they started performing different actions to proceed with their attack.
To mitigate the exposure, the victim could implement NetworkPolicies to allow only specific ingress/egress traffic inside and outside of the cluster. NetworkPolicies can also be used to prevent lateral movement and data exfiltration through the Kubernetes environment.
Thankfully, Sysdig Secure recommends least privileged NetworkPolicies to prevent unexpected network traffic from playing out.
Since Sysdig Secure analyzes the network traffic within your Kubernetes environment, we can easily whitelist the traffic that was historically relevant to the workload, and then drop any unexpected traffic.
The following image shows the Sysdig Secure Network Topology Graph.
Execution
Execution consists of techniques that result in adversary-controlled code running on a local or remote system, device, or other asset.
In the case of SCARLETEEL, the attacker executed a known malicious cryptomining binary called ‘xmrig.’ We will discuss both operations and how Sysdig Secure can prevent those threats at runtime.
Cryptomining in Kubernetes
There are multiple variations of Falco rules that are well-designed for detecting cryptomining behavior in containers, hosts, and Kubernetes. However, adversaries will do their best to evade detection and maintain a foothold where possible.
To begin, Falco can detect connections established to a cryptomining pool, whether it be an IP address, FQDN, or matching port number, as outlined in the following Falco rule.
An adversary could bypass this detection if they were to relay traffic through a Content Delivery Network (CDN) like Cloudflare, or through a simple HAProxy configuration, instead of pointing directly to those IPs, domain names, and ports. That’s why Network Policies are so important for securing your Kubernetes environment.
With Sysdig Secure’s new machine learning policy functionality, we detect the miner at the earliest possible stage, irrespective of filenames and network connections that were used.
As seen in the following screenshot, the machine learning algorithm detects based on behavior rather than matching string conditions. Additionally, xmrig was run as a container and was detected with an 88% confidence level based on its behavior.
Discovery
As mentioned earlier, cryptomining was just another step for the adversary.
Attackers often aim to achieve persistence in the cloud, exfiltrate sensitive data, and if they have enough privileges, they might create new resources such as EC2 instances or Kubernetes pods for cryptomining, which can impact an organization’s cloud bills.
Once the attacker obtained initial access into the cloud account, they started gathering information about the resources deployed in the AWS account. The activities reported in the table are just some of the API requests recorded in the AWS account.
Understanding vulnerable S3 buckets in public
A malicious actor can scrap an S3 bucket that allows public READ (LIST) access to list the objects it contains. This means that a malicious actor who has discovered the bucket’s URL can easily access and list all the objects within the bucket, including any sensitive data that might be stored there.
To exploit such an S3 bucket, a malicious actor would need to perform the following steps:
- Find the S3 bucket’s URL
- List the objects within the bucket
- Download the objects
There are several tools that facilitate this process, and attackers often use them to audit exposed buckets for misconfigurations. These are some examples:
To prevent such an exploit, it’s important to properly configure S3 bucket access controls to limit access to only authorized users and applications. This can be achieved by using bucket policies, IAM roles, and access control lists (ACLs) to restrict public access to the bucket and its objects. Additionally, it’s important to regularly monitor S3 bucket activity and access logs to detect any unauthorized access attempts and take appropriate action.
Credential access
Access keys are long-term credentials for an IAM user or the AWS account root user. You can use access keys to sign programmatic requests to the AWS CLI or AWS API (directly or by using the AWS SDK). For more information, see Signing AWS API Requests in the Amazon Web Services General Reference.
In the case of SCARLETEEL, a series of commands were run to get credentials from the metadata endpoint.
Lateral movement with AWS metadata endpoint
The AWS cloud metadata service, accessible via the IP address 169.254.169.254, provides valuable information to Amazon Web Services (AWS) instances, such as instance metadata, IAM role credentials, and user data.
While this service is essential for AWS instances to function correctly, it can pose a security risk if it’s accessible from within a container, as was seen in the SCARLETEEL attack. Thankfully, Sysdig Secure’s Falco implementation provides dedicated rules for cases where a container attempts to contact the cloud or EC2 metadata services, as seen in the below example:
- rule: Contact cloud metadata service from container
desc: Detect attempts to contact the Cloud Instance Metadata Service from a container
condition: >-
outbound and fd.sip="169.254.169.254" and container and
consider_metadata_access and not user_known_metadata_access
output: >-
Outbound connection to cloud instance metadata service
(proc.cmdline=%proc.cmdline connection=%fd.name %container.info
evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd
priority: NOTICE
tags:
- MITRE_T1552.005_unsecured_credentials_cloud_instance_metadata_api
- MITRE_T1552_unsecured_credentials
- MITRE_TA0006_credential_access
- MITRE_TA0007_discovery
source: syscallCode language: JavaScript (javascript)If a container can access the AWS metadata service, it can potentially retrieve sensitive information that can be used to compromise the security of the entire system. In the case of SCARLETEEL, the attacker was able to gain access to an AWS instance’s IAM role credentials, which could escalate their privileges and potentially gain access to other resources within the AWS environment.
When we have this kind of event, it is necessary to trigger the alarms and capture all subsequent events to really check if it is malicious. If we do not want to, or it is never necessary to collect this data in our environment, we can turn off the access to instance metadata.
Additionally, it’s important to ensure that containers are configured with the least privilege principle and access levels necessary for their intended function to reduce the attack surface. If insufficient network segmentation is in place, we can rely on Sysdig Secure policies to kill these hijacked processes.
Steal S3/Lambda credentials
Lambda functions and other serverless functions are commonly used to execute custom code without worrying about the infrastructure underneath, leaving a lot of flexibility for end users. There were different Lambda functions in the affected AWS account, mainly related to account automation.
The attacker started to enumerate and retrieve all of the Lambda functions located in a specific region in the AWS account using the proper API call. For example, you can use the AWS command below to list the functions. Behind the scenes, it is just REST API calls, so there are many ways to accomplish this task.
aws lambda list-functionsCode language: PHP (php)Stealing software from Lambda
After obtaining the list of functions, the attacker tried to dig deeper by downloading the Lambda code. Calling the AWS API, they were able to get the code location so that they could download the code that makes up the Lambda.
Ultimately, the attacker in this case successfully exfiltrated the Lambda cod. This emphasizes the need for deleting malicious behavior in IAM users, but also using a managed tool like Sysdig to limit the permissions associated with a Lambda role assigned to the IAM user.
Credential access via Terraform
Terraform state files contain all data in plain text, which may contain secrets. Storing secrets anywhere other than a secure location is never a good idea, and definitely should not be put into source control!
The attacker was able to list the bucket available and retrieve all of the data. Examining the data with different tools such as Pacu and TruffleHog during the incident investigation, it was possible to find both a clear-text IAM user access key and secret key in the terraform.tfstate file inside of an S3 bucket.
For more Terraform best practices, discover Terraform security guide.
Organization Lateral Movement
Lateral Movement consists of techniques that adversaries use to enter and control remote systems on a network. These techniques abuse default credentials, known accounts, and vulnerable services, and may also leverage dual-homed devices and systems that reside on both the IT and OT networks.
Excessive permissions
Similar to Kubernetes Network Policies and AWS Security Groups, we should apply the principle of least privilege to AWS users to prevent them from moving laterally. Users in AWS are created under the Identity & Access Management (IAM) service. For those struggling to limit IAM user permissions, Sysdig Secure automatically generates a least privileged IAM user policy based on permissions that are used vs. not used. This is most useful for users who are given full admin privileges by default, as seen in the below screenshot.
Most users are given excessive permissions in the cloud, as explained in the below video:
The real advantage here is that Sysdig Secure identifies where a user is of high risk, and provides admins a quick and simple way to remediate risk associated with cloud users. By removing a multi-tenancy role from a user who only needs to work in one specific AWS account and/or region, if their account were to be compromised, the adversary would have limited scope to move laterally from one AWS account to another.
Defense Evasion
Defense Evasion consists of techniques that adversaries use to avoid detection throughout their compromise. Techniques used for defense evasion include uninstalling/disabling security software or obfuscating/encrypting data and scripts.
Disable Cloudtrail
In the case of the SCARLETEEL attack, the adversaries disabled Cloudtrail Logging in AWS. That way, the threat detection monitoring tools that rely on audit activity from the cloud provider would no longer receive a new stream of events associated with the lateral movement in the cloud. The fact that an attacker can easily disable or modify Cloudtrail logging further emphasizes the importance of least privilege access in the cloud.
- rule: CloudTrail Logging Disabled
desc: >-
The CloudTrail logging has been disabled - this could be potentially
malicious.
condition: aws.eventName="StopLogging" and not aws.errorCode exists
output: >-
The CloudTrail logging has been disabled. (requesting user=%aws.user,
requesting IP=%aws.sourceIP, AWS region=%aws.region,
arn=%jevt.value[/userIdentity/arn], resource
name=%jevt.value[/requestParameters/name])
priority: warning
tags:
- MITRE_T1562.008_impair_defenses_disable_cloud_logs
- MITRE_TA0005_defense_evasion
- MITRE_T1562.001_impair_defenses_disable_or_modify_tools
- MITRE_T1562_impair_defenses
source: aws_cloudtrailCode language: JavaScript (javascript)Limiting the number of users who can modify specific services such as Cloudtrail ensures security teams can continue to monitor the adversaries’ behaviors without disruption.
Data Exfiltration
Data exfiltration is defined as when an unauthorized person extracts data from the secured systems where it belongs and moves it to insecure or unauthorized systems. Data exfiltration on the cloud could therefore means the unauthorized movement of data from cloud resources.
Steal proprietary software
Adversaries will often connect to command & control (C2) servers during the data exfiltration process. Being able to detect, in real time, any connections (inbound or output) to known bad C2 servers is critical if we wish to prevent anonymous data exfiltration. If a Tor exit node, for instance, was to communicate with your AWS account, Sysdig Secure can detect this action, as outlined in the below Falco rule.
- rule: AWS Suspicious IP Inbound Request
desc: >-
Detect inbound requests from known suspicious IP sources, such as TOR exit
nodes, to AWS services.
condition: >-
aws.sourceIP in (ti_anon_ips) and not (aws.eventName="ConsoleLogin" and
jevt.value[/responseElements/ConsoleLogin]="Failure")
output: >-
Suspicious IP Inbound Request (aws.sourceIP=%aws.sourceIP,
aws.region=%aws.region, aws.eventName=%aws.eventName, aws.user=%aws.user,
userAgent=%jevt.value[/userAgent], errorMessage=%jevt.value[/errorMessage])
priority: warning
source: aws_cloudtrailCode language: YAML (yaml)Sysdig Secure has a threat intelligence team responsible for updating these managed rules to detect the latest threats. Unlike open source Falco, the Sysdig Secure team is regularly updating lists associated with managed rules. Lists are designed for known bad IPs, domains, ports, hash values, binary names, as well as file/directory names. This takes a certain amount of responsibility away from the threat responders, as they can trust in Sysdig Secure to detect known bad IPs used in data exfiltration attacks.
Conclusion
Sysdig Secure was instrumental in detecting and thwarting the SCARLETEEL attack at the earliest possible stage, leveraging robust security measures such as zero trust networking via Kubernetes Network Policies and Sigkill actions to kill malicious container activity.
By quickly identifying the exploitation of a vulnerable pod and preventing the attacker from accessing sensitive information through the IAM role of the associated AWS user, Sysdig Secure’s multi-layered approach to detecting anomalous behavior and enforcing drift controls at runtime prevented container escape.
Through Sysdig Secure’s strong detection capabilities, the security team was alerted to these malicious activities before the attacker could escalate privileges and move laterally.
By securing vulnerable applications, limiting access to sensitive S3 buckets, and reducing the attack surface, we were able to prevent similar attacks in the future. From a posture standpoint, Sysdig Secure leverages OPA to enforce consistent policies across multiple infrastructure-as-code (IaC) sources (Terraform, Helm, Kustomize) and Kubernetes clusters, using a policy-as-code approach to security.
Finally, this incident underscores the need for end-to-end security solutions that detect and prevent threats in containers, cloud, and Kubernetes. It also highlights the critical importance of implementing proper security measures, staying vigilant with strong detections and alerts, and setting clear IAM restrictions to prevent attacks on Terraform state files.
We cannot ensure all threats will be detected, depending on their sophistication, but we can do our best to monitor all areas of the attack surface and enforce certain best practices to limit the blast radius in Kubernetes and Cloud.
The post How to Detect SCARLETEEL with Sysdig Secure appeared first on Sysdig.
Article Link: How to Detect SCARLETEEL with Sysdig Secure – Sysdig