Securing Amazon EC2 with CloudTrail and Sysdig

[MalBot]

Elastic Compute Cloud (EC2) is arguably one of the most popular AWS services, and really needs no introduction but here is one anyway.

The AWS Shared Responsibility model states that you are responsible for many things in EC2, including AWS EC2 Security Groups, securing the OS, the network access, and the Elastic Block Store (EBS) for all your instances, just to name a few.

AWS provides a range of security controls (most of which come at an additional cost) to help you meet your responsibilities, but this is left up to you, and your mileage may vary.

Although this blog is all about just the AWS EC2 service, Sysdig has you covered with more than 188 out-of-the-box rules for the most important AWS services you might be using.

Today, we will uncover how changes in your AWS EC2 configuration can create major security holes.

We will then hunt for these high-risk configuration events using native tools, and then we will compare this to how Sysdig Secure helps in these situations.

Which Amazon EC2 Configuration events create a Threat?

Sysdig has identified 20 risky configuration events, and this list is extensible.

Several of these events map to one or more well-known MITRE ATT&CK TTPs, https://attack.mitre.org/matrices/enterprise/cloud/iaas/ that are essential for cloud infrastructure threat mapping.

Security Event	Impact	Description
Allocate New Elastic IP Address to AWS Account	High	Detects that a public IP address has been allocated to the account.
Run Instances with Non-standard Image	High	Detects launching of a specified number of instances with a non-standard image.
Run Instances in Non-approved Region	Medium	Detects launching of a specified number of instances in a non-approved region.
Run Instances	Medium	Detect launching of a specified number of instances.
Revoke Security Group Ingress	High	Detect removal of the specified ingress rules from AWS EC2 securitygroup.
Revoke Security Group Egress	High	Detect removal of the specified egress rules from AWS EC2 securitygroup.
Replace Route	Medium	Detecting replacing an existing route within a route table in a VPC.
Modify Snapshot Attribute	Medium	Detects addition or removal of permission settings for the specified EC2 snapshot.
Modify Image Attribute	Medium	Detects modification of the specified attribute of the specified AMI.
Make EBS Snapshot Public	High	Detect making public an EBS snapshot.
Get Password Data	High	Detects retrieval of the encrypted administrator password for a running Windows instance.
EC2 Serial Console Access Enabled	High	Detects AWS EC2 serial Console Access enabled in the account for a specific region.
Disable EBS Encryption by Default	Medium	Detect disabling EBS encryption by default for an account in the current region.
Describe Instances	Medium	Detect description of the specified AWS EC2 instances or all EC2 instances.
Delete Subnet	High	Detects deletion of the specified subnet.
Delete Cluster	High	Detect deletion of the specified cluster
Create Snapshot	Medium	Detects creation of an EBS volume snapshot and stores it in Amazon S3.
Authorize Security Group Ingress	High	Detects addition of the specified ingress rules to AWS EC2 securitygroup.
Authorize Security Group Egress	High	Detects addition of the specified egress rules to AWS EC2 securitygroup.
Associate Elastic IP Address to AWS Network Interface	High	Detects that a public IP address has been associated with a network interface.

We will focus on three risky moves from this list that we can all agree are very interesting from a security perspective:

1. Run Instances with Non-standard Image
2. Authorize Security Group Ingress
3. Disable EBS Encryption by Default

Custom AMIs are Risky

According to Amazon, an Amazon Machine Image (AMI) is a template that contains a software configuration (for example, an operating system, an application server, and applications). From an AMI, you launch an instance, which is a copy of the AMI running as a virtual server in the cloud.

In your AWS account there are probably several different types of AMIs in use, and you need to know what is in each one, including the OS and the software packages. These versions of OS and software may contain vulnerabilities and you need a run-time solution for that.

But just as importantly, the first step is awareness. In other words, you can’t secure what you can’t see, so you need to be alerted instantly when a custom AMI is instantiated. Only then can you triage the situation and take corrective action.

Sysdig finds all occurrences of this event for you with the built-in Run Instances with Non-standard Image rule.

Wide-Open ports in Amazon EC2, you say?

There are certain ports that arguably SHOULD be open to the internet, such as those used for Web Servers such as port 443. But there are other ports that really should never be open like this, and if they are, it’s either a big mistake or an intrusion situation.

The following table shows some ports which should / should not be open to the public:

Port	Purpose	Should this be open to the Internet?
80,8080,8008	HTTP (cleartext)	Probably not, but unfortunately a lot of websites are cleartext
443,4434,4443	HTTPS (encrypted TLS)	Yes, probably should
25,587	SMTP	Maybe, although cloud providers may block this
22,3389	SSH and RDP	No!

Out of these ports, let’s pick port 22, for SSH because there really is no good reason why port 22 should be open to the Internet, making this a good candidate for our example.

It’s time to take your best guess: How many open instances of port 22 do you think exist right now in the AWS IP Space?

Let’s use Shodan public search to find out. Did you guess 3,000,000? If so, you win the prize (a free trial ) because there are roughly 3 million instances listening on port 22 within the global AWS IP Space.

The distribution amongst Amazon major IP blocks (according to Shodan) is as follows:

The Scene of the Crime

Can you manually find all security groups that are wide open to the world within your AWS account? Yes, and here’s a simplified example of using the AWS CLI for this:

The obvious downsides to this approach is it’s pretty manual and unsophisticated so let’s ramp it up a notch.

We found a fun project that finds exposed IP addresses in our AWS account, makes a Shodan API call to get a rating for them, and stores everything in a DynamoDB database.

To get started we just clone the repo. Running the Lambda, we get nice results as shown here in our AWS DynamoDB table:

Ignoring for a moment that every additional chunk of code (no matter how easily obtained) increases our management burden, do you see the fundamental problem here? If you guessed this suffers from the same TOCTOU problem, you are correct.

With this AWS Lambda, we only find out about public IP addresses when we run it. If someone in our AWS account makes an IP public after we run this, we won’t know until the next time we run.

And if they reverted their changes before we run the Lambda again, we will never know it happened.

Recreating the Crime Scene

So how do we create one of these wide-open AWS EC2 Security Group rules? Here we use the AWS CLI to create and revoke a security group that allows all access from 0.0.0.0/0. It’s actually doing three things:

1. Finding the InstanceID based on name tag
2. Finding the corresponding Security Group ID for this InstanceID
3. Adding 0.0.0.0/32:22 to this Security Group ID

In between these two AWS EC2 security group events, the attacker could have exploited an SSH session while the Amazon EC2 security group was wide open to the Internet on port 22.

Sysdig Secure finds all occurrences of new security ingress rules for you with the Authorize Security Group Ingress rule.

EBS Encryption

Encryption is, in general, a good thing. AWS EC2 EBS encryption is also a good thing, and should be enabled by default.

According to AWS, encryption operations occur on the servers that host EC2 instances, ensuring the security of both data-at-rest and data-in-transit between an instance and its attached EBS storage.

While it’s possible some rare and old instance types do not support encryption, that should be the exception rather than the rule, and handled on a case-by-case basis.

So, as shown here, if somebody goes and disables the default EBS Encryption setting for the entire AWS account, that is also a big deal from a security perspective, and you very much need to be alerted on this event:

Sysdig finds all occurrences of this risk event with the Disable EBS Encryption by Default rule.

Digging for Risks with CloudTrail

AWS wouldn’t leave us without a way to track our changes, and the great audit log in the sky named CloudTrail keeps track of everything.

Even better, CloudTrail doesn’t suffer from the same time-of-use problem, because it sees events as they happen, and doesn’t depend on the frequency of some function.

Here we can see some juicy AWS EC2 Security Group events, with all the right fidelity:

CloudTrail can of course have many millions of events, and it’s likely almost all of these will be good events which have no security implications at all. But how do you know which are which? Here a StackOverflow member posts a question that illustrates how painful finding interesting events can be:

Besides being essentially a giant log in the sky, millions of lines in size, CloudTrail doesn’t send us any alerts, but even if it could, it has no concept of right and wrong – so the alerting would not make any sense.

So, to make this useful, you could

1. Use a Tool that knows right from wrong, and can send Alerts
2. Roll up your sleeves and take a DIY approach using AWS native tools.

But as we wrote in an earlier blog post, it will be very difficult to get what you need using native tools .

Which brings us to Sysdig Secure for Cloud.

CloudTrail, meet Sysdig Secure

Sysdig watches CloudTrail constantly, so the moment CloudTrail knows about something, Sysdig knows about it too.

But unlike CloudTrail, Sysdig knows the difference between right and wrong, Sysdig will Alert us with the right data at the right time, and most importantly Sysdig works right out-of-the-box.

To show the breadth of high-risk Amazon EC2 events that Sysdig finds by default, let’s look at a couple of examples.

First, as we saw above, a Top 10 Worst-Practice is opening up a Security Group to 0.0.0/0 (the world) as shown here:

So for fun, let’s find the Who/What/When/Where if of a couple of high-risk events:

1. Creating a 0.0.0.0 Security Group
2. Disabling EBS Encryption

How are you going to do this? Find out by doing a quick walk-through using our mini-demo:

Sysdig and CloudTrail – Better Together

AWS EC2 provides a wide surface for attacks, and within that surface, security groups rank near the top. AWS Security Group limits are a critical component of a safe and secure deployment.

We also looked at another awful configuration event: Turning off encryption for AWS EBS. Both of these are events you definitely want to know about!

CloudTrail sees everything, but unfortunately doesn’t know right from wrong, and sometimes has millions of entries. Somewhere in CloudTrail, high-risk events are hiding like buried treasure.

Sysdig mines CloudTrail for these diamonds in the rough, finds them, runs them through your Sysdig security policy, and instantly gives you the Who/What/When/Where of the situation.

Sysdig Secure makes it easy to see if holes start to appear in your Amazon EC2 security, and makes it easy to fix.

Although this blog focused exclusively on EC2, Sysdig includes out-of-the-box security for pretty much all the important AWS services.

And Sysdig CloudTrail rules use the same format as the Open-Source Falco project (which has been downloaded more than 30 million times) giving you the power to make your own custom rules.

But don’t take our word for it – if you have a few minutes, get started today at no cost and see the proof for yourself.

The post Securing Amazon EC2 with CloudTrail and Sysdig appeared first on Sysdig.

Article Link: Securing Amazon EC2 with CloudTrail and Sysdig – Sysdig