The high-profile web hosting company Cloudflare said last week that a sophisticated attacker gained access to code repositories used by the company, and made off with sensitive internal code. This was just the latest such attack targeting the firm.
In a post on February 1 co-authored by Cloudflare CEO Matthew Prince, CTO John Graham-Cumming and CISO Grant Bourzikas, the company revealed the details of a November, 2023 malicious campaign that lasted roughly 10 days and saw malicious actors make off with 76 different source code repositories from Cloudflare’s Atlassian Bitbucket source code repository.
Cloudflare tied the incident to a breach at online identity provider Okta, which happened in October of last year. In that incident, malicious actors compromised the company’s internal network and downloaded a report that included names and email addresses of all clients that use its customer support system. The company said the malicious actors behind the campaign also accessed “additional reports and support cases” that contained contact information for Okta-certified users, some Okta employee information and Okta Customer Identity Cloud (CIC) customer contacts.
The Okta breach led to an earlier attack on Cloudflare, disclosed in October, in which attackers used an authentication token retrieved from Okta to “pivot into” Cloudflare’s Okta instance. The company said it detected and blocked that attack at the time with no discernable impact on its security posture.
Here's what we know about the latest Cloadflare breach – and lessons your team can learn from the incident.
[ Key takeaways: State of Software Supply Chain Security 2024 | Get the full report ]
Okta: The breach that keeps on giving
In the latest attack the threat actors leveraged a different set of Cloudflare credentials exposed in the Okta breach that were meant to be rotated, but that the company overlooked. Those included a Moveworks service token that gave them access to Cloudflare’s Atlassian system, and credentials for three service accounts leaked in the Okta compromise: one used by the SaaS-based Smartsheet application with administrative access to our Atlassian Jira instance; another for a Bitbucket service account used to access Cloudflare’s source code management system; and a final set of credentials for an AWS environment with no global access —and no customer or sensitive data, Cloudflare said.
With those credentials in hand, the threat actor had what they needed to establish a beachhead within Cloudflare’s environment with administrative access to Atlassian Jira. That allowed the threat actor to leverage the Atlassian Bitbucket git archive feature to download the 76 repositories to the Atlassian server from which they could be exfiltrated to an external source. While Cloudflare said it does not have proof that the repositories were exfiltrated, it assumes they were.
According to the post, the repositories that were downloaded contained source code related to key Cloudflare functions including how backups work; how the company’s global network is configured and managed; and how Cloudflare manages identity and remote access. There was also code related to Cloudflare’s use of Terraform and Kubernetes. (Terraform is an infrastructure as code application made by Hashicorp.)
Cloudflare was quick to point out that customer data was not compromised in the attack, noting in its post about the incident
“We saw no evidence whatsoever that the threat actor got access to our global network, data centers, SSL keys, customer databases or configuration information…Their access was limited to the Atlassian suite and the server on which our Atlassian runs.
It's no secret that stolen code can be a problem
If Cloudflare’s actual operating environment and customer data weren’t compromised, is the theft of raw source code a concern? The answer, say security experts, is “yes.”
Ashlee Benge, Director of Threat Intelligence at ReversingLabs, said source code leaks could includes exploitable software vulnerabilities, plaintext passwords, certificates used for authentication, and more.
"There is a lot of stuff hidden in source code."
—Ashlee Benge
Eric Milam, Senior Cyber Security Executive at ReversingLabs, said that with access to the raw source code, attackers can also gain an intimate knowledge of how a company’s systems “fit together."
“Removing the black box is another big advantage that may show its weakness over time."
—Eric Milam
Even as Cloudflare casted doubt on the raw value of the code itself in its post, the company acknowledged the risks of exposed secrets and code flaws.
“Our focus was not on someone having access to the source code, but whether that source code contained embedded secrets (such as a key or token) and vulnerabilities. [However], a small number of the repositories contained encrypted secrets, which were rotated immediately even though they were strongly encrypted themselves.”
In an internal “all hands on deck” effort in the wake of the attack dubbed “Code Red” (“no,” not that Code Red), Cloudflare directed its engineering teams to scour the stolen source code repositories for code vulnerabilities and other avenues for a malicious actor to “mount a subsequent attack.”
That robust effort to remediate the attack may prove sufficient to ward off follow-on attacks leveraging secrets or intelligence gained from the source code. However, Cloudflare should be on guard for the myriad ways that exposed code might undermine its security. Those range from references to particular staff or developers that might lurk in source code comments, or metadata and that could provide valuable intelligence for future attacks.
“I would be preparing to spearfish them after the buzz had died down about this compromise."
—Ashlee Benge
Third-party risk and the scourge of leaked secrets
This latest Cloudflare incident underscores the continuing risks to organizations posed by both third-party software and leaks of development secrets alike.
Cloudflare’s existing “zero trust” security model appears to have greatly limited the scope of the breach (at least by the company’s account). However, attackers's ability to leverage credentials stolen from a third-party identity provider (Okta) and gain persistent access to Cloudflare’s development environment underscores the acute exposure that even sophisticated firms have by way of third-party providers. A similar lesson lurked in the January 2023 disclosure by the firm CircleCI that it had been compromised.
Being prepared means being aware of what secrets and risks lurk in your code. ReversingLabs' State of Software Supply Chain Security 2024 reported the discovery of 40,000 secrets leaked across four major open source platforms in 2023: npm, PyPI, RubyGems and NuGet, in 2023 alone. Those included embedded and encrypted private keys, web service API keys and access credentials as well as plaintext credentials buried in network protocol strings.
While not all such credentials pose a security risk (placeholder- and expired-credentials may not pose any risk), organizations need to be aware of the security exposure that source code poses and the varieties of sensitive information that may lurk in both open source and proprietary code repositories.
Article Link: https://www.reversinglabs.com/blog/cloudflare-source-code-breach-lessons-learned