The Open Source Technology Improvement Fund (OSTIF) counters an often overlooked challenge in the open-source world: the same software projects that uphold today’s internet infrastructure are reliant on, in OSTIF’s words, a “surprisingly small group of people with a limited amount of time” for all development, testing, and maintenance.
This scarcity of contributor time in the open-source community is a well-known problem, and it renders the internet’s critical infrastructure vulnerable. To quote OSTIF, “because of the lack of a profit motive, core open-source projects are woefully underfunded and their resources are lacking. This leaves crucial Internet infrastructure susceptible to bugs, poor documentation, poor performance, slow release schedules, and even espionage.”
We couldn’t agree more.
Over the past year, we’ve had the pleasure of collaborating with open-source project teams through OSTIF on threat modeling assessments and secure code reviews. We believe our central mission of boldly advancing security and addressing technology’s most challenging risks closely aligns with OSTIF’s goals. Through our partnership with OSTIF, we have made significant contributions that improve the security posture of the open-source community. This blog post highlights some recent security assessments that OSTIF engaged Trail of Bits to conduct.
About our work
At Trail of Bits we do many types of security work. Some of our more popular offerings include secure code review powered by bespoke fuzzing harness and fuzz test development, our custom static analysis rulesets, and targeted manual review; threat modeling exercises involving architectural review, systems thinking, and threat scenario development; CI/CD pipeline hardening; and fix reviews. Optimally, an assessment involves engineers with expertise in several of these areas. This helps us provide our clients with the best value for their dollar.
Sometimes, we involve different types of expertise in an engagement by, for example, running threat modeling exercises and then performing a code review for the same client. When we follow threat modeling work with secure code review, our code review can start from the design-level findings that our threat modeling work resulted in. This means we can write fuzzing harnesses and fuzz tests targeting the most vulnerable areas of a given codebase more quickly! Following our secure code review with a fix review then gives our security engineers the chance to help guide and reassess the mitigations implemented based on our findings.
Due to the wide range of clients who engage us, client expectations and requirements can vary. In the OSTIF-organized assessments we’ll cover below, you’ll see different combinations of these types of work. Some of these engagements included both a threat modeling exercise and a secure code review, while others just focused on code review. Common to all of these projects is our development of a tailored security assessment strategy based on the nature of the project and the client’s needs.
Linux kernel release signing
The Linux kernel runs on devices from ordinary smartphones, to the servers that make up the web’s most widely used infrastructure, to supercomputers. The internet as we know it effectively runs on Linux. A critical part of Linux development is kernel release signing, which allows users to cryptographically verify the authenticity of kernel releases to ensure their trustworthiness.
In this review, which took place from March to April of 2021, we led technical discussions with the Linux Foundation and examined their documentation on the kernel release signing process. Our assessment included an audit of the management of signing keys, developers’ workflow for signing, and the cryptographic algorithms involved in the signing and verification steps.
One of our major recommendations was that the Linux Foundation enforce the use of smart cards to store private keys, which would prevent an attacker who compromises a developer’s workstation from being able to sign malicious code. We also advised that the Linux Foundation adopt wider key distribution methods to mitigate a compromise of the git.kernel.org server currently hosting public keys, replace older signature schemes like RSA and DSA with modern and more robust alternatives like ECDSA and Ed25519, and create documentation on key management policies to prevent mistakes in the signing process.
For further information about our findings and recommendations, refer to OSTIF’s announcement of the engagement and our full report.
CloudEvents
The CNCF Serverless Working group created the CloudEvents specification to standardize event declaration and delivery in a consistent, accessible, and portable way. It provides a standard format to share events across disparate cloud providers and toolsets, as well as SDKs in several programming languages.
From September to October of 2022, we performed a lightweight threat modeling assessment with the CloudEvents team, working to identify methods an attacker might use to compromise systems that implement the specification. Our team then followed up with a secure code review of the JavaScript, Go, and C# CloudEvents SDKs. For this engagement, we used a combination of automated testing tools, manual analysis, and a review of overall architecture and design. In total, we identified seven issues, including potential cross-site scripting in the JavaScript SDK and several vulnerable dependencies of all three SDKs.
Our final report includes these findings, the threat model we developed, several code quality recommendations, and guidance for using automated analysis tooling on the SDK codebases. Get all the details in OSTIF’s announcement of the engagement and our full report.
curl
It would be an understatement to say that curl is everywhere. This famous utility enables users to transfer data across a plethora of network protocols, with over 20 billion installations in “cars, television sets, routers, printers, audio equipment, mobile phones, tablets, set-top boxes, [and] media players,” according to curl’s website.
Thanks to OSTIF, our team had the privilege of conducting a review of both the curl binary and software library (libcurl) from September to October of 2022. We began our audit with a threat modeling assessment, a crucial exercise that deepened our understanding of the curl and libcurl internal components and how they work together. The resulting threat model significantly influenced our approach to reviewing the actual source code, concretizing our understanding of curl’s internals and helping us to decide which components to initially target for fuzzing and secure code review.
By the end of our code review, we found 14 issues, including two high-severity memory corruption vulnerabilities identified through fuzzing, which we conducted in parallel with our manual secure code review efforts. We helped expand curl’s fuzzing coverage and, through fuzzing that continued after the review ended, found several vulnerabilities including CVE-2022-42915, CVE-2022-43552, and a significant finding that started out as an off-hand joke. Check out our curl fuzzing coverage improvements and findings in the final report.
Kubernetes Event-Driven Autoscaling (KEDA)
KEDA is an automated scaling tool for Kubernetes containers. It comes with built-in support for numerous “scalers,” interfaces that can trigger scaling based on messages received from configured external sources, such as AWS SQS, RabbitMQ, and Redis Streams. KEDA efficiently manages the addition of Kubernetes pods to meet measured demand.
We started our review of KEDA in December of 2022 with a threat modeling exercise, walking through threat scenarios with the KEDA team. Using the threat model to inform our approach, we then conducted a code review. Our team used automated testing and manual review to discover eight findings. Among these findings was a failure to enable Transport Layer Security (TLS) for communication with Redis servers, creating a vulnerability that an attacker could exploit for person-in-the-middle attacks.
In addition to these findings, our final report presents our threat model, an evaluation of KEDA’s codebase maturity, a custom Semgrep rule we wrote to detect an encoding issue that we noticed was a pattern in the code, and long-term recommendations aimed at helping KEDA proactively enhance its overall security posture. Refer to our full report for more details.
Eclipse Mosquitto
In March of 2023, Trail of Bits had the opportunity to work with the Eclipse Foundation to assess the Mosquitto project. Mosquitto includes a popular MQTT message broker and client library (libmosquitto). Mosquitto has a broad range of applications, from home automation to bioinformatics to railway signaling infrastructure in the United Kingdom.
During this two-part engagement, we developed a threat model and then performed a secure code review of the broker application, libmosquitto, and associated command-line tools (e.g., mosquitto_passwd). Our threat model identified architecture-level weaknesses such as a lack of configurable global rate limiting in the broker and inadequate defenses against denial of service from infinite message looping. See our threat model report for these discoveries and more.
Our findings from the secure code review included a remotely triggerable buffer over-read in the broker that would cause heap memory to be dumped to disk, multiple file handling issues that could allow unauthorized users to access password hashes, and improper parsing of an HTTP header that could enable an attacker to bypass auditing capabilities and IP-based access controls for the Mosquitto WebSocket transport. Read more about these findings in our secure code review report.
Eclipse Jetty
Jetty is one of the oldest and most popular Java web server frameworks. It integrates with numerous other open-source applications including Apache Spark, Apache Maven, and Hadoop, as well as proprietary software like Google App Engine and VMWare’s Spring Boot. We were engaged to perform a lightweight threat model, secure code review, and fix review in March of 2023.
Due to the size of the Jetty codebase and the limited amount of time we had for threat modeling during this engagement, after determining the security controls to assess, we conducted a lightweight threat modeling exercise focused on identifying specific potential threats and insecure architectural patterns across components, rather than shallowly touching on many potential vulnerability types. Notable threat scenarios we discussed with the Jetty team included the implications of unsafe defaults; for example, we found that Jetty lacked default connection encryption, which could allow person-in-the-middle attacks against Jetty client connections, and that headers were inconsistently parsed, which could allow request smuggling or lead to other issues during, for example, HTTP/2 to HTTP/1 downgrade.
Oriented by the scenarios we explored during the threat modeling exercise, we conducted a three-week-long code review of Jetty. We discovered 25 findings including a possible integer overflow when parsing HTTP/2 HPACK headers (CVE-2023-36478) leading to resource exhaustion, a command injection vulnerability due to erroneous command-line argument escaping (CVE-2023-36479), and an XML external entity (XXE) injection vulnerability in the Maven metadata file parser. Our full report includes our threat model, codebase maturity evaluation, full list of findings, and fix review.
Eclipse JKube
JKube is a collection of helpful plugins and libraries for building, editing, and deploying Docker and OCI containers with Kubernetes or Red Hat OpenShift, integrating directly with Maven and Gradle. JKube can also connect to the external Kubernetes or OpenShift cluster to watch, debug, and log events. Working with the JKube maintainers between March and May of 2023, we conducted a lightweight threat model, a secure code review, and a fix review evaluating changes made to JKube after our secure code review.
After developing an understanding of the many JKube components, dependencies, and integrations, we discussed several potential threat scenarios with the JKube maintainers. Our threat modeling exercise identified a lack of common security defaults and a number of unsafe default settings, as well as general patterns of insufficient, handwritten sanitization for multiple input format types and unsafe Java deserialization practices. Our secure code review tested and expanded on our findings regarding unsafe defaults in JKube-generated artifacts. Our fix review validated that the JKube maintainers’ code changes sufficiently mitigated our code review findings. Check out our full report.
Flux
A CNCF-graduated project, Flux is a GitOps and continuous delivery tool that keeps Kubernetes state synchronized with configuration stored in a source such as a Git repository. OSTIF engaged Trail of Bits for a secure code review of Flux between July and August of 2023.
Our review resulted in 10 findings, including a path traversal vulnerability that an attacker could exploit to write to files outside of a specified root directory, particularly when Flux is included as a library in other applications. Other issues we noted included a failure to set an expiration date on cached sensitive data and a dynamic library injection vulnerability in the Flux macOS binary stemming from the fact that Apple’s Hardened Runtime feature wasn’t enabled.
Our final report includes the details of all of our findings, a codebase maturity evaluation, several code-quality issues that could contribute to a weaker security posture but were not thought to have an immediate security impact, and recommendations for incorporating regular analysis from the static analysis tools we used during the assessment into Flux’s CI/CD pipeline.
Dragonfly
Dragonfly is a peer-to-peer file distribution and image acceleration system. A CNCF-hosted project, Dragonfly features integrity checking for downloaded files, download speed limiting, the ability to isolate abnormal peers, and a public registry of artifacts that aims to be “the trusted cloud native repository for Kubernetes.” A subproject of Dragonfly, Nydus implements a content-addressable filesystem to enable high-efficiency distribution for cloud-native resources.
In July of 2023, Trail of Bits reviewed the Dragonfly codebase. Nineteen findings resulted from our secure code review, including five high-severity issues. Our findings included multiple server-side request forgery (SSRF) vulnerabilities that could enable unauthorized attackers to access internal services, an issue in the peer-to-peer API that could allow attackers to read and write to any file on a peer’s machine, and the ability for a peer to render the mTLS authentication scheme ineffective by obtaining a valid TLS certificate for any IP address.
Along with the details of our findings, our final report includes a codebase maturity evaluation, a list of code quality issues, guidance for running static and dynamic analysis tooling on the Dragonfly codebase, and our fix review.
What the future holds
These engagements may be complete, but we will continue to demonstrate our dedication to securing the internet’s open-source infrastructure going forward. We’re always learning, too! We iterate on and improve our methods, static analysis rulesets, and tooling with every assessment, incorporating new techniques and automated testing strategies, as well as client feedback. Expect more content in early 2024 about our ongoing and future work in partnership with OSTIF, including discussions of two more secure code reviews that are currently in progress. We also plan to publish a deep dive into the improvements we made to curl’s fuzzing infrastructure and technical details of some of the more interesting vulnerabilities we found during our OSTIF-organized OpenSSL and Mosquitto engagements.
Concurrently, Trail of Bits will continue supporting OSTIF’s mission through fix reviews (where contracted) for our completed secure code reviews and threat models to ensure that the vulnerabilities and design flaws we identified are mitigated. We’re very excited to take on further work that OSTIF has for us, whether it be threat modeling, secure code review, or providing security guidance in other ways.
Article Link: Securing open-source infrastructure with OSTIF | Trail of Bits Blog