ATT&CK® Evaluations Managed Services (2024): Actionable Insights and the Challenge of Dual Adversaries
Written By Lex Crumpton and Amy L. Robertson
Introduction
We are thrilled to unveil the results from the ATT&CK® Evaluations Managed Services 2024 — Round 2. Eleven participants took part in the evaluation, showcasing their capabilities in detecting and reporting on real-world adversary behaviors. You can explore the full results on the ATT&CK Evaluations website and the in-depth background on our chosen adversaries in the ATT&CK Evaluations Library.
Our mission for the Managed Services evaluations is to provide end-users with clear, objective insights into the defensive capabilities of participants. We also aim to foster a collaborative environment where gaps and areas for improvement in defensive coverage can be identified, driving innovation and advancing the overall security posture of the industry.
In this blog post, we will unveil the adversaries featured in Round 2, introduce new metrics used by the Evaluations team to assess the participating MSSPs, explore the addition of environmental noise, and guide you on accessing the release components.
Unmasking the Adversaries: Emulation Overview
For the Managed Services evaluations, we employ a “black box” methodology, where the emulated adversary is not revealed to the public or Evaluation participants prior to execution. This approach is designed to reflect real-world scenarios, where defenders identify and respond to threats without prior knowledge of the adversary’s identity or tactics.
This round emulated a multi-subsidiary compromise with overlapping operations focusing on defense evasion, exploiting trusted relationships, data encryption, and inhibiting system recovery.
Adversary Profiles: menuPass & ALPHV/BlackCat
Active since at least 2006, menuPass (aka APT10) is linked to China’s Ministry of State Security. Known for its global espionage campaigns, menuPass targets sectors like aerospace, construction, government, and telecommunications.
A ransomware-as-a-service operation, ALPHV BlackCat emerged in 2021, targeting various industries with a flexible ransomware strain capable of cross-platform attacks on Windows, Linux, and VMware systems.
The emulation mirrored menuPass living-off-the-land techniques, utilization of custom, fileless malware, anti-analysis tactics for defense evasion, and exploitation of trusted third-party relationships for credential access. For ALPHV/BlackCat, the signature behaviors we emulated and evaluated were defense evasion, data exfiltration, data encryption, data destruction, and system recovery obstruction behaviors.
The Multi-Vector Intrusion
The menuPass scenario featured an intrusion targeting two subsidiaries of a fictitious global pharmaceutical company. The attack began with the adversary using previously stolen credentials from an IIS server administrator to remotely access the Microsoft IIS web server. Leveraging DLL sideloading, menuPass executed SigLoader and FYAnti, eventually deploying QuasarRAT to establish command and control (C2) communications.
Following initial access, the adversary conducted network discovery and captured keylogging credentials, enabling lateral movement to the domain controller. Once on the domain controller, menuPass escalated the attack by dumping the NTDS database to obtain additional credentials and uncovering a domain trust to a second subsidiary network. The adversary identified a workstation with access to both subsidiary networks and pivoted to it, again using DLL sideloading to execute SigLoader, this time with SodaMaster as the final payload.
With access to the target workstation, menuPass compromised a file server using Impacket’s secretsdump to retrieve further credentials, which were then used with WMIexec.vbs to gain execution on the file server. The file stages of the attack involved using WinRAR and robocopy to compress and exfiltrate files of interest, completing the intrusion.
Image 1: menuPass Operations FlowThe ALPHV BlackCat scenario depicted an affiliate of the ALPHV BlackCat group launching an attack against a subsidiary of the same fictitious global pharmaceutical company. The emulation began with an Initial Access Broker gaining access to a contractor organization, subsequently providing the ALPHV BlackCat affiliate with Remote Desktop Protocol (RDP) access to the organization’s network.
Upon gaining access, the adversary performed network discovery and collected credentials from password stores on the initial workstation. These credentials facilitated the compromise of a backup server containing administrator privileges. Using the compromised administrator credentials, the adversary exfiltrated data from multiple hosts utilizing the ExMatter exfiltration tool. Following data exfiltration, the BlackCat ransomware was executed to encrypt virtual machines running on a Linux server and files on Windows hosts across the network.
In addition to encrypting data, the BlackCat ransomware inhibited system recovery options by deleting volume shadow copies and clearing Windows Event logs, significantly complicating recovery efforts and increasing the attack’s impact.
Image 2: ALPHV BlackCat Operations FlowA Refined Approach to Adversary Emulation
To help end users make an informed choice about the Managed Security Service Provider (MSSP) capabilities that are most valuable for their use case and environment, we introduced:
- new Reporting Metrics based on the Vendor Participants results, designed to provide a clear, comprehensive, and actionable assessment of MSSP performance.
Image 3: ATT&CK Evaluations Website Reporting Metrics- reviewing a Focused subset of executed behaviors based on a set of stringent criteria aimed at improving the depth and quality for the evaluation and;
Image 4: ATT&CK Evaluations Focus Step FeatureNew Reporting Metrics
These metrics — Actionability, Volume, Mean Time to Detect (MTTD), and Analysis/Enrichment — go beyond traditional evaluations to highlight both the quality and effectiveness of the security services provided.
Actionability: This metric ensures that alerts contain detailed and relevant information, enabling SOC analysts to effectively triage and respond to incidents. Actionability assesses whether alerts cover the essential aspects of an incident (What, Where, When, Who, Why), helping to streamline the incident response process and reduce the time analysts spend gathering additional information.
Volume: By capturing the total number of alerts received from each participant, this metric provides a balanced view of the alerting strategy. Volume is categorized into Email Alerts and Console Alerts, further broken down by severity (Low, Medium, High, Critical). This data helps MSSPs strike the right balance between alert frequency and relevance, a common challenge for every SOC.
Mean Time to Detect (MTTD): MTTD measures the average time it takes for an MSSP to detect a security incident or breach. Faster detection means quicker responses and less time for adversaries to cause harm. This metric highlights the importance of timely detection and is calculated by averaging the time taken to send an email alert about adversary activity after the incident occurred.
MTTD = [ SUM (Time Detected − Time Incident Occurred) ] / Number of Incidents
Analysis/Enrichment: This metric assesses the quality and effectiveness of MSSP After-Action Reports (AARs), focusing on clarity, accuracy, enrichment, and usability. High-quality analysis helps customers understand incidents thoroughly and apply these insights to improve their overall security posture and readiness for future threats.
Focused Evaluation
The selection criteria include alignment with the objectives of the round, the inclusion of new or diverse implementations of behaviors, and an emphasis on reflecting adversary tradecraft and operations. By prioritizing and scoping the evaluated behaviors to a carefully chosen subset, we can conduct a more in-depth analysis and gather more meaningful metrics beyond merely reporting status.
From the total 174 sub-steps executed, we selected 44 key sub-steps for focused evaluation. This selection was guided by the following considerations:
- Execution Feasibility: Ensuring that the Red Team (RT) successfully executed the technique for all vendors.
- Detection Capability: Verifying that the MSSPs have the capacity to detect the technique.
- Clarity in Detection Criteria: Ensuring that the detection criteria are concise and unambiguous, leaving no room for varied interpretations.
- Reflective of Adversary Tradecraft: Selecting techniques that accurately reflect adversary tradecraft and operations.
- Differentiation and Sophistication: Balancing the usage of new complex implementations and previously tested techniques to baseline or push defenses.
By honing in on these specific areas, our goal was to improve the quality of the evaluation and provide vendors with more actionable insights into their detection capabilities.
Evaluation Infrastructure Updates
The Managed Services 2024 Evaluation was the first to be conducted in Amazon Web Services (AWS) Cloud Infrastructure, allowing participants to access AWS Traffic Mirroring data natively. This move also enables the inclusion of macOS in the upcoming Enterprise 2024 ATT&CK Evaluations.
Environmental Noise
While the first round of our Managed Services offering introduced noise within the environment, this round introduced an automated approach to generating noise to improve upon consistency and robustness. By emulating the activity of several users and administrators within the network, noise generation enabled measurement of total alert volume while laying the foundation for several metrics for future evaluation rounds.
Release Components
Explore the detailed Results, Technique Scope, Environment diagrams, and Malware overviews on the ATT&CK Evaluations website. Access the technical collateral and emulation plans in the ATT&CK Evaluations Library website to understand, recreate, and defend against the adversary behaviors showcased in this evaluation.
Looking Forward
As we continue to advance our ATT&CK Evaluations, we are excited to share what’s on the horizon. In upcoming rounds (Enterprise 2024 ATT&CK Evaluations), we will further refine our evaluation methods and expand our focus to include even more sophisticated adversary behaviors and detection techniques. Stay tuned for our forthcoming blog posts, where we will guide you through navigating the results, explain how we calibrate the results to ensure accuracy and fairness, and provide insights on how to effectively use these results to enhance your security posture. Our goal is to equip you with the knowledge and tools needed to make informed decisions about your Managed Security Service Provider (MSSP) and stay ahead of evolving threats. Thank you for joining us on this journey, and we look forward to providing end-users with more clear, objective insights into the defensive capabilities.
About MITRE Engenuity ATT&CK® Evaluations
ATT&CK® Evaluations is built on the backbone of MITRE’s objective insight and conflict-free perspective. Cybersecurity vendors turn to the ATT&CK Evals program to improve their offerings and to provide defenders with insights into their product’s capabilities and performance. ATT&CK Evals enables defenders to make better informed decisions on how to leverage the products that secure their networks. The program follows a rigorous, transparent methodology, using a collaborative, threat-informed, purple-teaming approach that brings together vendors and MITRE experts to evaluate solutions within the context of ATT&CK. In line with MITRE Engenuity’s commitment to serve the public good, ATT&CK Evals results and threat emulation plans are freely accessible. https://attackevals.mitre-engenuity.org/
© 2024 MITRE Engenuity, LLC. Approved for Public Release. Distribution unlimited 23–03986–1.
ATT&CK® Evaluations Managed Services (2024): Actionable Insights and the Challenge of Dual… was originally published in MITRE-Engenuity on Medium, where people are continuing the conversation by highlighting and responding to this story.