By Michael Brown and Adelin Travers
The US Army’s Program Executive Office for Intelligence, Electronic Warfare and Sensors (PEO IEW&S) recently issued a request for information (RFI) on methods to implement and automate production of an artificial intelligence bill of materials (AIBOM) as part of Project Linchpin. The RFI describes the AIBOM as a detailed list of the components necessary to build, train, validate, and configure AI models and their supply chain relationships. As with the software bill of materials (SBOM) concept, the goal of the AIBOM concept is to allow providers and consumers of AI models to effectively address supply chain vulnerabilities. In this blog post, we summarize our response, which includes our recommendations for improving the concept, ensuring AI model security, and effectively implementing an AIBOM tool.
Background details and initial impressions
While the US Army is leading research efforts into adopting this technology, our responses to this RFI could be useful to any organization that is using AI/ML models and is looking to assess the security of these models, their components and architecture, and their supply chains.
Project Linchpin is a PEO IEW&S initiative to create an operational pipeline for developing and deploying AI/ML capabilities to intelligence, cyber, and electronic warfare systems. The proposed AIBOM concept for Project Linchpin will detail the components and supply chain relationships involved in creating AI/ML models and will be used to assess such models for vulnerabilities. As currently proposed, the US Army’s AIBOM concept includes the following:
- An SBOM detailing the components used to build and validate the given AI model
- A component for detailing the model’s properties, architecture, training data, hyperparameters, and intended use
- A component for detailing the lineage and pedigree of the data used to create the model
AIBOMs are a natural extension of bill of materials (BOM) concepts used to document and audit the software and hardware components that make up complex systems. As AI/ML models become more widespread, developing effective AIBOM tools presents an opportunity to proactively ensure the security and performance of such systems before they become pervasive. However, we argue that the currently proposed AIBOM concept has drawbacks that need to be addressed to ensure that the unique aspects of AI/ML systems are taken into account when implementing AIBOM tools.
Pros and cons of the AIBOM concept
An AIBOM would be ideal for enumerating an AI/ML model’s components that SBOM tools would miss, such as raw data sets, interfaces to ML frameworks and traditional software, and AI/ML model types, hyperparameters, algorithms, and loss functions. However, the proposed AIBOM concept has some significant shortcomings.
First, it would not be able to provide a complete security audit of the given AI model because certain aspects of model training and usage cannot be captured statically; the AIBOM tool would have to be complemented by other security auditing approaches. (We cover these approaches in more detail in the next section.) Second, the proposed concept does not account for AI/ML-specific hardware components via a hardware bill of materials (HBOM). Like the rest of the ML supply chain, specialized hardware components that are commonly used in deployed AI/ML systems like GPUs may have unique vulnerabilities like data leakage and should thus be captured by the AIBOM.
Additionally, the AIBOM tool would miss important AI/ML-specific downstream system dependencies and supply chain paradigms like machine-learning-as-a-service prediction APIs (common with LLMs). For instance, an AI model provider may be subject to attack vectors that would be difficult or impossible to detect, such as poisoning of web-scale training datasets and “sleeper agents” within LLMs.
Ensuring AI/ML model security
Many aspects of AI/ML model training and use cannot be captured statically and thus would limit the proposed AIBOM concept’s ability to provide a complete security audit. For example, it would not capture whether attackers had control over the order in which a model ingests training data, a potential data poisoning attack vector. To ensure that the given AI model has strong supply chain security, the AIBOM concept should be complemented by other security techniques, such as data cleaning/normalization tools, anomaly detection and integrity checks, and verification of training and inference environment configurations.
Additionally, we recommend extending the AIBOM concept to account for data and model transformation components in AI/ML models. For example, the AIBOM concept should be able to obtain detailed information about the data labels and labeling procedure in use, data transformation procedures in the model pipeline, model construction process, and infrastructure security configuration for the data pipeline. Capturing such items could help detect and address vulnerabilities in the AI/ML model supply chain.
Implementing the AIBOM concept
There are several barriers to building effective and automated tools to conduct AIBOM-based security audits today. First, a robust database of weaknesses and vulnerabilities specific to AI/ML models and their data pipelines (e.g., model hyperparameters, data transformation procedures) is sorely needed. Proposed databases do not provide a strong definition of what an AI/ML vulnerability is and thus do not provide the ground truth needed for security auditing. This database should define a unique abstraction for AI/ML weaknesses and enforce a machine-readable format so that the abstraction can be used as a data source for AIBOM security auditing.
AIBOM tools must be used during the data collection/transformation and model configuration/creation stages of the AI/ML operations pipeline. In many cases (e.g., tools built on ChatGPT), these stages may be controlled by a third party. We advocate for third-party AI/ML-as-a-service providers to adopt transparent, open-source principles for their models to help ensure the safety and security of tools built using their platforms.
Finally, further research and development is needed to create tools for automatically tracing data lineage and provenance. Security and safety concerns with advanced AI/ML models have started to highlight the need for such capabilities, but practical tools are still years away.
Once these key research problems are solved, we anticipate that implementing AIBOM tools and auditing programs will require similar effort to implementing SBOM tools and programs. There will, however, be several key differences that will require specialized knowledge and skills. Today’s developers, security engineers, and IT teams will need to upskill in technical domains such as data science, data management, and AI/ML-specific frameworks and hardware.
Final thoughts
We’re excited to continue discussing and developing techniques and automated tools that support high-fidelity AIBOM-based security auditing. We plan to continue engaging with the community and invite you to read our full response for more details.
Article Link: Our response to the US Army’s RFI on developing AIBOM tools | Trail of Bits Blog