Traditional software supply-chain security focuses on code: open-source libraries, build systems and container images. AI applications add new links to that chain: pre-trained models, fine-tuning datasets, embedding models, retrieval corpora and a fast-moving ecosystem of AI libraries. Most organisations have not yet inventoried them, let alone secured them.
The risk is not theoretical. On 27 February, JFrog's security researchers reported finding around 100 malicious models on Hugging Face, the largest public model hub, including models that executed code on the machine of anyone who loaded them.
Link 1: model files that run code
Many machine learning models, particularly those saved with PyTorch's default format, are serialised with Python's pickle module. Pickle can execute arbitrary code during deserialisation, which means that loading a model can run an attacker's code.
# Dangerous: torch.load with pickle can execute code embedded in the file
model = torch.load("downloaded_model.bin")
# Safer: safetensors stores only tensors, with no executable content
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
# And with the transformers library, prefer safetensors and pin a reviewed revision
from transformers import AutoModel
model = AutoModel.from_pretrained(
"org/model-name",
revision="4c1a2e0", # a specific commit you have reviewed
use_safetensors=True,
)Hugging Face scans uploads and flags unsafe pickle imports, and the safetensors format was designed to remove this risk. But older models, private registries and models passed around internally often still use pickle.
Link 2: tampered and backdoored models
A model can behave normally on standard benchmarks while containing hidden behaviour:
- In July 2023, Mithril Security demonstrated "PoisonGPT": an open model surgically edited to spread a specific false fact, uploaded under a name resembling a legitimate organisation. It passed standard benchmark checks.
- In January, Anthropic researchers published work on "sleeper agent" models trained to behave well except when a trigger appears, and found the backdoor behaviour persisted through standard safety training.
Downloading "a Llama variant from Hugging Face" without checking its provenance is equivalent to installing an unsigned binary from an unknown publisher.
Link 3: poisoned data
Fine-tuning datasets can be poisoned to insert biases, backdoors or degraded behaviour. Retrieval corpora for RAG systems can be poisoned too: an attacker who can add or edit documents in your knowledge base can plant content that the model will retrieve and trust, including prompt-injection payloads.
Link 4: AI libraries and packages
The AI ecosystem moves extremely fast. Frameworks, inference servers, vector databases and orchestration libraries are young, and many have had serious vulnerabilities, including remote code execution in model-serving and agent frameworks.
A newer twist is package hallucination: code assistants sometimes suggest installing packages that do not exist. Security researchers showed last year that attackers could register those hallucinated names and wait for developers to install them.
Link 5: third-party model APIs
Using a hosted model moves some risk to the provider, but not all: you depend on their security, their data handling, their availability and their model updates, which can silently change your application's behaviour.
Securing the AI supply chain
flowchart LR I[Inventory<br/>models, datasets,<br/>libraries, APIs] --> V[Vet & verify<br/>publisher, format,<br/>licence, hashes] V --> P[Pin & mirror<br/>internal registry,<br/>fixed revisions] P --> S[Sandbox & scan<br/>load in isolation,<br/>scan for unsafe ops] S --> T[Test behaviour<br/>evals incl.<br/>adversarial cases] T --> M[Monitor<br/>drift, CVEs,<br/>provider changes]
- Build an AI bill of materials. For every AI application, record which models (with versions and hashes), datasets, embedding models, libraries and external APIs it uses. Extend your existing SBOM process to cover them.
- Prefer trusted sources and safe formats. Use models from verified publishers, prefer safetensors, and refuse pickle files from untrusted sources.
- Pin and mirror. Pin model revisions and library versions, and host approved models in an internal registry rather than pulling from public hubs at runtime.
- Scan and sandbox. Scan model files for unsafe serialisation, and load untrusted models only in isolated environments without credentials or network access.
- Evaluate behaviour, not just benchmarks. Test models on your own evaluation sets, including adversarial and safety cases, before and after every change.
- Control your data pipelines. Restrict who can modify fine-tuning datasets and retrieval corpora, track changes, and validate new content.
- Verify packages. Check that suggested packages exist, are maintained and are the ones you intended, especially when suggested by AI assistants.
- Assess API providers like any critical vendor: security certifications, data handling, retention, and change notification.
The bottom line
AI applications inherit risk from every model, dataset and library they are built on, and some of those components can execute code or hide behaviour in ways traditional dependency scanning does not catch. Treat AI components as part of your software supply chain: inventory them, verify where they come from, pin them, sandbox them and test how they behave.
Our secure code review and application security testing services include AI supply-chain review, from model provenance to serving infrastructure.