AI features are now assembled from many external components. A single product might call two commercial model APIs, run an open-weight model for a sensitive workload, use an embedding service and a hosted vector database, and depend on dozens of fast-moving ML packages. Each of those is part of your AI supply chain, and each can fail, change or be compromised.
This year has provided plenty of reminders:
- In March, researchers reported attacks on exposed Ray clusters, a popular framework for distributed AI workloads, used to steal credentials and computing resources.
- In May and June, Hugging Face disclosed unauthorised access to its Spaces platform, where some users' secrets may have been exposed, and asked users to rotate tokens.
- Researchers continued to find malicious models on public hubs using unsafe serialisation formats.
- In April, CISA, the NSA and partners published joint guidance on deploying AI systems securely, much of which concerns the environments and components around models.
Link 1: model APIs
Commercial model APIs process your prompts, which may include customer data, code and confidential documents. Treat providers as critical data processors:
- assess them through your vendor risk process: security certifications, data retention, training on customer data, sub-processors, data residency and breach notification,
- use enterprise agreements with appropriate data terms,
- route all usage through an AI gateway so you know which applications send what to which provider,
- pin model versions where possible, and re-evaluate when providers change models, and
- plan for outages: fallbacks or graceful degradation.
Link 2: open-weight models
Open-weight models, such as the Llama 3.1 family released in July, are increasingly used for data-sensitive or cost-sensitive workloads. They bring different risks:
- Provenance: download from the official publisher, verify checksums, and beware of lookalike repositories and fine-tunes of unknown origin.
- Format: prefer safetensors; refuse pickle-based files from untrusted sources.
- Behaviour: evaluate on your own tasks, including safety and security cases, before use. Fine-tuned derivatives may have had safety training removed or altered.
- Licences: open-weight licences vary and may restrict certain uses.
Link 3: ML packages and frameworks
The ML ecosystem moves quickly and has had its share of vulnerable and malicious packages:
- pin and hash dependencies, and install through an internal proxy,
- scan continuously, and prioritise patches for internet-exposed components,
- vet new packages, including those suggested by AI assistants, which sometimes recommend packages that do not exist, and
- keep ML infrastructure (orchestration, experiment tracking, serving) patched and never exposed without authentication.
Link 4: datasets
- Record provenance and licences for training, fine-tuning and evaluation data.
- Snapshot and checksum datasets; do not fetch mutable data at training time.
- Control write access to data used for fine-tuning and retrieval.
Link 5: AI platforms and hubs
Model hubs, hosted notebooks, managed vector databases and AI development platforms often hold tokens and secrets that connect to the rest of your environment.
- Use scoped, short-lived tokens where platforms allow.
- Keep an inventory of which platforms hold which credentials, so you can rotate quickly after a platform incident, as many had to this year.
- Apply SSO and MFA to platform accounts.
flowchart LR I[AI inventory<br/>models, APIs, packages,<br/>datasets, platforms, credentials] --> A[Assess<br/>vendor risk, provenance,<br/>licences] A --> C[Control<br/>gateway, proxy, pinning,<br/>safe formats] C --> E[Evaluate<br/>quality, safety, security<br/>before use & on change] E --> M[Monitor<br/>advisories, provider changes,<br/>platform incidents] M -->|incident| R[Respond<br/>rotate, roll back, switch provider] R --> I
An AI inventory: the foundation
You cannot manage supply-chain risk without knowing what you depend on. For each AI feature, record:
| Component | Examples of what to record |
|---|---|
| Model APIs | Provider, model and version, data sent, contract terms |
| Open-weight models | Source, version, checksum, format, licence, evaluation results |
| Packages | Lock files and SBOMs for training and serving environments |
| Datasets | Source, version, licence, sensitivity |
| Platforms | Accounts, tokens held, owners |
When the next platform breach or malicious model is announced, this inventory turns "are we affected?" into a query.
The bottom line
The AI supply chain is broader than most organisations have mapped: commercial APIs, open-weight models, ML packages, datasets and platforms all carry risk. Manage it with an AI inventory, vendor risk assessment for providers, provenance and safe formats for models, disciplined package management, protected datasets and a ready plan for rotation and rollback when a link in the chain is compromised.
Our application security testing and information security audit services include AI supply-chain and third-party AI risk reviews.