Secure AI Engineering

Securing ML Models, Training Data and Deployment Pipelines

MITRE's new ATLAS knowledge base catalogues real attacks on machine learning systems, and Microsoft has open-sourced Counterfit for testing them. It is a good moment for a practical guide to protecting the three assets that make an ML system: its training data, its models and the pipelines that connect them.

By EthicalHat Research · · 3 min read
Infographic: Protecting the assets of an ML system

Machine learning security has long suffered from a gap between research and practice. Academic papers describe adversarial examples, poisoning and model extraction in detail, but practitioners have had little guidance on which attacks matter and how to defend against them.

That is starting to change. In June, MITRE launched ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems), a knowledge base of adversary tactics and techniques against ML systems, modelled on ATT&CK and grounded in real-world case studies. In May, Microsoft open-sourced Counterfit, a tool for assessing the security of ML models. Together they give security teams a vocabulary and tooling.

This article focuses on the three assets every ML system has to protect.

Asset 1: training data

Training data determines what a model learns. It needs three kinds of protection.

Integrity: preventing poisoning. If attackers can influence training data, they can influence the model, degrading accuracy, biasing decisions or implanting backdoors.

  • Restrict who and what can write to training datasets, and log changes.
  • Validate new data for anomalies in distributions and labels.
  • Be especially careful with data from users, feedback loops and scraped sources.
  • Keep versioned snapshots so you can retrain from a known-good state.

Confidentiality: preventing leakage. Training datasets often aggregate sensitive information: customer records, transactions, medical data.

  • Apply strict access control and encryption.
  • Minimise and pseudonymise where possible.
  • Remember that models can leak training data through membership inference and inversion attacks. Where the data is highly sensitive, techniques such as differential privacy can limit what a model reveals.

Provenance: knowing what you trained on. Record sources, versions, licences and processing steps, so models can be reproduced and audited.

Asset 2: models

Trained models are valuable intellectual property, and a route to attacking the system.

Theft of model files. Protect model artefacts in registries and storage with access control, encryption and audit logging. Treat them like source code or signing keys.

Extraction through the API. Attackers can approximate a model by querying it many times.

  • Require authentication and apply per-client rate limits and quotas.
  • Return labels rather than full probability vectors where the use case allows.
  • Monitor for high-volume, systematic query patterns.

Tampering. A modified model file can behave differently, or contain malicious code if serialised with unsafe formats like pickle.

  • Record checksums or signatures at registration, and verify them when loading.
  • Avoid loading model files from untrusted sources with unsafe deserialisation.

Evasion. Adversarial inputs can cause targeted misclassification.

  • Evaluate robustness for the threats relevant to your domain.
  • Use adversarial training or input preprocessing where evasion is a realistic threat.
  • Do not rely on a single model as the only control for high-stakes decisions.

Asset 3: pipelines

Training and deployment pipelines decide which data, code and models reach production. Whoever controls them controls the model.

flowchart LR
  D[(Data store)] -->|read-only, scoped| T[Training job<br/>isolated, pinned image]
  C[Code repo<br/>reviewed] --> T
  T -->|write-only| R[(Model registry<br/>checksums, metadata)]
  R -->|approved promotion| S[Serving<br/>verifies checksum]
  S --> API[Authenticated,<br/>rate-limited API]
  API --> Mon[Monitoring<br/>drift, abuse]
  • Least privilege for every pipeline stage: training jobs read only the data they need and write only to their output location.
  • Reviewed code and pinned dependencies for pipeline steps.
  • Controlled promotion: only approved models move to production, through an audited process.
  • Isolated infrastructure: no internet-exposed notebook servers, orchestration dashboards or training clusters.
  • Monitoring and logging of pipeline runs, model registrations and deployments.

Using ATLAS for threat modelling

ATLAS organises ML attacks into tactics that will look familiar to anyone who knows ATT&CK: reconnaissance, resource development, initial access, ML model access, execution, persistence, defence evasion, discovery, collection, ML attack staging, exfiltration and impact. Its case studies describe real incidents and red-team exercises against production ML systems.

A practical way to use it: for each ML system, walk through the tactics and ask which techniques are feasible given how the system is built and exposed, and which controls above would detect or prevent them.

Testing

Tools such as Counterfit and the Adversarial Robustness Toolbox let security teams run evasion, extraction and inference attacks against models in a controlled way. Combine them with conventional penetration testing of the surrounding application, APIs and infrastructure, where most real-world weaknesses still sit.

The bottom line

Securing an ML system means protecting its data, its models and its pipelines, each with controls for integrity, confidentiality and availability. The new ATLAS knowledge base and open-source tools such as Counterfit make it far easier for security teams to threat-model and test these systems with the same rigour as any other critical application.

Our application security testing and red teaming services include ML-specific threat modelling and testing aligned to MITRE ATLAS.

EthicalHat Research

The EthicalHat research team: penetration testers, SOC analysts and engineers writing about security, software engineering and AI.

Want an expert opinion on your own systems?

Email info@ethicalhat.com or call +1-844-838-4422. Our security engineers are happy to talk it through.