Machine learning has moved quickly from research projects into production systems that approve loans, filter content, detect fraud, recommend products and increasingly help make safety-critical decisions. In many organisations, those systems were built by data science teams working outside the normal software development process, with no threat modelling, no security review and no plan for how the model might be attacked.
That was acceptable for experiments. It is not acceptable for production.
Why ML needs its own security thinking
An ML system is still software. It has code, dependencies, infrastructure, APIs and data stores, all with the usual vulnerabilities. But it also has new assets and new attack surfaces:
- Training data that determines behaviour, and can be poisoned or leaked.
- Models that encode valuable intellectual property, and can be stolen or reverse-engineered.
- Predictions that can be manipulated with carefully crafted inputs.
- Feedback loops where the system learns from its own users, and from attackers.
Research over the last few years has demonstrated attacks on each:
- Adversarial examples: inputs subtly modified to cause misclassification. In February, researchers showed a car's camera-based system could be made to misread a speed-limit sign with a small strip of tape.
- Data poisoning and backdoors: research such as BadNets (2017) showed models can be trained to behave normally except when a hidden trigger is present.
- Model extraction: querying a prediction API enough to build a functional copy of the model.
- Membership inference and model inversion: learning whether specific records were in the training data, or reconstructing sensitive features, from model outputs.
A security lifecycle for ML
Established secure development lifecycles, such as Microsoft's SDL, have a proven shape: requirements, design, implementation, verification, release and response. Adapting that shape to ML gives a practical starting point.
flowchart LR R[Requirements<br/>what harm is possible?] --> D[Data<br/>provenance, access, quality] D --> T[Training<br/>secure pipeline, reproducibility] T --> E[Evaluation<br/>accuracy + robustness + misuse] E --> Dep[Deployment<br/>hardened serving, API limits] Dep --> M[Monitoring<br/>drift, abuse, extraction] M -->|incidents, new data| R
1. Requirements and threat modelling
Ask the usual questions, plus ML-specific ones:
- What decisions does the model influence, and what is the harm if it is wrong or manipulated?
- Who can influence the training data or feedback?
- Who can query the model, how often, and what do they see (labels, scores, probabilities)?
- What would a competitor or attacker gain by copying the model?
- Does the training data include personal or sensitive information?
2. Data
- Record the provenance of every dataset: where it came from, who can change it, and its licence.
- Apply access control to training data as strictly as to production data.
- Validate new data for anomalies and label distribution shifts that might indicate poisoning.
- Minimise sensitive data, and consider privacy-preserving techniques where appropriate.
3. Training
- Treat training code as production code: version control, code review, dependency scanning.
- Run training in isolated environments with least-privilege access to data and credentials.
- Make training reproducible: record code versions, data versions, parameters and environment.
4. Evaluation
Accuracy on a held-out test set is not enough. Also evaluate:
- robustness to noise and adversarial perturbations relevant to the domain,
- performance on important subgroups, to catch bias,
- behaviour on out-of-distribution inputs, and
- whether outputs leak information about training data.
5. Deployment
- Serve models behind authenticated APIs with rate limiting, which makes extraction attacks slower and more visible.
- Return the minimum information needed: a label rather than full probability vectors, where possible.
- Harden the serving infrastructure like any other production service.
- Protect model files at rest and in transit, and control who can deploy new versions.
6. Monitoring and response
- Monitor data and prediction drift, which can signal changing conditions or manipulation.
- Watch for query patterns that suggest extraction or probing.
- Have a plan to roll back a model, retrain on cleaned data, or take a feature offline.
Who owns it
ML security sits between data science, engineering and security teams, which is why it often falls through the gaps. Effective organisations:
- involve security in ML design reviews for high-impact models,
- give data scientists training on ML-specific threats, and
- bring ML projects into the same secure development and change processes as other software.
The bottom line
Machine learning systems are software with extra attack surfaces. Protecting them does not require inventing security from scratch: it requires extending the secure development lifecycle to cover data, training, models and predictions, and making someone responsible for it. The time to do that is before ML systems make high-stakes decisions, not after.
Our application security testing and risk assessment services help organisations threat-model and secure ML systems alongside their other applications.
