Security teams are increasingly asked to assess applications that have machine learning at their core: fraud models, recommendation engines, document classifiers and image recognition services. Many of those teams are experienced in application security but new to ML.
How much of what they know still applies? More than they might fear, but not all of it.
What stays the same
An ML-powered application is still an application. In our experience, most real-world vulnerabilities in ML systems are ordinary AppSec issues:
- prediction APIs without authentication or rate limiting,
- injection and deserialisation flaws in data processing code,
- insecure storage of training data and model files,
- over-privileged cloud credentials in training jobs,
- exposed notebook servers and experiment dashboards, and
- vulnerable dependencies in ML frameworks and data libraries.
Traditional techniques (threat modelling, code review, dependency scanning, penetration testing and cloud configuration review) find these problems. Any ML security review should start there.
What is different
ML systems have a property traditional software does not: their behaviour is learned from data, not written in code. That creates attack surfaces with no direct equivalent in classic AppSec.
| ML attack | What happens | Closest AppSec analogy |
|---|---|---|
| Data poisoning | Manipulated training data changes model behaviour | Tampering with source code or configuration |
| Backdoors (trojans) | Model behaves normally except on a trigger | A hidden logic bomb |
| Evasion (adversarial examples) | Crafted inputs cause wrong predictions | Input validation bypass |
| Model extraction | Queries used to replicate the model | Intellectual property theft via an API |
| Membership inference | Determine if a record was in training data | Information disclosure |
| Model inversion | Reconstruct sensitive attributes of training data | Information disclosure |
The analogies help AppSec teams reason about impact, but the mechanics differ.
Key differences in practice
Vulnerabilities are statistical
In traditional AppSec, a SQL injection either exists or it does not, and a patch removes it. In ML, a model is more or less robust to adversarial inputs; poisoning shifts behaviour by degrees. You measure and mitigate rather than patch to zero.
The "code" includes the data
A code review of the training pipeline tells you little about what the model learned. Assurance has to include data provenance (where it came from and who could change it) and behavioural evaluation (how the trained model actually behaves).
Testing is different
Traditional security tests are deterministic: send this payload, expect this response. ML robustness testing is statistical: generate many perturbed inputs, and measure how often predictions change. Tools and libraries for this exist, but they require ML skills to use well.
# Simple robustness probe: how often does small random noise flip the prediction?
import numpy as np
def flip_rate(model, x, epsilon=0.01, trials=200):
base = model.predict(x[None])[0].argmax()
flips = 0
for _ in range(trials):
noisy = np.clip(x + np.random.uniform(-epsilon, epsilon, x.shape), 0, 1)
flips += model.predict(noisy[None])[0].argmax() != base
return flips / trials # random noise is a weak test; targeted attacks are far strongerThe attack surface includes outputs
In most applications, returning more detail in an API response is a minor information disclosure. For ML services, returning full probability scores rather than a label materially helps attackers extract models and craft evasion attacks.
Feedback loops are attack vectors
Systems that learn from user behaviour, reports or labels (spam filters, recommendation systems, fraud models with analyst feedback) can be manipulated by attackers who deliberately feed them misleading data.
Building ML security capability
AppSec teams do not need to become ML researchers, but they do need:
- enough ML literacy to understand training, evaluation and deployment pipelines,
- checklists that extend threat models with data, model and prediction threats,
- partnerships with data science teams, who understand model behaviour, and
- awareness of tools and research on adversarial robustness and privacy attacks.
A combined review approach
- Classic AppSec review of code, APIs, infrastructure and dependencies.
- Data review: provenance, access control, integrity checks and sensitive data handling.
- Model review: evaluation beyond accuracy, including robustness, bias and leakage.
- Deployment review: output minimisation, rate limits, monitoring and rollback.
- Feedback review: how the system learns after deployment, and who can influence it.
The bottom line
ML security is not a replacement for application security; it is an extension of it. Start with the fundamentals, which catch most real problems, and then add the ML-specific layer: data integrity, model robustness, output minimisation and protection of feedback loops. Teams that combine AppSec rigour with ML understanding are best placed to secure these systems.
Our application security testing and secure code review services cover ML-powered applications, from APIs and pipelines to data and model handling.