Secure AI Engineering

ML Security vs Traditional Application Security

Application security teams are increasingly asked to review systems with machine learning at their core. Some of their existing knowledge transfers directly; some does not. We compare ML security with traditional application security, map ML attacks to familiar concepts and show where new skills are needed.

By EthicalHat Research · · 3 min read
Infographic: Traditional AppSec vs ML security

Security teams are increasingly asked to assess applications that have machine learning at their core: fraud models, recommendation engines, document classifiers and image recognition services. Many of those teams are experienced in application security but new to ML.

How much of what they know still applies? More than they might fear, but not all of it.

What stays the same

An ML-powered application is still an application. In our experience, most real-world vulnerabilities in ML systems are ordinary AppSec issues:

  • prediction APIs without authentication or rate limiting,
  • injection and deserialisation flaws in data processing code,
  • insecure storage of training data and model files,
  • over-privileged cloud credentials in training jobs,
  • exposed notebook servers and experiment dashboards, and
  • vulnerable dependencies in ML frameworks and data libraries.

Traditional techniques (threat modelling, code review, dependency scanning, penetration testing and cloud configuration review) find these problems. Any ML security review should start there.

What is different

ML systems have a property traditional software does not: their behaviour is learned from data, not written in code. That creates attack surfaces with no direct equivalent in classic AppSec.

ML attackWhat happensClosest AppSec analogy
Data poisoningManipulated training data changes model behaviourTampering with source code or configuration
Backdoors (trojans)Model behaves normally except on a triggerA hidden logic bomb
Evasion (adversarial examples)Crafted inputs cause wrong predictionsInput validation bypass
Model extractionQueries used to replicate the modelIntellectual property theft via an API
Membership inferenceDetermine if a record was in training dataInformation disclosure
Model inversionReconstruct sensitive attributes of training dataInformation disclosure

The analogies help AppSec teams reason about impact, but the mechanics differ.

Key differences in practice

Vulnerabilities are statistical

In traditional AppSec, a SQL injection either exists or it does not, and a patch removes it. In ML, a model is more or less robust to adversarial inputs; poisoning shifts behaviour by degrees. You measure and mitigate rather than patch to zero.

The "code" includes the data

A code review of the training pipeline tells you little about what the model learned. Assurance has to include data provenance (where it came from and who could change it) and behavioural evaluation (how the trained model actually behaves).

Testing is different

Traditional security tests are deterministic: send this payload, expect this response. ML robustness testing is statistical: generate many perturbed inputs, and measure how often predictions change. Tools and libraries for this exist, but they require ML skills to use well.

python
# Simple robustness probe: how often does small random noise flip the prediction?
import numpy as np

def flip_rate(model, x, epsilon=0.01, trials=200):
    base = model.predict(x[None])[0].argmax()
    flips = 0
    for _ in range(trials):
        noisy = np.clip(x + np.random.uniform(-epsilon, epsilon, x.shape), 0, 1)
        flips += model.predict(noisy[None])[0].argmax() != base
    return flips / trials   # random noise is a weak test; targeted attacks are far stronger

The attack surface includes outputs

In most applications, returning more detail in an API response is a minor information disclosure. For ML services, returning full probability scores rather than a label materially helps attackers extract models and craft evasion attacks.

Feedback loops are attack vectors

Systems that learn from user behaviour, reports or labels (spam filters, recommendation systems, fraud models with analyst feedback) can be manipulated by attackers who deliberately feed them misleading data.

Building ML security capability

AppSec teams do not need to become ML researchers, but they do need:

  • enough ML literacy to understand training, evaluation and deployment pipelines,
  • checklists that extend threat models with data, model and prediction threats,
  • partnerships with data science teams, who understand model behaviour, and
  • awareness of tools and research on adversarial robustness and privacy attacks.

A combined review approach

  1. Classic AppSec review of code, APIs, infrastructure and dependencies.
  2. Data review: provenance, access control, integrity checks and sensitive data handling.
  3. Model review: evaluation beyond accuracy, including robustness, bias and leakage.
  4. Deployment review: output minimisation, rate limits, monitoring and rollback.
  5. Feedback review: how the system learns after deployment, and who can influence it.

The bottom line

ML security is not a replacement for application security; it is an extension of it. Start with the fundamentals, which catch most real problems, and then add the ML-specific layer: data integrity, model robustness, output minimisation and protection of feedback loops. Teams that combine AppSec rigour with ML understanding are best placed to secure these systems.

Our application security testing and secure code review services cover ML-powered applications, from APIs and pipelines to data and model handling.

EthicalHat Research

The EthicalHat research team: penetration testers, SOC analysts and engineers writing about security, software engineering and AI.

Want an expert opinion on your own systems?

Email info@ethicalhat.com or call +1-844-838-4422. Our security engineers are happy to talk it through.