Adversarial Machine Learning: How Attackers Can Fool AI Security Systems

In 2014, researchers showed that adding carefully calculated, almost invisible noise to an image of a panda made a state-of-the-art image classifier label it, with high confidence, as a gibbon. That result launched a field of research: adversarial machine learning.

For years, adversarial examples were mostly an academic curiosity about images. Now machine learning sits inside spam filters, malware detection, fraud systems and network defences, and attackers have every incentive to fool it. Security is where adversarial ML stops being theoretical.

#What an adversarial example is

An adversarial example is an input deliberately modified so that a model makes a mistake, while the input still does what the attacker wants.

• For an image classifier, the change may be invisible to humans.
• For a malware classifier, the file must still run and do its malicious job, so the attacker changes features the model relies on without changing behaviour: adding benign-looking strings or imports, padding sections or repacking.
• For a phishing classifier, it may mean rephrasing text, swapping characters for lookalikes or hiding content in images.

Real-world examples have made headlines. In 2019, researchers bypassed an ML-based endpoint product by appending strings from a popular video game to malware. In 2020, researchers showed a car's camera-based speed-sign recognition could be tricked into reading 35 mph as 85 mph with a small strip of tape.

#Why models are fooled

ML models learn statistical patterns that separate classes in their training data. They do not "understand" what makes a file malicious. If a feature such as the presence of certain strings strongly correlates with benign files in training, the model will lean on it, and an attacker who adds that feature can push a malicious file across the decision boundary.

# Conceptual: a greedy black-box evasion loop against a detector the attacker can query.
def evade(sample, detector, benign_features, max_steps=200):
    candidate = sample
    for _ in range(max_steps):
        if detector.score(candidate) < detector.threshold:
            return candidate                       # now classified as benign
        # try each functionality-preserving change, keep the one that lowers the score most
        best = min(
            (apply(candidate, f) for f in benign_features),
            key=detector.score,
        )
        candidate = best
    return None

#What attackers need

• White-box attacks assume full knowledge of the model, and are the most effective.
• Black-box attacks only need to query the model and observe its output. Attackers can buy the product, use a public scanning service or test against a deployed filter.
• Transfer attacks craft adversarial examples against a different model the attacker trained themselves. Examples often transfer between models trained on similar data.

Security models are unusually exposed on all three counts: attackers control the inputs, can often test repeatedly, and get clear feedback (blocked or not).

#Beyond evasion

• Poisoning: corrupting training data or feedback so the model learns the wrong thing, which is a risk for systems that learn from user reports or shared sample feeds.
• Backdoors: training a model to behave normally except when a specific trigger is present.
• Model extraction: querying a model enough to build a functional copy, which then makes white-box attacks possible.
• Membership inference: learning whether particular data was in the training set, a privacy risk.

Microsoft and MITRE's Adversarial ML Threat Matrix, published last October, catalogues these techniques in an ATT&CK-style framework, and is a useful starting point for threat modelling ML systems.

#How defenders can respond

Layer your detection. Never let a single ML model be the only barrier. Combine static ML with signatures, behavioural detection and anomaly detection, so evading one does not mean evading all.

Prefer robust features. Features that are hard to change without breaking functionality, such as behaviour at runtime, are harder to manipulate than features an attacker can freely add, like strings and padding.

Train adversarially. Include adversarial and evasive examples in training data, and retrain as new evasion techniques appear.

Limit feedback. Avoid exposing detailed scores to untrusted parties, rate-limit scanning interfaces and monitor for probing patterns (many near-identical submissions).

Protect the pipeline. Validate training data sources, review labels from external feeds, and control who can change models.

Test like an attacker. Include adversarial ML in red team exercises against your ML-based controls. Community competitions, such as the machine learning security evasion contests run in recent years, show how quickly skilled attackers find weaknesses.

#The bottom line

Machine learning makes security controls smarter, and it also gives attackers a new target. Any ML model that stands between an attacker and their goal will eventually be probed, and many can be fooled. Treat models as part of your attack surface: layer them, harden them and test them.

Our red teaming includes testing of ML-based security controls, and our endpoint security service layers ML with behavioural and signature-based detection.