Adversarial Machine Learning

Adversarial machine learning is the study of attacks that manipulate AI systems, such as tricking a model with specially crafted inputs, poisoning its training data or extracting its secrets, and of defenses against them.

AdvancedCategory: AI and emerging threats

Also called: AML, adversarial AI, adversarial attacks, adversarial examples, evasion attack

What is adversarial machine learning?

Machine learning models learn patterns from data, and those patterns can be exploited. Adversarial machine learning (AML) is the field that studies how attackers manipulate AI systems and how to defend them. NIST's report on the subject, AI 100-2, provides a taxonomy and common terminology so organizations can assess and manage these risks.

MITRE's ATLAS knowledge base does something similar to MITRE ATT&CK for AI: it catalogs real-world attacker tactics and techniques against AI-enabled systems.

Main types of attacks

NIST groups attacks on traditional (predictive) AI systems into three broad classes:

  • Evasion: crafting inputs that fool a trained model into a wrong answer. Classic examples are tiny changes to an image that make a classifier mislabel it, stickers that confuse a self-driving car's sign recognition, or malware tweaked to slip past an AI-based detector. These crafted inputs are called adversarial examples.
  • Poisoning: corrupting the training data or the model itself so it learns the wrong thing or contains a hidden backdoor. See AI data poisoning.
  • Privacy attacks: extracting information the model shouldn't reveal, such as whether a specific person's data was used in training (membership inference), reconstructing training data, or copying the model itself. See AI model theft.

For generative AI, NIST also covers misuse and abuse attacks, including prompt injection, indirect prompt injection and jailbreaking, which push a model to ignore its rules or act against its user.

Why it matters for businesses

Security tools increasingly rely on machine learning: spam filters, malware detection, fraud scoring, identity verification and face recognition. Attackers actively probe these systems for blind spots. Businesses deploying AI that makes decisions, such as approving transactions, screening applicants or moderating content, need to assume someone will try to game it.

Defenses

NIST notes that many defenses involve trade-offs, and none is complete. Common measures include:

  1. Adversarial testing and AI red teaming before deployment
  2. Protecting training data and model pipelines from tampering
  3. Limiting model output detail and rate-limiting access to reduce extraction
  4. Input validation and monitoring for unusual patterns
  5. Layered controls, so no single AI decision is the only line of defense
  6. Human review for high-stakes decisions

Related terms

Sources

  1. AI 100-2 E2025: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST)
  2. MITRE ATLAS (MITRE)

Drafted with AI assistance, then edited and fact-checked by a person. How we use AI.