Adversarial Machine Learning Attacks

Adversarial machine learning studies how AI systems can be deceived, manipulated, or turned against their intended purpose. Classic examples include:

• Evasion attacks: carefully crafted inputs cause a model to misclassify what it sees.
• Data poisoning: tampered training data plants hidden behaviors.
• Model extraction: attackers reconstruct a proprietary model by repeatedly probing it.

With the rise of large language models, the field now also covers jailbreaks that bypass safety guardrails. It also covers prompt injection, where malicious instructions hidden in documents, web pages, or tool outputs hijack a model's behavior.

Why it matters now

The stakes have shifted as AI moves from answering questions to taking actions. Autonomous agents now browse the web, call APIs, write code, and operate with real credentials inside corporate systems. That autonomy can turn a manipulated model into something like an insider threat. It also raises hard questions about identity, permissions, and how much human oversight is enough.

At the same time, attackers are using AI to speed up vulnerability discovery and exploit development. Defenders, including government and military organizations, are deploying AI agents of their own, which adds new attack surfaces even as it strengthens defenses.

What you'll find here

This hub tracks research and real-world incidents across the adversarial landscape, including:

• New jailbreak and prompt injection techniques, and research into why safety measures fail
• Security gaps in AI agents, from unmonitored deployments to sandbox and containment weaknesses
• AI-assisted offensive security and exploit development
• Governance frameworks, oversight recommendations, and practical controls for securing AI systems
• Red-teaming results and evaluations from labs, academics, and industry groups