Red Teaming

Attacking AI systems on purpose to find failures before others do.

The UK AISI and Gray Swan challenge logged about 1.8 million attacks on 22 agent models in 2025, and 62,000 succeeded. A 2026 follow-up by US and UK evaluators broke every one of 13 frontier agents at least once.

Attack success in the UK AISI and Gray Swan challenge: Llama 3.3 70B 6.49%; GPT-4o 2.41%; Claude 3.7 Sonnet Thinking 1.47% ATTACK SUCCESS IN THE UK AISI AND GRAY SWAN CHALLENGE Llama 3.3 70B 6.49% GPT-4o 2.41% Claude 3.7 Sonnet Thinking 1.47% PERCENT, 2025
Attacks in the challenge
1.8M
Successful breaks
62,000
Jailbreak success without and with classifiers
86% vs 4.4%
Registry incidents
2

Evidence

Paid red teaming moved from conference events to standing bounties

Documented cases: red Teaming
DateCaseCountryWhat happenedSource
Aug 2024DEF CON 32 Generative Red Team 2United States495 participants filed 200 flaw reports against OLMoSource
Feb 2025Constitutional Classifiers public demoUnited States339 jailbreakers; four passed all eight levelsSource
Mar to Apr 2025UK AISI and Gray Swan challengeUK and United States$171,800 paid to 161 red teamersSource
Apr 23, 2026OpenAI GPT-5.5 bio bug bountyUnited States$25,000 for the first universal bio jailbreakSource
Nov 1, 2025GB/T 45654-2025ChinaTest bank of 2,000+ questions with at least 95% correct refusalsSource

Developers

Anthropic runs a standing model-safety bounty; most others pay only for security bugs

Frontier developers
DeveloperThird-party testingBug bounty
OpenAI United StatesYes UK AISI pre-deployment accessYes security bounty only
Anthropic United StatesYes UK AISI cyber evaluationYes model-safety bounty up to $25,000
Google DeepMind United StatesYes UK AISI, Apollo Research and othersPartial AI bounty excludes jailbreaks
Meta United StatesYes Apollo ResearchPartial security and privacy only
xAI United StatesUnknownYes HackerOne
Microsoft United StatesUnknownYes Copilot AI bounty
Amazon United StatesPartial independent evaluations publishedUnknown
Mistral FranceUnknownNo
DeepSeek ChinaNo CAISI evaluated after releaseNo
Alibaba (Qwen) ChinaUnknownUnknown
Zhipu AI (Z.ai) ChinaUnknownNo
Moonshot AI ChinaNo UK AISI assessed after releaseNo
ByteDance (Seed) ChinaUnknownUnknown

Tag colors: green yes, amber partial, red no, gray not found.

Registry

Two registry incidents bear on red teaming

Registry incidents where this discipline was tested in practice. Each links to its full record.

Registry incidents linked to this page
IncidentDateDeveloperRegulator action
EchoLeak Turns Microsoft 365 Copilot Into a Data LeakJan 2025MicrosoftNone reported.
Claude Opus 4 Blackmails an Engineer in a TestMay 2025AnthropicNone.

Rules

The EU code and China's standard both require adversarial testing

Rules and frameworks: red teaming
RuleJurisdictionWhat it requiresStatus
GPAI Code of Practice, safety chapterEURed teaming and external evaluation for systemic-risk modelsFrom Aug 2, 2025
GB/T 45654-2025ChinaQuestion-bank testing with refusal thresholds before filingIn force Nov 1, 2025
California SB 53CaliforniaDisclose whether third-party evaluators took partIn force Jan 1, 2026

Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.

Record

Page Record

Datasets
ai_trust_evidence.csv (5 rows); registry incidents (2 linked); developer scorecard (13 developers, static table)
Key sources
DEF CON 32 Generative Red Team 2
Constitutional Classifiers public demo
UK AISI and Gray Swan challenge
OpenAI GPT-5.5 bio bug bounty
GB/T 45654-2025
Last verified
October 2026