Evidence
Drone war in Ukraine and AI targeting in Gaza set the precedents
| Date | Case | Country | What happened | Source |
|---|---|---|---|---|
| Apr 3, 2024 | Lavender targeting system | Israel | Reported to have flagged about 37,000 people with about 20 seconds of human review each; the IDF denies | Source |
| Apr 2025 | Virology Capabilities Test | United States | OpenAI o3 scored 43.8% against 22.1% for expert virologists | Source |
| Jul 2025 | Record Shahed month | Russia and Ukraine | 6,297 drones launched, about 203 a day | Source |
| Undated | PLA 200-drone swarm | China | State TV showed one soldier controlling more than 200 drones | Source |
Registry
One registry incident involves weaponization
Registry incidents meet a stricter bar than the cases above: public sources name the developer, the failed control and who found it.
| Incident | Date | Agent | Control that failed |
|---|---|---|---|
| Chinese State Group Runs Espionage Through Claude Code | Sep 2025 | Coding agent | Misuse safeguards bypassed by splitting attacks into routine-looking tasks |
Exposure
10 of 17 agent types rate very significant or critical on command and control channels or multi-agent coordination
From the attack surface ratings, the agent types where this risk has the widest path in.
| Agent type | Dimension | Rating |
|---|---|---|
| Fleets and swarms | Multi-agent coordination | Critical |
| Workflow agents | Multi-agent coordination | Critical |
| Robotaxis and autonomous vehicles | Command and control channels | Very significant |
| Humanoid robots | Command and control channels | Very significant |
| Autonomous trucks | Multi-agent coordination | Very significant for platoons |
| Drones | Command and control channels | Very significant |
| Smart home assistants | Command and control channels | Very significant |
| Coding and research agents | Command and control channels | Very significant |
| Commerce agents | Command and control channels | Very significant |
| Enterprise agents | Command and control channels | Very significant |
Controls that bound it
- Behavioral Envelopes: hard limits on action enforced outside the model
- Human Oversight: human authority to approve, intervene and stop
- Monitoring and Anomaly Detection: seeing failures and misuse while they happen
Rules
Weapons autonomy has UN resolutions and national directives, not a treaty
| Rule | Jurisdiction | What it requires | Status |
|---|---|---|---|
| UNGA resolution A/RES/80/57 | UN | Asks CCW states to finish elements of an instrument on autonomous weapons | First Committee vote Nov 6, 2025 |
| CCW Group of Governmental Experts | UN | Negotiating a rolling text | No agreement as of 2026 |
| DoD Directive 3000.09 | US | Appropriate human judgment over autonomous weapons | Rewrite ordered Jun 2026 |
| California SB 53 | California | Frontier developers report incidents; CBRN help counts as catastrophic risk | Signed Sep 29, 2025 |
Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.
Record
Page Record
- Risk group
- Human Risks
- Datasets
- ai_risk_evidence.csv (4 rows); registry incidents (1 linked); ai_attack_surface.csv
- Key sources
- Lavender targeting system
Virology Capabilities Test
Record Shahed month
PLA 200-drone swarm - Last verified
- October 2026