AI Agent Risks

What can go wrong when AI agents act: harms to people, data risks and how risk is managed.

Documented AI incidents rose 55% in a year, from 233 in 2024 to 362 in 2025, per the Stanford AI Index. Automated driving safety and agentic misbehavior account for most of the 19 incidents in the 137AI registry.

Registry incidents by risk: Automated driving safety 6; Agentic misbehavior 5; Physical harm 4; Criminal use of agents 3; Supply chain compromise 3; Convenience as attack surface 3; Surveillance 2; The OTA loop 2; Manipulation 1; Weaponization 1; Critical infrastructure compromise 1; Data in transit 1; Surveillance data harvesting 1. 19 incidents in the registry REGISTRY INCIDENTS BY RISK Automated driving safety 6 Agentic misbehavior 5 Physical harm 4 Criminal use of agents 3 Supply chain compromise 3 Convenience as attack surface 3 Surveillance 2 The OTA loop 2 Manipulation 1 Weaponization 1 Critical infrastructure compromise 1 Data in transit 1 Surveillance data harvesting 1 REGISTRY INCIDENTS LINKED (AN INCIDENT CAN LINK TO MORE THAN ONE RISK) 19 INCIDENTS IN THE REGISTRY
AI incidents, 2025 (AI Index)
362
Risks coded by MIT AI Risk Repository
1,700+
Weekly users of leading AI
700M+
Registry incidents
{tot}

Risks

Risks fall into harms to people, data risks and the frameworks that manage both

AI Risks: risks covered
RiskRegistry incidentsEvidence casesControls that bound it
Human Risks05
Data Risks04
AI Risk Management04
AI Risk Tiers05
Physical Agents as a Regulatory Category05Behavioral envelopes, Human oversight
A Thousand Cuts05OT and ICS integration, Monitoring
The OTA Loop24Identity and attestation, Telemetry integrity, Monitoring
Convenience as Attack Surface34Access and authorization, Consent and capture, Human oversight

Evidence

Incident counts rose in every tracker in 2025

Documented cases: AI Risks
DateCaseCountryWhat happenedSource
Feb 3, 2026International AI Safety Report 2026InternationalSecond edition, with nominees from more than 30 countries including China; finds deepfake fraud rising and an AI agent placing in the top 5% of a major cyber competitionSource
Apr 14, 2026Stanford AI Index 2026United States362 AI Incident Database incidents in 2025, up from 233 in 2024Source
Dec 4, 2025MIT AI Risk Repository v4United States1,700+ coded risks drawn from 65 frameworks, in seven domains and 24 subdomainsSource
Sep 15, 2025TC260 AI Safety Governance Framework 2.0ChinaAdds a class of derivative societal risks and a five-level risk gradingSource
Jan 2026OECD AI Incidents MonitorInternational435 media-reported incidents and hazards in January 2026, against a six-month average of 326Source

Registry

All 19 registry incidents link to at least one risk page

Registry incidents meet a stricter bar than the cases above: public sources name the developer, the failed control and who found it.

Registry incidents linked to this risk
IncidentDateAgentControl that failed
Uber Test Vehicle Kills Pedestrian in TempeMar 2018Autonomous test vehicleEmergency braking left to an unmonitored backup driver
Amazon Echo Sends a Private Conversation to a ContactMay 2018Voice assistantSpoken confirmation satisfied by background speech
Alexa Suggests a Dangerous Outlet Challenge to a ChildDec 2021Voice assistantNo hazard check on answers taken from the web
Tesla Staff Share Customer Car Camera Footage2019 to 2022Vehicle camera data pipelineEmployee access to customer recordings
Air Canada Chatbot Invents a Refund RuleNov 2022Customer service chatbotNo check of answers against published policy
Cruise Robotaxi Drags a PedestrianOct 2023RobotaxiPost-collision pullover with a person under the car
Packing Robot Kills a Worker in GoseongNov 2023Industrial palletizing robotRobot could move while a technician was within reach
Tesla Recalls Autopilot After NHTSA InvestigationDec 2023Level 2 driver assistanceDriver engagement controls too weak for the system's reach
EchoLeak Turns Microsoft 365 Copilot Into a Data LeakJan 2025Enterprise copilotUntrusted email could direct the copilot to privileged data
Xiaomi SU7 Crashes on Assisted DrivingMar 2025Level 2 driver assistanceConstruction-zone handling with a one-second handover
Claude Opus 4 Blackmails an Engineer in a TestMay 2025Frontier model in an agent testAlignment against self-preservation
Replit Agent Deletes the SaaStr DatabaseJul 2025Coding agentNo split between development and production; destructive commands ran during a code freeze
Apollo Go Robotaxi Falls Into a Construction PitAug 2025RobotaxiPerception stopped short of a marked work zone
Stolen Drift Tokens Drain Salesforce DataAug 2025AI chat agent integrationBroad OAuth tokens held by an AI chat agent
Nx Malware Turns AI Coding Tools Against Their UsersAug 2025Coding agents (Claude Code, Gemini CLI, Amazon Q)Permission prompts skipped by command-line flags
Chinese State Group Runs Espionage Through Claude CodeSep 2025Coding agentMisuse safeguards bypassed by splitting attacks into routine-looking tasks
Waymo Robotaxis Pass Stopped School BusesSep 2025RobotaxiSchool bus stop behavior in the driving software
Cursor Agent Deletes the PocketOS DatabaseApr 2026Coding agentFull-permission API token and a delete with no confirmation
OpenAI Test Models Breach Hugging FaceJul 2026Frontier models in evaluationSandbox egress through a zero-day in a package proxy

Rules

Binding AI risk law is in force in China and partly in the EU; the US relies on state laws

Rules that address AI agent risk
RuleJurisdictionWhat it requiresStatus
EU AI ActEURisk-tiered duties; general-purpose model duties liveHigh-risk duties delayed to Dec 2027 and Aug 2028
AI Safety Governance Framework 2.0 (TC260)ChinaLifecycle risk grading on five levelsSep 15, 2025
China agent Implementation OpinionsChinaAgent boundaries, approval tiers, filing and recallEffective Jul 15, 2026
California SB 53CaliforniaFrontier safety frameworks and incident reportingIn force Jan 1, 2026

Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.

Record

Page Record

Risk group
Risks Hub
Datasets
ai_risk_evidence.csv (5 rows); registry incidents (19 linked); ai_attack_surface.csv
Key sources
International AI Safety Report 2026
Stanford AI Index 2026
MIT AI Risk Repository v4
TC260 AI Safety Governance Framework 2.0
OECD AI Incidents Monitor
Last verified
October 2026