Registry
11 failed oversight controls across 10 registry incidents
Each row is a control that was in place, or should have been, and did not hold. The incident record explains who was responsible and what changed afterward.
| Incident | Date | Control | Responsible | What failed |
|---|---|---|---|---|
| Uber Test Vehicle Kills Pedestrian in Tempe | Mar 2018 | Operator oversight | Uber ATG | The company had no countermeasure for a backup operator watching her phone |
| Amazon Echo Sends a Private Conversation to a Contact | May 2018 | Confirmation before sending | Amazon | Background speech was taken as both the contact name and the confirmation |
| Air Canada Chatbot Invents a Refund Rule | Nov 2022 | Answer accuracy against policy | Air Canada | The chatbot contradicted the airline's published bereavement policy |
| Air Canada Chatbot Invents a Refund Rule | Nov 2022 | Accountability for automated statements | Air Canada | The airline tried to disown its chatbot's statement |
| Cruise Robotaxi Drags a Pedestrian | Oct 2023 | Regulatory disclosure | Cruise | Crash reports left out the drag after Cruise had shown NHTSA the video |
| Packing Robot Kills a Worker in Goseong | Nov 2023 | Isolation during maintenance | Plant and installer | The robot could move while a technician was within reach before a test run |
| Tesla Recalls Autopilot After NHTSA Investigation | Dec 2023 | Driver engagement monitoring | Tesla | Controls did not keep drivers attentive and did not match how easily Autosteer could be engaged |
| Xiaomi SU7 Crashes on Assisted Driving | Mar 2025 | Driver handover time | Xiaomi | The warning came about one second before the driver took over, too late to avoid impact |
| Replit Agent Deletes the SaaStr Database | Jul 2025 | Code-freeze enforcement | Replit | The agent ignored an instruction to make no changes without approval |
| Apollo Go Robotaxi Falls Into a Construction Pit | Aug 2025 | Passenger recovery | Baidu | Residents, not the operator, got the passenger out |
| Cursor Agent Deletes the PocketOS Database | Apr 2026 | Approval for destructive actions | Cursor | The agent deleted without asking |
Implementations
Robotaxi operators use remote advisors; China writes approval tiers into law
Waymo's fleet response agents, about half of them in the Philippines, answer vehicle questions without driving. Tesla follows unsupervised Austin robotaxis with chase cars. Beijing's 2025 regulations require a monitoring platform and a safety officer who takes over on faults. California now licenses remote operators. None of seven companies Senator Markey asked would say how often remote staff intervene.
| Implementation | Vendor or body | Country | What it enforces | Status |
|---|---|---|---|---|
| Chase-car supervision | Tesla | United States | Trailing vehicles monitor unsupervised robotaxis in Austin | From Jan 22, 2026 |
| Fleet response and Emergency Response Team | Waymo | United States | About 70 remote agents on duty for about 3,000 vehicles; advise, do not drive | Disclosed Mar 2026 |
| Monitoring platform and manual takeover duty | Beijing municipal government | China | Real-time monitoring; safety officer takes over on fault or leaving the permitted area | Effective Apr 1, 2025 |
| Remote operations personnel licensing | California DMV | United States | Licensing and training for remote drivers and assistants | Adopted Apr 28, 2026 |
| Three-tier decision authority | CAC, NDRC and MIIT | China | Routine actions automated; significant ones need approval; hard-to-reverse ones not automated | Effective Jul 15, 2026 |
Row color shows the agent class served: orange physical, teal personal and ambient, blue software.
Recent events
- Waymo's chief safety officer confirmed overseas guidance agents at a Senate Commerce hearing on February 4, 2026.
- Uber's Tempe backup driver pleaded guilty to endangerment on July 28, 2023; Uber was not charged.
- Cruise's often-quoted 1.5 workers per vehicle counted all staff; its remote assistance ratio was one per 15 to 20 vehicles.
- A Lancet study found endoscopists' detection rate without AI fell from 28.4% to 22.4% after routine AI use.
Exposure
Command and control channels rates very significant or critical for 10 of 17 agent types
The human oversight discipline covers the command and control channels dimension of the agent attack surface ratings. These agent types depend on it most.
| Agent type | Dimension | Rating | Why |
|---|---|---|---|
| Robotaxis and autonomous vehicles | Command and control channels | Very significant | Fleet dispatch APIs, remote operations channels, in-vehicle command paths, V2X interfaces |
| Humanoid robots | Command and control channels | Very significant | Teleoperation interfaces, fleet management APIs, voice and natural language command channels, and on-device control systems |
| Autonomous trucks | Command and control channels | Very significant | Fleet management dispatch, remote operations channels, V2V protocols for platoons |
| Drones | Command and control channels | Very significant | Radio control links can be jammed, hijacked or spoofed |
| Fleets and swarms | Command and control channels | Very significant | Fleet management APIs, remote operations channels, dispatch interfaces all carry the authority to direct many agents at once |
| Smart home assistants | Command and control channels | Very significant | Voice command channels are particularly exposed |
| Coding and research agents | Command and control channels | Very significant | The prompt is the command |
| Workflow agents | Command and control channels | Very significant | Goal specification through prompts |
| Commerce agents | Command and control channels | Very significant | The prompt is the command |
| Enterprise agents | Command and control channels | Very significant | Instructions arrive through chat, email, tickets and other agents |
Rules
The EU and China require oversight by design; US rules cover vehicles and weapons
| Rule | Jurisdiction | What it requires | Status |
|---|---|---|---|
| EU AI Act Article 14 | EU | High-risk systems designed so people can monitor, override and stop them | Delayed to Dec 2027 |
| China agent Implementation Opinions | China | Users keep final authority; approval for significant actions; no automation of hard-to-reverse ones | Effective Jul 15, 2026 |
| Beijing Autonomous Vehicle Regulations | Beijing, China | Monitoring platform; takeover duty on faults | Effective Apr 1, 2025 |
| California DMV AV regulations | California | Remote operator licensing and training | Adopted Apr 28, 2026 |
| DoD Directive 3000.09 | US military | Appropriate levels of human judgment over use of force | Updated Jan 25, 2023 |
Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.
Limits
A human in the loop fails when the job is mostly watching
Uber's backup driver was looking at a phone, and Tesla drivers stopped watching the road. Oversight works when the system hands humans decisions they can make in time, not hours of monitoring.
Record
Page Record
- Datasets
- ai_control_implementations.csv (5 rows for this page); ai_incident_controls.csv (11 linked); ai_attack_surface.csv
- Key sources
- Beijing municipal government
CAC, NDRC and MIIT
California DMV
Tesla
Waymo - Source notes
- Chase-car supervision: secondary source; Fleet response and Emergency Response Team: secondary source; Three-tier decision authority: secondary source
- Last verified
- October 2026