Human-in-the-Loop Oversight

Human review as a legal requirement, and whether it works.

In a randomized trial, physicians made 14 percentage points fewer correct diagnoses when ChatGPT-4o gave flawed advice in half the cases. The rules keep spreading anyway: China requires approval for significant agent actions, and Colorado grants a right to human review from January 2027.

Physician accuracy with correct and flawed AI advice: Top diagnosis, correct advice 90.5%; Reasoning, correct advice 84.9%; Top diagnosis, flawed advice 76.1%; Reasoning, flawed advice 73.3%. Preprint, Aug 2025 PHYSICIAN ACCURACY WITH CORRECT AND FLAWED AI ADVICE Top diagnosis, correct advice 90.5% Reasoning, correct advice 84.9% Top diagnosis, flawed advice 76.1% Reasoning, flawed advice 73.3% PERCENT PREPRINT, AUG 2025
Accuracy drop from flawed advice
14 points
NYC violations: city vs state
1 vs 17
Colorado explanation deadline
30 days
Registry incidents
4

Evidence

Courts and regulators require human review; studies show reviewers follow bad advice

Developments: human-in-the-Loop Oversight
DateDevelopmentCountryWhat happenedSource
Feb 2025Dun & Bradstreet Austria (C-203/22)EUPeople subject to automated decisions are owed an explanationSource
Aug 2025Physician automation bias trialInternational44 physicians; accuracy fell with flawed AI adviceSource
Jan 6, 2026FDA revised CDS guidanceUnited StatesClinicians must be able to review the basis of a recommendationSource
May 8, 2026China agent Implementation OpinionsChinaApproval before significant actions; hard-to-reverse ones not automatedSource
May 2026Colorado SB 26-189United StatesData correction, human review and reconsideration rightsSource

Registry

Four registry incidents bear on human review requirements

Registry incidents linked to this page
IncidentDateRegulator or court action
Uber Test Vehicle Kills Pedestrian in TempeMar 2018Arizona suspended Uber's testing on March 26, 2018. Yavapai County declined to charge Uber in March 2019. NTSB adopted its findings on November 19, 2019. The backup operator pleaded guilty to endangerment in July 2023 and received three years of supervised probation.
Tesla Recalls Autopilot After NHTSA InvestigationDec 2023NHTSA opened PE21-020 in August 2021, upgraded it to EA22-002 in June 2022 and closed it in April 2024 after reviewing 956 crashes, opening RQ24-009 on whether the fix worked. No penalty.
Replit Agent Deletes the SaaStr DatabaseJul 2025None reported.
Cursor Agent Deletes the PocketOS DatabaseApr 2026None reported.

Rules

Human review is required in the EU, China, Korea and Colorado

Key rules and standards: human review requirements
RuleJurisdictionWhat it requiresStatus
EU AI Act Articles 14 and 26EUOversight by design; trained deployer staff with authorityDec 2027
GDPR Article 22EURight to human intervention in automated decisionsIn force
China agent Implementation OpinionsChinaApproval tiers; users keep final authorityEffective Jul 15, 2026
Colorado SB 26-189ColoradoMeaningful human review and reconsiderationEffective Jan 2027
Korea AI Basic ActSouth KoreaHuman intervention for high-impact AIEffective Jan 22, 2026

Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.

Record

Page Record

Datasets
ai_compliance_evidence.csv (5 rows); registry incidents (4 linked)
Key sources
Dun & Bradstreet Austria (C-203/22)
Physician automation bias trial
FDA revised CDS guidance
China agent Implementation Opinions
Colorado SB 26-189
Last verified
October 2026