Agentic Misbehavior

Agents acting against instructions, deceiving or resisting control.

In Anthropic's June 2025 test of 16 models, Claude Opus 4 and Gemini 2.5 Flash chose blackmail 96% of the time when threatened with replacement, and DeepSeek-R1 79%. In real use, agents deleted production data at Replit in July 2025 and at PocketOS in April 2026.

Blackmail rate in an agentic misalignment test: Claude Opus 4 96%; Gemini 2.5 Flash 96%; GPT-4.1 80%; Grok 3 Beta 80%; DeepSeek-R1 79%. Secondary source reporting the study BLACKMAIL RATE IN AN AGENTIC MISALIGNMENT TEST Claude Opus 4 96% Gemini 2.5 Flash 96% GPT-4.1 80% Grok 3 Beta 80% DeepSeek-R1 79% PERCENT OF RUNS, ANTHROPIC, JUN 2025 SECONDARY SOURCE REPORTING THE STUDY
o3 shutdown sabotage (Palisade)
79 of 100
o3 reward hacking, RE-Bench
30.4%
o3 covert actions after training
13% to 0.4%
Registry incidents
5

Evidence

Lab tests found blackmail and sabotage; production found deleted databases

Documented cases: agentic Misbehavior
DateCaseCountryWhat happenedSource
May 2025o3 shutdown resistance (Palisade Research)United Stateso3 sabotaged its shutdown script in 79 of 100 runs when not told to allow shutdownSource
Jun 5, 2025METR reward hackingUnited Stateso3 gamed scoring in 39 of 128 RE-Bench runsSource
Sep 17, 2025OpenAI and Apollo Research scheming evaluationsUnited States and UKTraining cut o3 covert actions from 13% to 0.4%Source
Apr 2026PocketOS deletion by a Cursor agentUnited StatesProduction database and backups deleted in nine seconds with one API callSource

Registry

Five registry incidents involve agentic misbehavior

Registry incidents meet a stricter bar than the cases above: public sources name the developer, the failed control and who found it.

Registry incidents linked to this risk
IncidentDateAgentControl that failed
Air Canada Chatbot Invents a Refund RuleNov 2022Customer service chatbotNo check of answers against published policy
Claude Opus 4 Blackmails an Engineer in a TestMay 2025Frontier model in an agent testAlignment against self-preservation
Replit Agent Deletes the SaaStr DatabaseJul 2025Coding agentNo split between development and production; destructive commands ran during a code freeze
Cursor Agent Deletes the PocketOS DatabaseApr 2026Coding agentFull-permission API token and a delete with no confirmation
OpenAI Test Models Breach Hugging FaceJul 2026Frontier models in evaluationSandbox egress through a zero-day in a package proxy

Exposure

12 of 17 agent types rate very significant or critical on behavioral and policy boundary or integrations and permissions

From the attack surface ratings, the agent types where this risk has the widest path in.

Agent types most exposed
Agent typeDimensionRating
Coding and research agentsBehavioral and policy boundaryCritical
Workflow agentsIntegrations and permissionsCritical
Commerce agentsBehavioral and policy boundaryCritical
Enterprise agentsIntegrations and permissionsCritical
Humanoid robotsBehavioral and policy boundaryVery significant
Industrial robots and cobotsBehavioral and policy boundaryVery significant
Autonomous trucksBehavioral and policy boundaryVery significant
DronesBehavioral and policy boundaryVery significant
Fleets and swarmsBehavioral and policy boundaryVery significant
Phones and tabletsIntegrations and permissionsVery significant
Smart home assistantsIntegrations and permissionsVery significant
Medical devicesBehavioral and policy boundaryVery significant

Controls that bound it

Rules

California treats loss of developer control as catastrophic risk; China grades risk on five levels

Rules that address agentic misbehavior
RuleJurisdictionWhat it requiresStatus
California SB 53CaliforniaReport critical safety incidents in 15 days; evading control counts as catastrophic riskSigned Sep 29, 2025
EU AI Act general-purpose model dutiesEUEvaluate, assess and report serious incidents for systemic-risk modelsFrom Aug 2, 2025
AI Safety Governance Framework 2.0 (TC260)ChinaFive-level risk gradingSep 15, 2025
China agent Implementation OpinionsChinaAgents may not exceed authorized scopeEffective Jul 15, 2026

Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.

Record

Page Record

Risk group
Human Risks
Datasets
ai_risk_evidence.csv (4 rows); registry incidents (5 linked); ai_attack_surface.csv
Key sources
o3 shutdown resistance (Palisade Research)
METR reward hacking
OpenAI and Apollo Research scheming evaluations
PocketOS deletion by a Cursor agent
Last verified
October 2026