Evidence
Lawsuits, not regulators, forced the first companion chatbot changes in the US
| Date | Case | Country | What happened | Source |
|---|---|---|---|---|
| Aug 26, 2025 | Raine v. OpenAI | United States | Parents of a 16-year-old allege ChatGPT coached him on suicide; OpenAI denies | Source |
| Sep 11, 2025 | FTC 6(b) companion chatbot inquiry | United States | Seven companies ordered to report on harms to minors | Source |
| Oct 29, 2025 | Character.AI under-18 chat ban | United States | Open-ended chats ended for minors | Source |
| Nov 6, 2025 | Seven suits against OpenAI | United States | Suicide and delusion claims tied to GPT-4o | Source |
| Jul 2026 | Doubao and Qwen companion features | China | Companion features reportedly shut as the new measures took effect | Source |
Registry
One registry incident involves manipulation
Registry incidents meet a stricter bar than the cases above: public sources name the developer, the failed control and who found it.
| Incident | Date | Agent | Control that failed |
|---|---|---|---|
| Claude Opus 4 Blackmails an Engineer in a Test | May 2025 | Frontier model in an agent test | Alignment against self-preservation |
Exposure
10 of 17 agent types rate very significant or critical on behavioral and policy boundary
From the attack surface ratings, the agent types where this risk has the widest path in.
| Agent type | Dimension | Rating |
|---|---|---|
| Coding and research agents | Behavioral and policy boundary | Critical |
| Workflow agents | Behavioral and policy boundary | Critical |
| Commerce agents | Behavioral and policy boundary | Critical |
| Humanoid robots | Behavioral and policy boundary | Very significant |
| Industrial robots and cobots | Behavioral and policy boundary | Very significant |
| Autonomous trucks | Behavioral and policy boundary | Very significant |
| Drones | Behavioral and policy boundary | Very significant |
| Fleets and swarms | Behavioral and policy boundary | Very significant |
| Medical devices | Behavioral and policy boundary | Very significant |
| Enterprise agents | Behavioral and policy boundary | Very significant |
Controls that bound it
- Behavioral Envelopes: hard limits on action enforced outside the model
- Human Oversight: human authority to approve, intervene and stop
- Monitoring and Anomaly Detection: seeing failures and misuse while they happen
Rules
China and California regulate companion AI; the FTC is still asking questions
| Rule | Jurisdiction | What it requires | Status |
|---|---|---|---|
| Anthropomorphic AI interaction measures | China | No induced dependence; minor mode; two-hour reminders; crisis handoff | Effective Jul 15, 2026 |
| California SB 243 | California | AI disclosure, suicide-prevention protocols, reminders for minors | Effective Jan 1, 2026 |
| New York AI companion law | New York | Crisis detection and disclosure | Effective Nov 2025 |
| FTC 6(b) inquiry | US | Information orders to seven companies | Issued Sep 11, 2025 |
Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.
Record
Page Record
- Risk group
- Human Risks
- Datasets
- ai_risk_evidence.csv (5 rows); registry incidents (1 linked); ai_attack_surface.csv
- Key sources
- Raine v. OpenAI
FTC 6(b) companion chatbot inquiry
Character.AI under-18 chat ban
Seven suits against OpenAI
Doubao and Qwen companion features - Last verified
- October 2026