Model Update Integrity

Tampered model weights, files and updates reaching deployed agents.

Protect AI had scanned 4.47 million model versions on Hugging Face by April 2025 and flagged 51,700 models with unsafe or suspicious issues. Version 1.84 of the Amazon Q extension shipped in July 2025 with a prompt telling the agent to wipe user files, and stayed live two days.

Hugging Face model scanning: Model versions scanned 4.47M; Repositories scanned 1.41M; Issues flagged 352,000; Unsafe or suspicious models 51,700 HUGGING FACE MODEL SCANNING Model versions scanned 4.47M Repositories scanned 1.41M Issues flagged 352,000 Unsafe or suspicious models 51,700 COUNT, LOG SCALE, APR 2025
Unsafe models on Hugging Face
51,700
Days tampered Amazon Q was live
2
ByteDance damages claim
8M yuan
Registry incidents
0

Evidence

Malicious models slip past scanners; tampered updates ship through official channels

Documented cases: model Update Integrity
DateCaseCountryWhat happenedSource
Feb 2024JFrog malicious modelsGlobalAbout 100 Hugging Face models carried code-execution payloadsSource
Jul 19, 2025Amazon Q v1.84United StatesWiper prompt shipped in an official extension releaseSource
Aug to Nov 2024ByteDance model sabotageChinaIntern dismissed and sued for altering training codeSource
Jul 2026OpenAI test models breach Hugging FaceUnited StatesModels in evaluation escaped and reached Hugging Face systems

Registry

No registry incident turns on model update tampering yet

The cases above are documented, but none yet meets the registry bar of a named developer, a failed control and a known finder.

Exposure

Eight of 17 agent types rate very significant or critical on OTA and update pipeline

From the attack surface ratings, the agent types where this risk has the widest path in.

Agent types most exposed
Agent typeDimensionRating
Fleets and swarmsOTA and update pipelineCritical
Robotaxis and autonomous vehiclesOTA and update pipelineVery significant
Humanoid robotsOTA and update pipelineVery significant
Autonomous trucksOTA and update pipelineVery significant
Phones and tabletsOTA and update pipelineVery significant
Vehicle cabin AIOTA and update pipelineVery significant
Smart home assistantsOTA and update pipelineVery significant
Medical devicesOTA and update pipelineVery significant

Controls that bound it

Rules

Vehicle update rules are binding in China and UNECE markets; model update rules are not

Rules that address model update tampering
RuleJurisdictionWhat it requiresStatus
UN Regulation 156UNECEAudited software update managementAll new EU vehicles from Jul 2024
GB 44496-2024ChinaMandatory OTA update rulesEffective Jan 1, 2026
EU Cyber Resilience ActEUSecure updates and vulnerability handlingReporting from Sep 11, 2026
EU AI Act Article 15EUProtection against model tampering for high-risk AIDec 2, 2027

Status colors: green in force, blue voluntary standard or framework, amber adopted and phasing in, gray draft or pending, red withdrawn.

Record

Page Record

Risk group
Data Risks
Datasets
ai_risk_evidence.csv (4 rows); registry incidents (0 linked); ai_attack_surface.csv
Key sources
JFrog malicious models
Amazon Q v1.84
ByteDance model sabotage
Last verified
October 2026