137AI > AI Risks & Management

AI Risks & Management

Autonomous and ambient AI agents create two kinds of harm. The first lands on people: injury, theft, manipulation, surveillance, impersonation, and coordinated attack carried out through agents that move, speak, watch, or transact on someone's behalf. The second lands on the systems that make agents work: poisoned telemetry, tampered training data, compromised model updates, fleet-wide coordinated data attacks, and supply-chain compromise of the software that ships to deployed agents. Both surfaces are real, both are growing, and both call for a working discipline of identification, assessment, and treatment that operates across them.

AI & Human Risks

Human risks are the harms agents create for people directly. A humanoid robot can be directed to walk into a store and remove merchandise. A robotaxi can be summoned to move contraband without a driver to question the cargo. A smart glasses wearer can record a private conversation a bystander never consented to. A software agent with payment authority can be tricked through prompt injection into completing an unauthorized purchase. A fleet of autonomous delivery vehicles can be coordinated by an organized network into a logistics platform that no human-staffed equivalent could match for deniability or scale. Human Risks covers the taxonomy in detail across criminal misuse and the autonomous crime economy, physical safety and bodily harm, personal manipulation and coercion, surveillance and privacy invasion, impersonation and social engineering, weaponization and coordinated attack, cyber-physical compromise, agentic misbehavior and autonomous escalation, multi-agent coordinated misuse, and AI-enabled critical infrastructure compromise.

The category is distinguished by the fact that the harm reaches people through the agent's intended capability. The robotaxi enables transport. The humanoid enables physical labor. The smart glasses enable hands-free capture. The software agent enables autonomous action. Each capability is the value the system was built to deliver, and each is also the path by which harm reaches a person when the system is misused, compromised, or operated outside its intended envelope. Mitigations are correspondingly varied: behavioral constraints at the action layer, identity and accountability requirements for operators, geofences and zone restrictions, runtime monitoring and intervention authority, and legal frameworks that allocate liability when harm occurs.

AI & Data Risks

Data risks are the harms that arise inside the pipelines that feed, train, update, and coordinate AI agents. Every deployed agent sits inside a loop: sensors capture data, telemetry flows to a cloud or fleet backend, the data informs model training and operational decisions, updated models and policies flow back to the deployed agents over the air. Each stage of this loop is a target. Sensors can be spoofed at the source. Telemetry can be intercepted in transit. Training data can be poisoned to corrupt the next generation of models. Model updates can be compromised before they reach the fleet. Fleet-wide coordinated attacks can ride the orchestration plane to thousands of agents at once. Data Risks covers telemetry capture integrity, transit security, training data poisoning, model update integrity, fleet-scale coordinated attacks, surveillance material harvesting through compromised devices, supply-chain-of-updates attacks, and OT/ICS telemetry and digital twin deception.

What separates data risks from conventional cybersecurity is the closed loop. In traditional IT, a data breach exposes information and the harm flows from the disclosure. In AI agent systems, corrupted data shapes the behavior of the next model trained on it, and that model is then deployed back to the fleet, where its altered behavior produces more data that feeds the next training cycle. Attacks that successfully reach training data or model update pipelines compound rather than dissipate. Mitigations are cryptographic and architectural: signed updates, hardware roots of trust, staged rollout discipline, telemetry provenance, attestation at every link in the loop, and detection mechanisms that can catch corruption introduced anywhere along the path.

AI Risk Management

Risk management is the discipline of figuring out what can go wrong, ranking the possibilities by likelihood and consequence, choosing which to treat and how, and accepting whatever residual risk remains after treatment. The discipline is not new. ISO 31000 codifies the general enterprise version. NIST has published a Risk Management Framework specifically for AI. Sector regulators publish their own variants for healthcare, finance, transportation, and critical infrastructure. Risk Management covers how these frameworks apply to autonomous and ambient AI agents, where they need adaptation, and what assessment patterns work in the field.

The adaptations matter because conventional risk management was built around the assumption that risks can be enumerated, controls can be designed to prevent or reduce them, and residual risk can be measured against a tolerance threshold. Autonomous and ambient agents complicate every step. The risks are not fully enumerable because the agents' capability surface is open-ended and grows with each model update. The controls cannot prevent the structural attack surface because the surface is the value of the system. The residual risk cannot be measured cleanly because the same agent in a different deployment context has a different threat profile. Effective risk management for these systems treats the enumeration as ongoing, the controls as bounding rather than preventing, and the residual risk as a property to be monitored and recovered from rather than driven to zero.

Related Coverage

Agents | Security & Trust | Governance | Controls