137AI > AI Risks & Management

AI Risks & Management

Autonomous and ambient AI agents create two kinds of harm. The first lands on people: injury, theft, manipulation, surveillance, impersonation, and coordinated attack carried out through agents that move, speak, watch, or transact on someone's behalf. The second lands on the systems that make agents work: poisoned telemetry, tampered training data, compromised model updates, fleet-wide coordinated data attacks, and supply-chain compromise of the software that ships to deployed agents. Both surfaces are real, both are growing, and both call for a working discipline of identification, assessment, and treatment that operates across them.

The Risks & Management pillar covers three areas: the harms agents create for people, the harms that arise inside the data pipelines that feed and coordinate agents, and the risk management discipline that operates across both. Each area has its own dedicated treatment; this page is the overview that locates them relative to each other.

The Three Areas

Human Risks covers the harms agents create for people directly — the harms that reach a person through the agent's intended capability. A humanoid robot can be directed to remove merchandise from a store; a robotaxi can be summoned to move contraband without a driver to question the cargo; a smart glasses wearer can record a conversation a bystander never consented to; a software agent with payment authority can be tricked through prompt injection into completing an unauthorized purchase. The area covers the taxonomy in detail across criminal misuse and the autonomous crime economy, physical safety, personal manipulation, surveillance, impersonation, weaponization, cyber-physical compromise, agentic misbehavior, multi-agent coordinated misuse, and critical infrastructure compromise. The mitigations are correspondingly varied — behavioral constraints at the action layer, identity and accountability requirements for operators, geofences and zone restrictions, runtime monitoring and intervention authority, and legal frameworks that allocate liability when harm occurs.

Data Risks covers the harms that arise inside the pipelines that feed, train, update, and coordinate AI agents. Every deployed agent sits inside a closed loop where sensors capture data, telemetry flows to backends, data informs model training, and updated models flow back to the deployed agents. Each stage of the loop is a target. The area covers telemetry capture integrity, transit security, training data poisoning, model update integrity, fleet-scale coordinated attacks, surveillance material harvesting, supply-chain-of-updates attacks, and OT/ICS telemetry and digital twin deception. What separates data risks from conventional cybersecurity is the closed loop — corrupted data shapes the behavior of the next model trained on it, and that model is deployed back to the fleet, so attacks that reach training data or update pipelines compound rather than dissipate. The mitigations are cryptographic and architectural — signed updates, hardware roots of trust, staged rollout discipline, telemetry provenance, and attestation at every link in the loop.

Risk Management covers the discipline of figuring out what can go wrong, ranking the possibilities by likelihood and consequence, choosing which to treat and how, and accepting whatever residual risk remains after treatment. The discipline is not new — ISO 31000 codifies the general enterprise version, NIST has published a Risk Management Framework specifically for AI, and sector regulators publish their own variants. The area covers how these frameworks apply to autonomous and ambient AI agents, where they need adaptation, and what assessment patterns work in the field. The adaptations matter because conventional risk management assumes risks can be enumerated, controls can prevent or reduce them, and residual risk can be measured against a tolerance threshold — and autonomous agents complicate every step.

How the Areas Combine

The areas combine because the two harm surfaces and the discipline that operates across them are not separable in practice. Human Risks and Data Risks are connected through the loop: a data risk that corrupts a model becomes a human risk when the corrupted model is deployed to an agent that acts on people, and a human risk incident produces data that feeds back into the loop. Risk Management is the discipline that operates across both surfaces — the same enumeration, assessment, and treatment practice applies whether the risk lands on people or on the systems.

The risk management adaptations are what make this pillar distinct from conventional risk practice. Conventional risk management was built around the assumption that risks can be enumerated, controls can be designed to prevent them, and residual risk can be measured against a tolerance threshold. Autonomous and ambient agents complicate every step — the risks are not fully enumerable because the capability surface is open-ended and grows with each model update; the controls cannot prevent the structural attack surface because the surface is the value of the system; the residual risk cannot be measured cleanly because the same agent in a different deployment context has a different threat profile. Effective risk management for these systems treats enumeration as ongoing, controls as bounding rather than preventing, and residual risk as a property to be monitored and recovered from rather than driven to zero. The integration with the engineering controls covered in the Controls pillar, the governance frameworks covered in the Governance pillar, the trust posture covered in the Security & Trust pillar, and the compliance practice covered in the Compliance & Conformity pillar produces the comprehensive risk practice that AI deployment at scale depends on.

The Reframe

Autonomous and ambient AI agents create harm on two surfaces — the people the agents act on, and the data systems that make the agents work — and the two surfaces are connected through the loop that turns corrupted data into corrupted behavior and back again. The risk management discipline that operates across both must treat enumeration as ongoing, controls as bounding rather than preventing, and residual risk as something to monitor and recover from rather than drive to zero.

Related Coverage

Agents | Security & Trust | Governance | Controls