137AI > Risks & Management > AI Risk Management

AI Risk Management

Risk management is the working discipline of figuring out what can go wrong, ranking the possibilities by likelihood and consequence, choosing which to treat and how, and accepting whatever residual risk remains after treatment. The discipline is not new. ISO 31000 codifies the general enterprise version. Sector regulators publish their own variants for healthcare, finance, transportation, and critical infrastructure. NIST has published a Risk Management Framework specifically for AI. What is new is the application of these frameworks to autonomous and ambient AI agents, where the assumptions underlying the conventional approach hold less cleanly and the assessment work is closer to a moving target than a one-time exercise.

General Frameworks: ISO 31000 and Enterprise Risk Management

ISO 31000 is the international standard for risk management as an organizational discipline. It defines a process of establishing context, identifying risks, analyzing them, evaluating them against tolerance thresholds, treating those that exceed tolerance, monitoring the controls, and reviewing the process. The standard is deliberately framework-agnostic, applicable to financial risk, operational risk, strategic risk, and increasingly technology risk. Most large enterprises operate some version of ISO 31000 through their enterprise risk management function, often with a risk register, periodic assessment cycles, and a governance committee that reviews material risks. ISO 31000 & Enterprise Risk Management covers how the standard applies to AI agent deployments, the adaptations operators are making to handle agent-specific risks within an established ERM program, and the patterns that distinguish mature AI agent risk management from a bolt-on to a conventional risk register.

NIST AI Risk Management Framework

NIST published the AI Risk Management Framework in 2023, with subsequent profiles addressing generative AI and other use cases. The framework organizes AI risk management into four functions: govern, map, measure, and manage. Govern establishes the policies and accountability structures. Map identifies the AI system in context, including its purpose, scope, and stakeholders. Measure assesses the system's behavior, performance, and trustworthiness characteristics. Manage allocates resources to treat identified risks. The framework is voluntary, but it is increasingly cited in procurement requirements, contractual obligations, and regulatory commentary as the de facto reference for AI risk management practice. NIST AI Risk Management Framework covers the four functions in detail, the profile additions, the assessment artifacts the framework expects, and the integration patterns between NIST AI RMF and other frameworks operators already run.

Threat Modeling for Agent Systems

Threat modeling is the structured exercise of enumerating what an attacker would want to do to a system, what paths exist for them to do it, and what controls bound the paths. The classical methodologies — STRIDE, PASTA, attack trees, kill chains — were developed for software systems with relatively stable boundaries and well-defined trust zones. Agent systems complicate the exercise because the attack surface is the value surface, the action authority is broader than the immediate user authorization, and the inputs include open-ended natural language that can carry instructions as well as data. Effective threat modeling for autonomous and ambient agents combines elements from the classical software approaches with patterns specific to AI: prompt injection as a first-class attack class, model corruption as a persistent threat, fleet-scale amplification, agent-to-agent deception, and the cyber-physical bridge for agents with action in the world. Threat Modeling for Agent Systems covers the classical methodologies, the adaptations needed for agent contexts, and worked examples across physical, personal and ambient, and software agent categories.

Sector-Specific Risk Frameworks

Some industries have their own established risk management frameworks that AI agent deployments must operate within. Healthcare operates under HIPAA risk analysis requirements and FDA guidance for AI as a medical device. Financial services operates under model risk management guidance, including SR 11-7 in the United States and equivalent frameworks elsewhere. Transportation operates under NHTSA and FMCSA safety case requirements. Critical infrastructure operates under CISA guidance and sector-specific risk frameworks for the energy, water, and chemical sectors. Each sector framework predates the current generation of AI agents, and each is being adapted to handle the distinctive risks autonomous and ambient agents introduce. Sector-Specific Risk Frameworks covers the sectoral landscape, the adaptations underway, and the cases where AI agent deployment exposes gaps in the sector framework that have not yet been resolved.

Assessment and Ongoing Monitoring

Conventional risk assessment cycles run on a quarterly or annual cadence with point-in-time evaluation against a defined scope. AI agent systems do not hold still. Model updates change the agent's behavior. New tool integrations expand its action surface. New deployment contexts change the threat profile. New attacks discovered in research alter the likelihood estimates for known risks. Effective assessment for these systems is more continuous than periodic, more behavior-watching than checkbox, and more able to update its picture of the system as the system itself changes. The supporting practices include automated behavioral testing, red team exercises that evolve with the agent, telemetry-based anomaly detection, and assessment artifacts that link to the deployed agent's current state rather than a frozen snapshot. Assessment & Ongoing Monitoring covers the practices, the tooling, the artifacts, and the cadence patterns that operators are converging on for agent systems.

Treatment and Residual Risk Acceptance

Risk treatment is the work of doing something about a risk that exceeds tolerance: avoiding it by changing the system, reducing it by adding controls, transferring it through insurance or contractual allocation, or accepting it as the cost of the activity. For AI agents, treatment is constrained by the structural property that the attack surface is the value surface. Avoidance often means giving up the agent's intended capability. Reduction often means making the surface auditable, attributable, and recoverable rather than closing it. Transfer is constrained by an insurance market that is still developing products for autonomous agent exposure. Acceptance requires an honest accounting of what residual risk the operator is choosing to live with, documented in a form that allocates accountability if the residual risk materializes. Treatment & Residual Risk Acceptance covers the treatment options available for AI agent risks, the patterns for documenting residual risk acceptance, and the governance practices that prevent residual risk from accumulating quietly until an incident forces a reckoning.

Related Coverage

Risks & Management | Human Risks | Data Risks | Compliance & Conformity