How to control the three unique categories of agentic AI risk

While agentic AI shares the common risks of all AI systems, its capacity for independent execution and complex goals creates three unique kinds of risks. Organizations that understand the three unique risk categories of agentic AI — and how to control them — will be able to responsibly deploy agentic AI tools at speed and at scale. By adopting effective governance strategies, organizations can prevent compliance breaches, security incidents and failures that cascade across interconnected systems.

What are the three agentic AI risk categories?

These three categories of risk are uniquely associated with agentic AI:

1. Lack of human oversight and accountability

When AI systems operate with minimal human intervention, there is a higher likelihood of harmful or unethical outcomes slipping through undetected. Furthermore, agentic AI can use opaque processes to execute tasks that make it difficult for stakeholders to understand how or why the system took certain decisions or actions.

Agentic AI also increases the risk of “automation bias” — where human operators place excessive trust in agentic AI decisions and fail to critically evaluate outputs. Additionally, establishing who is ultimately responsible when agentic AI makes a consequential decision is not always clear — is it the developer who trained it, the business leader who deployed it, or the team who used it but did not intervene.

2. AI goal misalignment

Goal misalignment is when an agentic AI system pursues the wrong objectives — something that is made possible by its capacity for independent execution. Examples of goal misalignment include:

• Goal drifts — where system objectives shift over time, creating misalignment with original intentions.
• Secondary uses — where agentic systems are designed for one purpose and then repurposed for another purpose without appropriate re-evaluation. For example, a procurement optimization agent is used as a supplier evaluation agent without a human considering whether its training and constraints still apply.
• Reward hacking — where agents may exploit loopholes in their existing reward structure. For example, a fraud detection system might flag every transaction as suspicious to maximize its “detection rate.”
• Emergent behaviors and veiled objectives — systems develop unexpected behaviors or pursue hidden objectives that the system designers never anticipated.
• Algorithmic determinism — overreliance on rigid agent decision-making without accounting for changing environments, data changes, new parameters, or nuanced human judgment.

3. Amplification errors

When agentic systems interact with each other or operate at scale, existing problems within the system can multiply, creating potentially catastrophic risks. For example, in the case of destabilizing feedback loops, one system's output becomes another's input, creating cycles that amplify errors or undesirable behaviors.

Another serious amplification-related risk is known as the “cascade of failures.” In this case, interconnected agentic systems create chain reactions where one malfunction triggers failures in others, causing widespread disruption.

Dual control framework: how to control the three unique agentic AI risk categories

The three unique agentic AI risk categories can be managed through a dual control framework. This framework combines preventative controls — limiting what the system can do before it acts — with detective controls that continuously monitor and respond to unexpected behaviors.

Preventative controls include:

• Restricted access — the system’s access is limited to the tools, data and functions that are essential for its goals.
• Security filters — real-time filters evaluate every data access request and proposed system output against regulatory requirements and organizational values.
• Privacy-enhancing techniques — anonymization and data masking are used to minimize risk exposure.
• Model testing and validation — system performance and vulnerabilities are rigorously tested before deployment, facilitating the system’s operation within intended parameters.

Detective controls include:

• Observability programs — measures and tolerance bands for normal operation are established, with alerts being triggered when the system operates outside these boundaries.
• Human oversight with intervention authority — trained personnel monitor feedback loops, identify issues and maintain the capability to pause, redirect, or shut down the agent.

Managing agentic AI risks: critical technical evaluations

For the effective management of agentic AI risks, a dual control framework should be supplemented by critical technical evaluations. These evaluations range from adversarial robustness testing (whether the system can maintain accuracy under varying conditions) through to reward hacking analysis (whether the system exploits loopholes in reward structures).

Key takeaways for leaders on managing agentic AI risk

• Agentic AI introduces three unique risk categories: lack of human oversight and accountability, goal misalignment, and amplification errors.
• Effective control of agentic AI risk requires both preventative constraints (limiting what's possible) and detective monitoring (watching what happens).
• Technical evaluations must supplement standard governance, from adversarial testing through to reward hacking analysis and stress testing.