Building on preventative measures, you’ll also need to adopt and implement enhanced detective controls. This allows you to keep a continuous eye on the how agentic AI is behaving and performing.
Detective controls might include:
- Observability programs using predetermined measures and tolerance bands to detect out-of-bounds behaviour.
- Alert mechanisms sending timely alerts through an incident management process and facilitating appropriate responses from designated people when anomalies are identified.
- Human oversight: enforcing rigorous human oversight and control mechanisms to assess effectiveness and robustness of agentic AI systems; monitoring, intervening and overriding the agent’s actions when necessary.
Keep in mind, existing detective controls initially intended for traditional or generative AI applications may not be comprehensive enough to effectively observe agentic AI. Because these systems can potentially reach their own defined goals without human-in-the-loop oversight, they require more real-time, automated and continuous monitoring.
Even in highly autonomous systems, human oversight is still important. In the case of agentic AI systems, human oversight should generally:
- Outline human roles and responsibilities in overseeing agentic AI systems.
- Monitor feedback loops to identify issues and drive continuous improvement.
- Record, log and disclose the system’s behaviour and decisions as required to ensure explainability and transparency.
- Offer clear, effective means of intervening in agentic AI system operations, including the ability to pause, redirect or even shut down the system.
- Underscore the importance of training human operators and users to understand the capabilities and limitations of agentic AI systems and to develop the skills needed for effective oversight.
What’s more, because agentic AI systems span such a variety of goals and use cases, organizations can’t rely on a single or static list of behaviours to monitor and measure. These will need to be customized for the agentic system’s specific goals, risks and impacts and then assessed in the context of its use, with a feedback loop to the observation program as the agentic AI system’s capabilities evolve over time.
Consider technical requirements when addressing agentic AI controls and governance
In the same way that agentic AI’s complex interactions and dynamic environments require organizations to take a fresh look at internal controls, these systems also require additional technical evaluations over and above what might be performed for existing AI systems. Good governance must now also include tailoring technical evaluations of internal controls specifically for agentic AI systems.
For example, agentic AI may require you to evaluate the agent’s ability to perceive its environment accurately in the face of adversarial attacks. Addressing that control could mean testing a self-driving car’s perception with manipulated images or sensor data designed to fool the system.
In another case, you may need to measure the agent’s response time when faced with different scenarios and loads. This kind of latency and throughput analysis control might take shape in evaluating the agent’s response when faced with an unusual transaction or event, and assessing its ability to action or escalate.
Specifically, what kinds of technical evaluations could be tailored for agentic AI systems in support of appropriate governance?
- Adversarial attacks on perception: evaluates agent’s ability to perceive its environment accurately in the face of adversarial attacks.
- Out-of-distribution data: assess the agent’s performance when encountering data or situations it hasn’t been trained on.
- Stress testing: subjects the agent to high volumes of inputs, complex scenarios or unexpected events to identify performance bottlenecks, unintended adaptation or failure points.
- Simulation testing: uses simulated environments to test the agent’s behaviour in a variety of scenarios, including other AI/gen AI, edge cases and other events.
- Reward hacking analysis: evaluates agent’s behaviour and potential rewards structure exploits.
- Sensitivity analysis of reward: tests how changes to reward function parameters affect agent behaviour.
- Fairness evaluation: uses appropriate fairness metrics to quantify and compare agent’s performance across different groups.
- Adversarial robustness testing: evaluates agent’s performance under various conditions and determines its ability to maintain alignment, accuracy and reliability.
- Latency and throughput analysis: measures agent’s response time when faced with different scenarios and loads.
- Scalability testing: tests horizontal and vertical scalability to assess agent’s ability to handle increased demand.
Building a holistic governance framework to reflect these kinds of agentic AI use cases and qualities is essential as you reframe technical risk management controls for the agentic AI age.