Why agentic AI governance needs real-time trust layers

IT leaders need a new risk model to establish agentic AI ambitions that deliver.

In brief

• Agentic AI systems act autonomously under uncertainty, creating risks that traditional governance models cannot effectively manage.
• New threats include hallucinations, prompt injection attacks and hidden costs from uncontrolled multi-agent workflows.
• A trust layer playbook embeds monitoring, decision rights and human escalation directly into AI systems.

As organizations look to scale their agentic AI initiatives, they’re discovering a hard truth: Traditional command-and-control governance doesn’t work for agent-driven autonomous systems.

• Hallucinations: The tendency of large language models (LLMs) to make things up is well documented. While hallucinations have decreased as the models evolve, even a 98% accuracy rate is not sufficient for business-critical workflows, such as finance.
• Prompt injection: Bad actors can use specially crafted prompts to trick LLMs to override existing rules and execute malicious commands. These types of attacks can create significant privacy, financial and reputational risk, particularly with customer-facing agents.
• Hidden costs: Multi-agent workflows, left unchecked, can get caught in infinite loops that dramatically increase token use and drive up costs. Unexpected costs can eat into ROI projections and erode leadership confidence in AI investments.

Addressing these challenges requires a new model for assessing and mitigating risk. EY Consulting professionals call it a “trust layer” playbook.

“The old paradigm of ‘test and deploy’ is being replaced by ‘monitor and adapt continuously,’” says Guarrera.

This new paradigm of continuous monitoring and adapting requires organizations to build trust into the infrastructure of how the business runs, not just into a policy document that sits adjacent to it. This shifts governance practices from oversight to architecture.

A trust layer playbook for agentic AI centers on three primary principles:

1. Establish clear decision rights based on how much freedom an agent has. Every use case for deploying agents requires its own risk assessment. Teams will need to assess permissions and data access for agents — and account for agents that may attempt to actively seek to work around or change permissions in order to complete an assigned task.
2. Implement real-time monitoring systems and processes to track AI performance patterns. Observability is critical for identifying hallucinations, infinite loops or other risk signals. Guarrera suggests establishing “accuracy” as a managed metric for measuring performance drift from agents. Real-time monitoring helps encode compliance — with every action signed, traced and reversible, giving humans executive oversight.
3. Build escalation paths so that high-impact or strategic calls automatically go to humans. For early-stage agentic AI deployments, it’s better to err on the side of more human oversight throughout the process. “Until you can take the training wheels off, more human in the loop is better,” says.

Governance in the age of agentic AI requires a new playbook. Establishing new governance frameworks that maintain appropriate oversight while enabling AI autonomy moves beyond traditional command-and-control structures. Organizations need frameworks that establish clear boundaries for different levels of AI decision-making, implement real-time monitoring systems that track AI performance patterns, and create escalation protocols that bring humans into the loop for strategic decisions and maintain accountability for all outcomes.

“The goal isn’t to slow things down,” says Guarrera. “It’s to make things safe to go faster.”

This article was originally published on CIO.com