A lightbulb moment for AI governance

The real breakthrough in AI governance happens when it moves beyond guardrails and becomes a catalyst for adoption and value.

In brief

• Most existing risk frameworks are not designed for AI’s adaptive and evolving nature, making traditional approaches to governance increasingly inadequate.
• Only a small proportion of chief risk officers can identify the appropriate controls for key AI risks, highlighting significant governance capability gaps.
• Organisations need new controls focused on explainability, accountability, human supervision and AI-specific safeguards to manage systemic risks effectively. 

It took decades for electricity to be understood and regulated as infrastructure, rather than a tool. AI is much the same, shaping economies and societies at large. And just like electricity, there will be some use cases that are harmless whilst others represent catastrophic risks. Once we see AI as something that powers everything else, the way we think about risk fundamentally changes. But most risk frameworks today haven’t caught up.

Electricity didn’t change the world when it was invented. It changed it when people realised electricity powered everything else.

From where I stand, risk leaders are being asked to govern AI as if it were a discrete tool – something assigned to IT – when really it is essential infrastructure, embedded everywhere and augmenting our processes and technologies.

If Australia wants to be a trusted centre for responsible AI, then this matters. Because trust is built at the infrastructure level, not the tool level.

The latest EY Responsible AI Pulse Survey makes this tension visible.

Just 11% of Australian chief risk officers (CROs) could identify the appropriate controls for five core AI risks. These were: hallucinations; algorithmic blind spots or training limitations; bias in training data; legal ownership in autonomous systems; and transparency in AI-generated content.

It doesn’t surprise me that CROs scored below average, even though AI risk sits in their remit.

CROs are trained to manage risks with a fixed shape: inputs, controls, outputs. But AI systems, especially machine learning, aren’t static. They learn from data patterns, retrain over time, and evolve in sometimes unpredictable ways.

AI doesn’t behave in ways that CROs have been trained to manage other types of technology risks. AI risk isn’t fixed. It’s a moving target.

Traditional risk management also relies on transparency. With AI, decision-making can be opaque. Deep learning models can often function as black boxes, with small changes in inputs producing disproportionately large changes in outcomes. Explaining how an AI system arrived at a particular decision – to a board, a regulator or a customer – can be difficult to justify if you don’t have an audit log.

This challenge is amplified by scale. Because AI is embedded across business units, failures are rarely contained. When electricity grids fail, the impact is systemic and can result in widespread blackouts or catastrophic fires. AI failures are similar. Small governance gaps are amplified. One issue can cascade across functions.

All this means the “it’s not my problem” attitude is about as helpful as shrugging when the power fails.

Working with clients who are building and deploying AI solutions, I see the controls necessary to manage this risk, particularly around interpretability, explainability and accountability. The distinctive attributes of AI require the introduction of net new controls and legacy control readiness. Areas such as third-party risk, conduct risk and business process risk need to be revisited in the context of AI, and new controls such as human supervision need to be considered.

So how do we manage these new risks?

The accidents and hazards experienced during the dawn of electrification sparked public outrage and eventually led to the development of regulation of electricity.

AI risk may follow a similar trajectory, unless we build controls and safeguards in by design. AI systems are learning, adapting and scaling regardless of whether regulation is ready or not, so best practice design methods need to be considered from the outset.

We don’t think of electricity as a tool. We switch it on so our tools can work. AI is reaching that same point. Risk frameworks – and risk specialists – need to catch up.