AI governance risks falling short once systems go live

Government organisations need to pay as much attention to who owns AI systems once they are deployed as they do to the governance processes used to approve them, according to Holly Foxcroft, business information security officer at OneAdvanced.

“For many organisations, the challenge isn’t putting an AI governance framework in place, but ensuring there is clear accountability once a system moves from approval into day-to-day use,” said Foxcroft.

AI deployments often cut across multiple teams. In the public sector, digital teams may own the technology, information governance teams the data protection impact assessment, while individual services remain responsible for outcomes for citizens.

“Each holds a piece of the picture, but nobody necessarily owns the join between the pieces. Once the pilot ends and the tool moves into business as usual, ownership can default to whoever is still in the room, rather than whoever should hold it,” Foxcroft told Think Digital Partners.

Foxcroft argued that responsibility for AI risk shouldn’t automatically fall to technology teams just because they manage the system. Instead, the service owner should ultimately own the risk because they are accountable for the service and understand the potential consequences for citizens.

Technology teams should be responsible for ensuring systems perform as expected and identifying issues such as model drift, while senior leaders need to set the organisation’s risk appetite and decide whether AI is appropriate for a particular use case in the first place.

Suppliers also have responsibilities and may be contractually liable when problems occur, but Foxcroft warned that outsourcing technology does not remove the public body’s responsibility to citizens.

“If a contract is the only place risk ownership is written down, that suggests the internal accountability model hasn’t actually been built,” she said.

Human oversight should reflect the consequences

Rather than deciding where humans need to remain involved based purely on AI’s technical capabilities, Foxcroft said organisations should consider the stakes involved and how easily a bad decision can be reversed.

“The starting point should be the stakes and reversibility of the decision, rather than simply what the technology is capable of,” she said. “If a decision is wrong, how quickly will that become clear, how easily can it be reversed, and what is the impact on the person affected in the meantime?”

Here, low-risk activities such as suggesting appointment slots or directing queries to the correct queue could operate with relatively light human oversight.

But that changes when AI influences decisions involving entitlement, liberty, safeguarding or access to statutory services. In those circumstances, human oversight must be meaningful, she said, with staff given “the time, information and authority to challenge the output”, rather than rubber-stamping an automated recommendation because of workload pressures.

Turn AI principles into everyday procedures

At the same time, Foxcroft said frameworks can describe what good AI governance looks like without answering the practical questions facing staff using the technology.

“What they rarely include is what the user does on a Thursday morning when they already have a queue building and need to decide: should AI be used to clear this?” she said.

That gap between principles and procedures can also contribute to shadow AI, particularly when technology is rolled out unevenly across an organisation. Operational teams therefore need to be involved in developing governance arrangements rather than simply being handed policies produced by security, governance or risk teams.

“Operationalisation therefore needs to be treated as a deliverable for every AI system, with the same rigour as the framework itself,” said Foxcroft.

Prepare for AI failure before it happens

Public bodies also need to plan for what happens when an AI system makes a mistake in a live service.

Foxcroft identified four capabilities that should be in place before deployment: mechanisms for detecting errors, hallucinations or security incidents; clear authority to pause or roll back a system; established accountability; and a route to remediation for citizens affected by an error.

Incident response arrangements should be documented and tested through exercises before problems occur.

While Foxcroft said many public sector organisations are unlikely to be fully prepared for operational AI failures yet, she argued they do not need to build an entirely new resilience capability.

“The organisations best placed to respond will be those that extend their existing operational resilience and incident management capability to cover AI, rather than treating it as something separate that needs to be built from scratch,” she said.

“Most public bodies already have the muscle for handling service issues. The job now is stretching it to cover this one too.”