Government organisations risk overstating the benefits of AI unless they evaluate how systems perform in practice, including whether employees use them and whether they genuinely improve public services.
During a panel discussion at Think AI for Government 2026, representatives from the Home Office, Department for Transport, Incubator for AI (i.AI) and Snowflake (pictured) examined how public bodies can test, assure and evaluate AI as adoption increases.
The discussion highlighted the limitations of measuring success through time savings alone, as well as the need for ongoing monitoring once AI tools move beyond controlled pilots.
Dr Ana Wheelock Zalaquett, head of AI research at the Department for Transport, challenged the assumption that evaluation should take place towards the end of a project.
“Evaluation should happen throughout the life cycle of an AI tool or system, and it starts at discovery, and it should never end,” she said.
Early evaluation could help organisations define the problem they were trying to solve, identify who would be affected and determine whether AI was an appropriate response.
Later stages should establish whether a system worked as intended and, ultimately, whether it delivered better outcomes than existing approaches.
When promising pilots meet reality
Samuel O’Sullivan, AI evaluation lead at the Home Office, described the challenges of maintaining assurance as AI tools move into everyday operations.
The department has tested AI systems supporting asylum decision makers, initially using tightly controlled pilots with extensive checks on outputs and close feedback from users. Some tools have now been in development and use for two or three years, raising questions about how much monitoring remains practical as deployment expands.
“We all know the point of these tools is to realise efficiencies. We can’t remove all that efficiency through checking,” he said.
However, O’Sullivan suggested that one of the biggest lessons concerned the people expected to use the technology. Despite promising pilot results, some tools had experienced much lower uptake than anticipated when rolled out more widely.
“Quite frankly, adoption uptake has been pretty low, much lower than we would have liked,” he said.
Introducing AI into established working practices represented a significant change for employees, who were also receiving conflicting messages about the technology’s benefits and risks.
Understanding those concerns, he argued, should be part of the evaluation process rather than an afterthought.
Accountability must be built in
Fawad Qureshi, global field CTO at Snowflake, raised a different challenge: the growing ability of citizens to use AI when interacting with government.
He described using an AI agent to challenge a parking penalty notice, illustrating how individuals could increasingly automate correspondence and navigate administrative processes. That shift could create additional demand for public services, while increasing the importance of ensuring that government responses were accurate and accountable.
Qureshi argued that public bodies faced a particular responsibility because their decisions could have serious consequences for individuals.
“You need to design that there will be an objection, there will be a question,” he said, arguing that traceability should be built into systems from the outset.
If you liked this content…
He also cautioned against allowing AI to make autonomous decisions in complex, ambiguous situations affecting people’s lives.
While AI could support activities such as document summarisation, decisions about asylum applications should remain with human officials, he argued.
The distinction, he suggested, was between using technology to support public servants and delegating consequential judgements to systems that might not understand the full context.
Measuring more than efficiency
Lucy Makinson, head of impact & evaluation at i.AI, described how different stages of evaluation required different approaches.
Her team was working with developers of AI tutoring tools, initially examining whether products met teachers’ needs before moving towards safety testing and, eventually, assessments of educational impact.
Establishing whether a tool improved learning could take years, she explained, particularly when researchers needed to understand whether benefits persisted over time.
Makinson also questioned the tendency to measure AI through financial savings because these were easier to quantify and compare across projects.
“We definitely don’t want to just be looking at cost savings and efficiency,” she said.
AI applications varied so widely that attempting to capture their value through a single financial measure risked overlooking what individual systems were intended to achieve.
O’Sullivan agreed that evaluation needed to start with a clear understanding of existing performance.
“What is a good asylum decision? How good are we at doing that in business as usual?” he asked.
Without that baseline, organisations could struggle to demonstrate whether AI was improving outcomes.
Challenging the AI hype
The panel concluded by addressing how organisations could evaluate technology objectively amid competing claims about AI’s capabilities.
Makinson argued that rigorous evaluation was essential to navigating the extremes of enthusiasm and scepticism surrounding AI.
“Evaluation is the only way of getting around hype,” she said.
O’Sullivan encouraged public bodies to challenge suppliers’ claims through practical testing, including ambitious use cases, rather than accepting assurances about what systems could achieve.
Wheelock Zalaquett similarly suggested that government buyers should be more demanding when suppliers declined to share evaluation evidence.
Together, the panellists made the case for treating assurance as an ongoing responsibility, from identifying the original problem through to monitoring live services.
