Who takes the blame when AI agents get it wrong?

When an AI agent makes a decision that goes wrong, who takes the blame? And is knowing what happened enough if nobody is held accountable? As government moves towards more autonomous AI, Snowflake Global Field CTO Fawad Qureshi explains why audit trails alone won’t cut it.

How does accountability change as government moves from AI that provides answers to AI that can take action?

I think about this as a progression: Answer → Recommend → Act. The governance requirements increase significantly at each step. An AI system summarising a policy creates a very different risk from one recommending whether somebody qualifies for a benefit, and that’s different again from a system actually stopping a payment or changing a citizen’s entitlement.

Existing governance becomes insufficient when we continue governing the model but fail to govern the wider system around it. With agents, the unit of governance is no longer just the model. It’s the model, agent, data, tools, identity, policies, permissions, actions and humans.

The more authority we delegate, the more important it becomes to define boundaries before the agent acts, rather than simply investigating afterwards.

You’ve described an emerging ‘agentic supply chain’. Why does this make accountability more complicated?

The agent presenting the final answer may not be the agent that introduced the error. One agent might retrieve information, another interpret policy, another calculate eligibility, another make a recommendation and another execute the action. An error introduced several steps upstream can propagate through the entire chain.

Public bodies will need provenance across that chain. We need the equivalent of Git blame for AI decisions: which agent introduced this fact, which data supported it, which policy was applied, which model interpreted it, and who ultimately authorised the action.

Without that chain of custody, “the AI did it” risks becoming an explanation for decisions that nobody can properly reconstruct.

Knowing what an AI agent did isn’t the same as knowing whether it should have been allowed to do it. What does effective governance look like?

Observability might tell me that Agent A called Agent B, which accessed Dataset C and changed System D. Governance asks whether those actions were authorised, whether approved policies and data were used, and whether human approval was required.

For government, I would think about this through six questions:

• Who owns the outcome?
• What can the agent access, decide, spend, change or delegate?
• What evidence must be retained?
• How is it tested before and after deployment?
• Who can stop it or reverse an action?
• And what happens when the rules are breached?

Governance without the ability to intervene or enforce can quickly become paperwork.

If an AI-supported decision affecting a citizen is wrong, is having a detailed audit trail enough?

The test I would use is simple: can we reconstruct the decision well enough to understand why this citizen received this outcome?

That means knowing the original request, the agents involved, the model and version, the data and policies used, significant intermediate decisions, permissions, human approvals and what changed as a result.

But even a perfect audit trail isn’t enough. An AI system can work exactly as designed, follow every policy and still produce an outcome that an organisation must answer for.

This is where three concepts become important. Traceability asks what happened. Accountability asks who must act. Enforcement asks what happens if nobody does.

Without effective mechanisms for challenge, redress and enforcement, a system may be able to explain why someone received an unfair outcome without providing any meaningful way to correct it.

How should public bodies decide what AI agents are authorised to do, and when human approval is necessary?

An agent should have an explicit identity and delegated authority, just as a human employee does. We should know who it represents, what information it can access, which tools it can use and what decisions or actions it can execute.

Authority should be bounded by context. An agent may retrieve information but not change it; recommend a payment but not release it; schedule an inspection but not impose a penalty.

Human approval should remain at clearly defined boundaries based on consequence, reversibility, financial value, legal effect and risk to the citizen. Autonomy should not imply unlimited authority.

What happens to accountability when AI agents operate across government departments and suppliers?

A citizen shouldn’t find themselves caught between a government department saying its supplier’s agent made the mistake, the supplier blaming a model provider, and the model provider pointing to the data.

Technical provenance needs to cross organisational boundaries, but so do contractual responsibilities. Government needs to define who owns the outcome, what evidence suppliers must retain, how incidents are investigated and who is responsible for remediation.

You cannot outsource a component and assume you have outsourced accountability for the public service it delivers.

Ultimately, where should human accountability sit as AI agents become more autonomous?

Accountability has to sit with the humans and institutions that decide to deploy the system and grant it authority.

That doesn’t mean a human needs to approve every individual action. If that were required, much of the value of agentic systems would disappear. It means humans remain accountable for defining boundaries, monitoring whether they remain appropriate, intervening when necessary and providing redress when things go wrong.

There is also an important distinction between a human being in the loop and a human being accountable. Clicking “approve” on hundreds of machine-generated decisions doesn’t automatically create meaningful human oversight. Accountability requires genuine authority, sufficient information to challenge the system and the ability to intervene.

We may delegate the task. We may delegate the decision. We may even delegate the action. But we cannot delegate accountability.

“Computer says no” cannot be the defence. And “computer says why, but nobody has to do anything about it” isn’t much better.