Nobody Knew They Were Making a Decision
Nobody Knew They Were Making a Decision
How AI quietly redistributes authority, and why nothing records that it happened
Most conversations about AI governance are about oversight, ethics, and capability. Those matter. But what determines whether an AI-assisted decision has an owner is happening somewhere else entirely, and almost nobody is looking at it.
Four people in the chain, none of them deciding
Consider a complaints process, which is a good example precisely because it is mundane. A complaint arrives, a system scores it, and anything the system is sufficiently confident about is resolved without a person ever seeing it. Most of the time, this works as intended: routine cases clear quickly, staff who would otherwise have handled them do something more useful, and the service improves.
Then one comes through that should have been escalated, and it costs you a customer. In a public service, it costs considerably more.
Ask who owns complaints, and someone will name a leader without hesitating. Ask who owned that particular decision, and you get four answers.
Whoever set the confidence level. Whoever approved the tool. The leader whose name sits on the service. And the person on the front line who never saw the case, because the system resolved it.
All four are in the chain. None of them made a decision they understood themselves to be accountable for.
That is not a failure to assign ownership, and it will not be fixed by assigning it more carefully. It is a structure that splits one decision across four positions and splits authority from consequence in the process. Whoever set the threshold held real authority over thousands of future outcomes and bore none of the consequences.
The leader whose name is on the service carries the consequence and holds no authority over the threshold.
A decision right nobody experiences as one
Accountability gaps are not new, and organisations have been drifting out of alignment with their own org charts for as long as org charts have existed. What makes this different is worth stating precisely, because it explains why the gap opens so quietly and stays open so long.
A threshold set inside a system is a decision right that nobody experiences as a decision right.
Set a confidence level at 0.85, and you have allocated authority over every case that follows. You have decided which complaints reach a human being and which do not, for thousands of people you will never meet, on the basis of a number. But it does not feel like exercising authority. It feels like configuring software.
So whoever is technically closest to the tool does it, often for entirely sensible technical reasons, by someone competent trying to make the system work well. It does not go to a governance forum, because it is a setting rather than a policy. It is not minuted because nobody present understood that they were making a decision. And it is recorded nowhere as a decision, which means that six months later, when the escalation that should have happened didn’t, there is no document to return to.
Decision rights therefore get distributed by people who do not know they are distributing them, to a system that cannot carry consequence at all.
Compare the speed. An org chart takes weeks or months to drift out of alignment with who actually decides what, and however out of date it becomes, everyone can at least see the chart and argue about it. This kind of drift happens the moment someone opens a settings screen, and there is nothing to look at afterward.
Two questions that don’t work, and two that do
The standard governance questions are not wrong so much as insufficiently specific, and both can now be answered affirmatively by organisations where the gap is wide open.
Do we have human oversight? Most organisations can say yes. There is a review step, a person in the loop, an escalation path. The honest follow-up question is how often those humans actually depart from the system’s recommendation.
This matters more than it sounds. Where volume is high, time is short, and the system is right most of the time, agreement becomes the rational default. Not through negligence, but because departing costs the reviewer time and exposure while agreeing costs nothing. What you end up with is a human in the loop who is functionally a rubber stamp, inside an organisation that believes it has oversight because the process shows one.
So the operative question is not whether oversight exists but what the departure rate is. A near-zero departure rate is not evidence the system is performing well. It is evidence the oversight is not functioning, and it should trigger review rather than reassurance.
Who is accountable? Someone can always be named, and in a well-run organisation someone will have been. The question is whether the person named can change what actually produced the outcome – the model, the threshold, the training data, the policy encoded in the tool.
If they cannot, you have not assigned ownership. You have assigned exposure and attached a name to it. That is worse than leaving it unassigned, because it produces an official who is personally answerable for outputs they have no means to correct, and it satisfies the governance requirement while leaving the fault entirely in place.
What follows
Two things, and neither is technical.
Configuration decisions that determine outcomes at scale need to be recorded as decisions.
When a confidence level, an automation boundary, or an exclusion rule is set, that is a policy choice affecting people at scale, and it should carry a named owner, a stated rationale, and a review point with the same weight as any other delegated authority.
This sounds like bureaucracy and is closer to the opposite. The absence of a record is what makes the drift invisible: without one, nobody can establish later what was decided, by whom, on what basis, or whether it was ever revisited. It also makes error identification considerably harder, since the first question after a bad outcome – what determined this, and who chose that determinant – has no answer.
Accountability needs to be tested against authority before assigning it.
For any AI-assisted decision, one question settles it: can the person accountable for the outcome change what produced it?
If yes, the assignment is real. If no, accountability sits somewhere else, usually with whoever controls the configuration, the procurement, or the policy, and it should be named there. This is a five-minute test that almost nobody runs, and it would prevent a substantial share of the accountability gaps that surface later as incidents.
The shape of the problem
None of this argues against automating decisions. Most of what these systems do, they do well, and the routine cases they resolve are cases that did not need a person.
It is an argument about what happens to authority when a decision process changes shape. Introducing an AI system into an existing process does not neutrally speed it up. It moves where the decision is actually taken – usually upstream, into a configuration screen, and usually without anyone recording that it has moved. The people setting such parameters are often technical rather than accountable, and they experience it as configuration rather than delegation.
Which is the whole difficulty in one sentence. Deciding is becoming a configuration question. Carrying the consequence is still purely a structural one.