The Defensible Criterion Drives Out the Right One
The Defensible Criterion Drives Out the Right One
Why organisations keep choosing measures they know are inadequate
Here is a pattern that shows up in three different kinds of decision, and the sameness is the point.
An academic appointment.
A faculty is choosing between candidates. One has a strong publication record in highly ranked journals. The other has fewer publications but asks better questions, work that opens something rather than extending an established literature. Everyone in the room knows the ranking measures the journal rather than the article, and most of them have said so publicly. The first candidate is appointed.
A promotion.
Someone has been in the organisation fourteen years. Nothing has gone wrong, they are well liked, and the general view is that they are not really suited to the role in question. Someone says the sentence everyone recognises: they’re due. They are promoted.
An operating framework.
A function adopts a set of criteria for assessing its own effectiveness, a dozen or so, covering the right subjects, assembled thoughtfully. Everyone involved knows that scoring well against the criteria is not the same as being effective. The criteria are adopted anyway, and within a year they are what the function reports on.
Three decisions, three domains, and the same thing happening in each. A criterion known to be inadequate wins against one known to be better.
The usual explanation is laziness, or risk aversion, or institutional inertia. I don’t think it’s any of those. Participants aren’t confused about which measure is sound. They are responding to something the measure does that has nothing to do with its accuracy.
What the defensible criterion actually provides
Consider the position of the person making the decision.
If they appoint the candidate with the stronger publication record and the appointment disappoints, nobody will ask why they chose that person. The choice was defensible on face value. The failure will be attributed to circumstances: to a poor fit, a difficult period for the department.
If they appoint the candidate with the better questions and the appointment disappoints, the question will be asked, and it will be asked of them personally. They will have to explain why they went against the available measure. The failure becomes theirs in a way it otherwise would not have been.
The asymmetry is not about the quality of the decision. It is about what happens to the decision-maker afterwards, and it runs the same way in all three cases.
Seniority settles a promotion argument because nobody can dispute it. A judgement about suitability exposes whoever made it to the three people who were passed over, each of whom now has a grievance with a name attached.
A framework score is a number produced by a process. A view about whether the function is actually working is a judgement someone has to stand behind, and that someone can be wrong in public.
So the defensible criterion is not chosen because people think it is better. It is chosen because it transfers responsibility from the person to the measure. That is a genuine service, and under uncertainty it is worth a great deal.
Which is why the usual remedies don’t work
Almost everyone who has written about this has concluded that the answer is better judgement: committees should read the work rather than count it, managers should assess suitability rather than tenure, leaders should look past the scorecard.
All correct. None of it addresses the thing producing the behaviour.
Someone who is fully persuaded that rankings measure the container rather than the content still faces the same exposure when they depart from the ranking. Their conviction has not changed what happens to them if the appointment goes badly. The argument has been won, and the arithmetic is untouched.
This is why the case has been made for decades: the specific objection to journal impact factors has been a matter of formal institutional position since at least 2013, and practice has barely moved. Nobody disputes it. Agreement doesn’t alter the exposure.
The deeper cost: the criterion becomes the thing
There is a second effect, and it is worse than the first.
Once the defensible criterion becomes the basis for decisions, it stops being a proxy and becomes the objective. People optimise for it, because that is what rational people do with a measure that determines outcomes. Academics publish for the ranking. Employees accumulate tenure visibly. Functions report against the criteria.
At that point, the criterion is no longer even a poor proxy for what it stood in for. It has become a separate activity with its own skills and its own winners, and everyone acting sensibly within it has severed the relationship with the original purpose.
So the organisation ends up with a measure nobody defends on the merits, which everyone optimises for, which no longer tracks what it was introduced to track and which remains in place because the alternative requires someone to carry a judgement personally.
What would actually change it
Not better judgement, and not a better criterion. The thing to change is what the decision-maker is exposed to.
Make the reasoning the accountable act. Right now, only departing from the measure requires justification. Following it requires none. That asymmetry is the whole mechanism, and it can be reversed: require that every decision of this kind record its reasoning – what was considered, what was rejected, why this choice – whether or not it followed the metric. Where the reasoning is always recorded, the judgement itself becomes the thing being accounted for, and deferring to a number is no longer the free option.
This sounds like bureaucracy and is closer to the opposite. It removes the protection that made the poor criterion attractive.
Make it collective and make it visible. A decision carried by one person is a decision one person can be blamed for. A committee that records a reasoned position, signed by all of them, distributes the exposure that currently makes the defensible criterion worth having. The same works in reverse: where a decision is nominally collective but the reasoning is attributed to whoever argued for the unconventional choice, the protection has not been distributed at all.
And ask who can afford to argue. In any room where this is happening, the cost of dissenting is unevenly distributed. The newest member of a committee, the person whose reappointment is not secure, the manager without a protector – they pay most for departing from the measure, and they are frequently the ones with the clearest view. If the only people who can afford to argue for judgement are the ones with nothing at stake, the room is not deliberating. It is ratifying.
The uncomfortable summary
Organisations do not choose bad measures because they cannot recognise good ones. They choose measures that protect the people who have to decide.
Which means every argument about better metrics, better frameworks and better criteria is being conducted at the wrong level. The question is not which measure is sound. It is what happens to the person who has to stand behind an answer the measure didn’t give them. And until that changes, the defensible criterion will go on winning against the right one, chosen by people who know exactly what they are doing and why.