判断的噪声：为什么同一个问题，会得到不同的答案
We have spent three decades correcting bias, yet we almost never measure how differently the same case is judged by a different person. Bias has a direction, a villain, and moral weight; noise has only a distribution and no story — and it is precisely the most expensive and most neglected part of the error in any judgment system.
I. The Same Question, Different Answers
Begin with a scene that needs no theory. The same case file goes to two judges. One sentences three years, the other five. Both follow the law, both read the file carefully, neither takes a bribe, neither carries an obvious prejudice. The difference is not in their principles but in the judgment each made in that moment.
In the study of judgment, that difference has a name: noise.
Noise and bias are not the same thing. Bias is systematic deviation — everyone errs in the same direction, say by over-punishing a particular kind of defendant. Noise is random dispersion — different people, different days, different settings produce different answers with no common direction. Bias has a direction one can point to; noise does not. Bias can be argued with, criticised, corrected; noise can only be measured.
The trouble is that we have spent enormous effort correcting bias while almost never measuring noise. Bias has victims, narratives, and moral weight. Noise has only a distribution, and no story.
II. Why Noise Stays Invisible
Noise hides for three reasons.
First, it cancels out in the aggregate. If half the judges lean harsh and half lean lenient, the system looks, on average, fair. The mean conceals the spread. When we read the report, we see the average, not the variance.
Second, no one sees the counterfactual. You cannot know what sentence you would have received from a different judge. A judgment happens once, and noise only becomes visible in repetition. To see it, you must deliberately manufacture repetition: give the same material to several judges working independently, then compare. Most organisations never do this.
Third, organisations reward the appearance of consistency, not the fact of it. A well-written ruling with a full set of reasons looks professional; whether it differs from the ruling next door is a question nobody asks. Surface coherence substitutes for substantive agreement.
A well-known example suggests the scale of the problem. Ask a group of underwriters at the same insurance company to quote independently on the same fictitious policy, and the quotes often differ by enough to decide whether the deal happens at all. No one has made a mistake; each is drawing on experience. That is exactly what makes noise disturbing: it requires no one to be wrong.
III. The Cost of Noise Is Not Only Unfairness
The most direct cost of noise is unfairness: identical conduct, different treatment. But the deeper cost is that noise makes learning impossible.
If a system produces different outputs from the same input, you cannot tell whether the input changed or the judgment did, whether the method worked or luck did. Noise drowns the signal. An organisation that cannot distinguish a good judge from a lucky one cannot accumulate real experience. It builds causality on noise and rewards and punishes at random.
This is why noise functions as a hidden tax. Every inconsistent judgment spends trust. The person on the receiving end does not call it noise; they call the system unreliable. Over time, people stop trusting rules and start trusting relationships — because relationships at least are predictable.
IV. The Counter-Argument: Noise Is Sometimes a Feature
At this point the obvious conclusion is that noise should be eliminated wherever possible. That conclusion arrives too quickly.
Noise is not always bad. In exploratory domains, dispersion of judgment is itself a source of adaptation. If every researcher reached an identical conclusion about the same question, science would have no disputes and therefore no breakthroughs. If everyone in an organisation rated risk the same way, it would lose the ability to see danger from more than one angle. In ecology, diversity is the source of resilience; in judgment, dispersion may be too.
More awkwardly, over-standardisation manufactures a false objectivity. Compressing judgment into a scorecard looks like it removes noise, but it merely relocates judgment from the person to the weights — and who set the weights, and why, is often less transparent than the original judgment. Using rules to kill noise frequently just hides noise inside the rules.
There is a subtler counter-argument still: the wisdom of crowds depends on noise. If everyone's judgment is highly correlated, aggregation does not improve accuracy; averaging approaches the truth only when errors are independent. A certain amount of dispersion is not an obstacle to aggregation but the reason it works. Driving noise to zero may drive out the capacity to correct error along with it.
So the real question is not how to eliminate noise but which noise should be compressed and which should be preserved.
V. The Judgment: Measure First, Correct Second
If the core problem with noise is that it is invisible, the first step is not correction but measurement.
Three things follow.
First, audit dispersion. Do not look only at the average; look at how the same class of judgment is distributed across people and across time. A simple method: sample real cases, have several judges assess them independently, and compare. If the differences are wild, the problem is not the individual judge but the judgment system.
Second, institute decision hygiene. Independent assessment, delayed aggregation, structured evaluation, after-action review — what these share is that they design the process to reduce random fluctuation before judgment happens, rather than arguing about who was right afterwards. It sounds like bureaucracy, but it differs from bureaucracy in one respect: it constrains the process, not the conclusion.
Third, preserve legitimate space for noise. Where exploration, dissent, or creation is needed, do not rush to flatten judgment into a single line. There, noise is not an error but an option.
The order matters: measure first, then decide what to compress, and only then talk about standardisation. Skipping measurement and going straight to standardisation usually replaces one kind of noise with another.
VI. Coda: Noise Is the Shadow of Judgment
Every judgment casts a shadow. Bias is the shadow leaning in one direction; noise is the shadow trembling in place. We are used to asking why the shadow leans that way, and rarely ask why it trembles.
Noise is hard to deal with not because it is complicated but because it has no villain. Bias can be blamed on a person, a stance, a system. Noise has no one to accuse; it is simply the imperfection of judgment itself. To admit noise is to admit that our judgment is not a precision instrument but a fluctuating process.
And that is precisely why it can be improved. What cannot be seen cannot be managed; what can be measured can at least be bounded. A mature judgment system is not one without noise, but one that knows where its noise lives, how large it is, and whether it is acceptable.
