沉默的证据：为什么我们总是从幸存者身上学习，然后学错
Every methodology of success rests on a sample that has already been filtered by the outcome. The problem is not that we cannot see the failures, but that we rarely notice the filtering itself — and so mistake the traits of survivors for causes
1. The Planes That Came Back
In 1943 the Statistical Research Group at Columbia University was handed a very concrete problem: add armour to bombers. Armour is heavy, so it can only go on the parts that matter. Which parts?
The military brought clean data: the distribution of bullet holes on aircraft that had returned. Most on the wings, fewer along the fuselage, very few near the engines and fuel tanks. Read straight off the table, the answer was obvious — reinforce where the holes are densest.
Abraham Wald concluded the opposite: reinforce where the holes are scarcest — engines, fuel tanks, cockpit.
His reasoning was a single sentence long: you are only looking at the planes that came back. A bomber hit in the engine did not return, so its holes were never recorded. The sparsely punctured areas on surviving aircraft were not areas where hits were unlikely; they were areas where a hit removed the aircraft from the page. Wald wrote this up formally ("A Method of Estimating Plane Vulnerability Based on Damage of Survivors," Statistical Research Group, 1943, CRC 432, reprinted by the Center for Naval Analyses in 1980).
The story is retold in management training constantly, usually as an example of "thinking outside the box." That retelling discards the part that matters. Wald did not have a clever intuition. He did something slower and harder: he built an estimation model that used the visible damage distribution to infer the invisible survival function. He did not guess the answer; he treated the absence itself as data and modelled it.
That is the question this essay takes up: when we learn from experience, what exactly do the observations that never entered the sample take away with them — and under what conditions can we still learn something true from a broken sample?
2. Three Kinds of Censoring, Only One of Which Is Tractable
Statistics has a name for this: censoring. It does not mean the data are wrong. It means some observations were never recorded, for a reason. The reason matters, because there are two fundamentally different kinds.
The first is random censoring. In a five-year clinical trial, some patients move away in year three and are lost to follow-up. Provided the loss is unrelated to the disease, this is manageable: Kaplan-Meier estimation, for instance, keeps such patients in the risk set for as long as they were known to be alive rather than simply deleting them. Random censoring costs precision, not validity.
The second is informative censoring: the probability of being missing depends on the outcome. Funds close, companies delist, founders quit, stores shut — they vanish from the database, and the reason they vanish is that they did badly. Deleting them does not shrink the sample; it systematically removes the lower half of the distribution. Every mean, growth rate and survival rate computed on what remains is inflated.
There is a third kind, easiest to miss: present but silent. A customer who does not complain is not a satisfied customer. An employee who does not object has not agreed. A reader who gives no feedback has not understood. This kind leaves no trace at all, because the missing observations are still inside your sample — their information was simply never collected.
All three point at the same hinge, and this is the load-bearing claim of the whole argument:
Missingness is not the problem. Missingness correlated with the outcome is the problem.
How incomplete a sample is matters far less than whether who disappeared and why they disappeared are entangled with the outcome you want to explain. If they are, a larger sample will not save you. You do not have a small sample. You have a sample that the outcome selected.
3. Why Survivor Stories Are So Persuasive
If survivorship bias were merely a statistical nuisance, it would not be this stubborn. It is stubborn because three cognitive forces reinforce it at once.
The first is narrative completeness. Survivors come with a full story: a beginning, a turning point, decisions, results. Failures come as fragments — the company is gone, the people do not want to talk, the records are scattered. The more complete the story, the more it resembles an explanation; the fragments do not form a story, so they never enter the discussion. What we discuss is therefore never "all cases" but "cases that have a story."
The second is availability. Visible cases capture attention, and attention is scarce. In an information-rich environment this gets worse rather than better: the successes that can be told are told repeatedly, while the failures that cannot be told never get a first telling.
The third layer is subtler: survivor narratives arrive with the appearance of causality. Because it succeeded, every one of its features is read as a cause. Phil Rosenzweig, in The Halo Effect, sorts the errors of this literature into three families: reliance on secondhand accounts, which imports the halo; reversed causality (the company succeeded first and thereby earned the luxury of focus, rather than focusing its way to success); and "organizational physics," the habit of treating correlation as causation.
Jim Collins's Good to Great is a good specimen for testing all three. Published in 2001, covering 1965 to 1995, it selects eleven companies that made the leap and pairs each with a comparison company in the same industry facing similar opportunities that did not. That design is far more careful than most books of its kind — it genuinely does compare. But the trouble sits further upstream: "great" is defined by the outcome. You first pick the winners by stock performance, then go looking for what they share. The shared traits are guaranteed to exist. That is a logical necessity, not a finding. It is post-hoc stratification.
Steven D. Levitt of the University of Chicago ran a direct test: a portfolio built from that list in 2001, the year of publication, would have underperformed the S&P 500. Of the eleven, only Nucor substantially beat the market; Abbott and Wells Fargo did acceptably. Circuit City, on the list, filed for bankruptcy protection in 2008 and liquidated the following year; Fannie Mae was taken into government conservatorship in September 2008.
This is not an accusation of dishonesty. It is a demonstration that when the sample is selected by the outcome, methodological rigour cannot repair the shelf life of the conclusion. The book describes a set of companies that were once great. The reader receives a recipe for greatness.
By contrast, The Outsiders studies eight chief executives against contemporaneous peers in the same industries and measures per-share value rather than revenue size — a design that dodges outcome-based selection at the outset. 100 Baggers studies stocks that returned a hundredfold, and its difficulty is not carelessness: the hundredfold threshold is the outcome. By definition, only companies that lived to that day are on the list.
4. The Counterargument: Censoring Is Not Always Fatal
Stop here and the conclusion would be "no book about success can be trusted." That conclusion is too strong, and it is wrong. At least three objections have real force.
First, Wald is himself the counterexample. Every observation he used came from survivors, and he reached the correct conclusion. Informative censoring is therefore not impassable — provided you write the missingness mechanism into the model and infer the shape of the unseen from the shape of the seen. That is mathematisable work, not mysticism. What is fatal is not censoring but not knowing that you are censoring.
Second, random censoring is entirely tractable. Clinical trials, equipment lifetimes, insurance claims — these fields deal with missing data daily, and they do it not by demanding completeness but by stating the assumption that missingness is independent of the outcome, then estimating under it. Missing data is acceptable. Missing assumptions are not.
Third, learning from survivors has independent value. It establishes that a feasible solution exists. You do not need to know why every attempt failed in order to know that a path can be walked; one counterexample kills "impossible," and one positive case kills "unworkable." This is an existence claim — weaker than a causal claim, but cheap, robust, and often the only thing available at the moment of decision. Demanding a complete sample before learning anything produces paralysis — and in history, war and entrepreneurship, where failure data are often unobtainable in principle, refusing to learn is worse than learning badly.
So the defence holds, but it has a precondition: all three objections depend on the missingness mechanism being explicitly statable. The moment the mechanism is unknown and correlated with the outcome, no causal claim built on survivors survives. What the counterargument wins is "do not despair," not "anything goes."
5. A Judgement: Three Tests
From this, three operational tests follow. Applied to any book about success, any management framework, or your own post-mortem on a decision.
One: is the missingness correlated with the outcome? If the reason for disappearance is the very result you want to explain — poor performance so the fund closed, losses so the trader quit, failure so no memoir was written — then the sample was selected by the outcome, and every rate you compute is a rate among survivors.
Two: is there an independent failure sample? Note the wording. Not "are there failures," but "were the failures' data collected the same way." Recalling a few counterexamples afterwards does not count, because the counterexamples you can recall have themselves been filtered. This step requires design, not memory.
Three: is your claim existential or causal? "Someone did this and succeeded" is an existence claim: weak, usually safe. "Because they did this, they succeeded" is a causal claim: strong, and it requires a comparison, an exclusion, and time. Presenting the first as the second is how most business narrative goes wrong.
Pass all three and what you hold is still only a correlation that held in a world which no longer exists. That is not pessimism; it is accurate cost accounting. Survivorship bias is not a bias that can be eliminated. It is the structural price of learning from finite experience.
The workable response is not to pretend the correction has been made, but to adjust the strength of the conclusion accordingly — to downgrade "this will work" into "this once worked, in a sample selected by the outcome, with the failures unverifiable." That sentence is far less forceful than the first. It also carries more information, and it is true.
Humility here is not a temperament. It is a technique: acknowledge that your sample was chosen, then make the strength of your claim match the strength of the sample. Fail at that, and the more you learn, the more uniformly wrong you become.
