The Diminishing Returns of Discovery: Why More Research Yields Fewer Breakthroughs

研究的收益递减:为什么投入越多,突破越少

The curves for researchers, papers, and R&D spending keep climbing; the curves for total factor productivity and major breakthroughs keep sliding. Bloom and his co-authors estimate that sustaining Moore's Law now requires more than eighteen times the researchers it did in the early 1970s, and Jones attributes the pattern to the burden of knowledge. This does not mean we have grown stupider. It more likely means we converted research from a long-horizon institution into a short-cycle project review. What is scarce is not ideas but the room in which anomalies are allowed to exist.

1. Two curves pointing opposite ways

Begin with a fact more uncomfortable than any thesis: for over half a century, the resources poured into research have grown exponentially while the productivity of research itself has steadily fallen.

In "Are Ideas Getting Harder to Find?" (American Economic Review, 2020), Nicholas Bloom, Charles I. Jones, John Van Reenen and Michael Webb supply the number most often quoted: sustaining Moore's Law — a doubling of chip density roughly every two years — now requires more than eighteen times as many researchers as it did in the early 1970s. The wording is restrained. Research effort is rising sharply, research productivity is falling sharply, and total output still grows only because inputs grow faster.

Semiconductors are not a special case. In agriculture the same authors find that average yields of corn, soybeans, wheat and cotton roughly doubled between 1960 and 2015, while the research devoted to those gains rose by a factor of three to more than twenty-five; research productivity fell by roughly 4 to 6 percent a year. At the firm level, more than 85 percent of publicly traded companies show rising R&D spending paired with falling research productivity.

One misreading must be blocked immediately: falling research productivity does not mean "no progress." It means progress is getting more expensive — each unit of it costs more money, more people, more time. The distinction matters, because "progress has stalled" is false, "progress has become costly" is true, and the policy implications diverge completely.

2. First explanation: the burden of knowledge

Benjamin F. Jones offered a structural account in the Review of Economic Studies in 2009, under a title with a grim joke in it: "The Burden of Knowledge and the 'Death of the Renaissance Man.'"

His observation is plain. Innovators are not born at the frontier. A person must spend a decade or more absorbing existing knowledge before standing anywhere near the point where one more step forward is possible. The larger the stock of knowledge, the longer that road.

The consequences are measurable: the age at which people make their first major contribution rises, doctoral training lengthens, single-authored breakthroughs become rarer, and teams grow larger. The "Renaissance man" — one person fluent in several fields and able to cross them — has not vanished so much as been squeezed out by the division of knowledge.

The strength of this account is that it needs no assumption that human intelligence is declining. It says only one thing: the frontier moves further from the starting line, and the cost of reaching it rises over time. If this mechanism dominates, falling research productivity is a necessary by-product of accumulating knowledge, and any "spend more" policy can delay it but never reverse it.

The gap is obvious, though: it explains "expensive," not "fewer." Rising cost can be offset by tools — the microscope, the computer, and now machine learning are all technologies for lowering the cost of reaching the frontier. If the burden of knowledge were the only variable, every tool revolution should have bent the curve upward again. And in some fields, it did.

3. Second explanation: the low-hanging fruit

The more popular story is that the low-hanging fruit has been picked. The metaphor is intuitive, and it smuggles in a premise: that the total quantity of fruit is fixed and the tree is only so large.

But "what can be discovered" is not a fixed stock; it is a space that expands with instruments. Before the telescope, the fruit in the sky did not exist; without X-ray diffraction, the double helix was not on any list waiting to be picked. Every new instrument grows new branches. So "the fruit is gone" holds in fields whose search space is closed, and fails in fields whose search space is still expanding.

The distinction is not rhetorical. Lithography is a case of a narrowing search space: the wavelength has been pushed to extreme ultraviolet, the physical limit is close, and each further step costs an order of magnitude. Computing, biology and materials discovery are cases of a still-expanding space. Lumping them together under "innovation has become harder" produces a false conclusion — that every field should be rescued the same way.

4. The counter: perhaps we are measuring wrong

Before treating "harder" as fact, the strongest counter must be stated: perhaps what declined is our measurement, not the research.

Three specific doubts.

First, total factor productivity misses quality improvements and free goods. Erik Brynjolfsson and others have argued repeatedly that GDP does not count free search engines, maps or encyclopaedias as output, so a large share of the digital economy's consumer surplus falls outside the statistical frame. If output is undercounted, the numerator of productivity looks too small and the decline is exaggerated.

Second, breakthroughs are defined in hindsight. We say "the twentieth century produced so many great breakthroughs" because a hundred years is long enough to filter out what mattered. The last two decades have not been filtered. Comparing "greatness already ratified" with "the present, not yet ratified" guarantees a verdict of decline — survivorship bias and nostalgia on the same ledger.

Third, general-purpose technologies lag. Paul David's classic work on electricity showed that decades passed after the electric motor before factories rearranged their floors and retrained workers, and only then did the productivity dividend arrive. Electrification did this; computing did this; artificial intelligence very likely will. Judging it a failure because ten years have not moved total factor productivity is to use the wrong time scale.

This counter has limits too. Measurement error struggles to explain four independent indicators — crop yields, drug approvals, firm-level sales growth, and R&D per process generation in semiconductors — all deteriorating in the same direction and at the same order of magnitude for forty years. The safer conclusion is therefore not "no decline" but "the decline is real, its size is overstated, and part of it is masked by lag."

5. The real variable: how research is organised

If rising cost is real and lag is real, the remaining question is why some periods and some institutions keep producing under the same burden of knowledge.

The Idea Factory offers half an answer. For a long stretch, Bell Labs did not review by quarter, did not require every project to point at a product, and let a group of clever people spend years on questions that were, for the moment, useless. The transistor, information theory, the laser, Unix and C all came out of that arrangement. What it relied on was not more money but a particular time structure: funding whose horizon is longer than the problem's.

By contrast, the dominant form of contemporary research is the project: a three-to-five-year grant cycle, mid-term review, publication counts, translation metrics. This form is good at incremental work inside an existing paradigm — it defines "deliverable" very clearly. And it is systematically bad at one thing: noticing anomalies.

Kuhn was explicit in The Structure of Scientific Revolutions: normal science solves puzzles within a paradigm, while revolutions begin with anomalies — results the paradigm cannot explain and that most people therefore ignore or force into place. Noticing an anomaly requires two conditions: the slack to look where you are not supposed to look, and the security not to lose your position because three years passed without output. Short-cycle review destroys both at once.

My judgement, then: the main cause of falling research productivity is not that ideas are scarcer, but that the institutional room for anomalies has narrowed. This is not an argument that money is useless; it is an argument that the time scale of money matters more than its quantity. Split the same sum into a hundred three-year projects, or hand it to ten groups allowed eight quiet years, and the structure of what comes out will differ entirely.

6. Judgement: two regimes, two remedies

The limits of this judgement must be admitted. It does not explain lithography: near a physical limit, no funding horizon buys back an order of magnitude in cost. It could also be falsified: if artificial intelligence really drives the cost of reaching the frontier toward zero, the burden-of-knowledge mechanism expires and every gloomy estimate above must be rewritten. That is the variable most worth watching this century, not a settled question.

In practice, then, fields should be sorted into two regimes with two remedies.

Fields whose search space is still expanding (computing, biology, materials discovery): the binding constraint is organisation, not money. The effective moves are longer funding horizons, less weight on short-term output, and a protected set of posts that do not point at products. More money under unchanged short-cycle review most likely produces more publishable and less cumulative work.

Fields near a physical limit (advanced process nodes, battery energy density, aerospace): the binding constraint is physics, not institutions. Diminishing returns here are real. The right response is to accept them and move resources to alternative paths — new material systems, new architectures, new principles — rather than piling more people onto the old one.

Return, finally, to those two curves. They do not prove that humanity has grown stupider, nor that progress is over. They prove one thing: when a society treats research as a production activity measurable by the quarter, what it buys is necessarily more papers and fewer surprises. And surprise is the one output no KPI can pre-order.

Related books