被取消的练习：为什么自动化越成功，人的失误越致命
The classic goal of automation is to remove the human from the system. What gets removed is the easy half; what remains is the hardest part — monitoring, diagnosis, takeover. All three can only be learned by doing, and doing is precisely what automation abolishes. A six-page paper from 1983 and a stall in 2009 point to the same design error
I. Three and a Half Minutes
Shortly after midnight on 1 June 2009, Air France 447 left Rio de Janeiro for Paris. The aircraft was an Airbus A330-203 carrying 216 passengers and 12 crew.
Entering the intertropical convergence zone, the cold, moisture-laden air froze the pitot probes. Airspeed readings failed. The autopilot disconnected, exactly as designed: it cannot fly an aircraft whose speed it does not know. Control returned to the cockpit.
What followed is reconstructed line by line in the French aviation accident bureau's final report of July 2012. The pilot flying pulled the sidestick back. The aircraft climbed, speed decayed, and it stalled. From autopilot disconnection to the surface of the Atlantic: about three and a half minutes.
One detail is worth pausing on. The stall warning did sound. But when the nose reached fifteen degrees and forward speed fell below sixty knots, the warning stopped — angle-of-attack data is no longer valid at that speed. By the time speed data returned and the warning resumed, the pilot faced a cockpit arguing with itself: an alarm that had just fallen silent, a set of contradictory readings, and a night sky with no ground reference anywhere in it.
I raise this not to re-litigate an accident. The bureau's conclusions, the judicial proceedings, the two competing reports — none of that is the subject here. The subject is a narrower question: the device that was supposed to tell the human "you are failing" depended on the very channel that was failing.
That is a structural property of automation, not a software defect at one company.
II. The Easy Half Is What Gets Removed
In 1983, Lisanne Bainbridge, a psychologist at University College London, published a six-page paper in Automatica, volume 19, number 6, titled "Ironies of Automation." It has since become one of the most cited documents in human factors engineering.
Her starting point is a claim that reads like a paradox: the more advanced a control system is, the more crucial the contribution of the human operator may become.
This is not rhetoric. She offers two lines of reasoning.
The first concerns the designer's intention. The classic goal of automation is to remove the human from the system — operators are unreliable, they tire, they lose attention, so they should be replaced. But what the designer removes is only the portion he could figure out how to automate. The tasks he could not figure out remain with the human. Hence the first irony: a design intended to eliminate the operator ends up assigning the operator the job of handling whatever the designer did not anticipate — with almost no thought given to supporting that job.
The second concerns the nature of the remaining work. Automation, Bainbridge writes, takes away the easy parts of the task and makes the difficult parts more difficult.
That sentence deserves unpacking. Automation absorbs steady-state operation: holding a temperature, maintaining a heading, welding at cadence. What remains is three things — monitoring (is the system still behaving), diagnosis (if not, where), and takeover (retrieve control and stabilise). These are structurally different skills, but they share one property: all three can only be acquired by doing them repeatedly, and repeated doing is precisely what automation abolishes.
III. Why Practice Cannot Be Taught
Bainbridge separates skill into three layers, each with its own mechanism of decay.
The first is physical skill. She cites Edwards and Lees' 1974 volume, The Human Operator in Process Control: an experienced operator making a set-point change uses the fewest possible actions and moves the process output smoothly and quickly to the new level; an inexperienced one oscillates around the target. The difference is not knowledge but feel — a sense of gain and timing. And this decays when unused, quickly. An operator who was once skilled and now merely supervises an automated plant is, in practice, an unskilled operator. If he takes over, he may throw the whole process into turmoil.
The second is cognitive skill. Long-term knowledge is retrieved in proportion to how often it is used — her example is disarmingly plain: think of any subject you have not touched since passing the exam. Working memory is worse still: on-line decisions rest not on raw data but on the operator's predictions and judgments about the current state of the process, and those take time to accumulate. She records a concrete number: manual operators typically arrive in the control room a quarter of an hour to half an hour before taking over, in order to get a feel for what the process is doing. A takeover that must happen immediately can only be made on the thinnest possible base of information.
The third is monitoring, and this layer is physiologically impossible. She cites Mackworth's vigilance research from the 1950s: even a highly motivated person cannot sustain effective visual attention to a rarely occurring signal for more than about half an hour. This means the very job design of "human as monitor of rare anomalies" violates basic human parameters from the outset. Her recommendation is to let automatic alarm systems carry that function — and then she immediately asks: when the alarm system fails, who notices? She mentions an even more absurd practice: requiring operators to keep a periodic log. People can copy down numbers while paying no attention whatsoever to what the numbers are.
Then comes the sharpest line in the paper. If a computer is used to make decisions precisely because human judgment and intuitive reasoning are inadequate in these situations, on what basis is the human supposed to judge whether the computer's decision is "acceptable"? Her conclusion: the human monitor is given an impossible task.
And finally the irony in its literal sense: it is the most successful automated systems — the ones that rarely need manual intervention — that may require the greatest investment in human operator training. The less intervention is needed, the less chance there is to practise; and when intervention is finally needed, it will be the hardest case there is.
IV. Three Objections
The argument is easily read as "therefore automate less." That is a misreading, and it drains the argument of its force. At least three objections deserve serious weight.
First, the data objection. Automation makes systems safer overall. The fatal accident rate in commercial aviation has fallen by more than an order of magnitude over recent decades, and that decline coincided with increasing automation. Bainbridge does not deny this. What she denies is that the benefit is evenly distributed. Automation shifts risk from frequent small errors to rare large ones: everyday slips get absorbed by the system, but the remaining case arrives when the human is least prepared. The expected cost of rare large errors may still be far below that of frequent small ones — so "automate less" is not the conclusion. "Redistribute the practice" is.
Second, the attribution objection. Norman argues throughout The Design of Everyday Things that most of what gets called "human error" is design that failed to account for human limits. Calling AF447 "pilot error" is technically accurate and explanatorily almost useless: those two pilots were not incompetent, they were making judgments in a cockpit whose readings contradicted one another, whose alarm fell silent at the decisive moment, and in which neither had ever practised a stall recovery at that altitude. This objection weakens the "humans are unreliable" narrative — and it equally weakens the proposed remedy of "hire better-trained people." The problem is not the quality of the person; it is the design of the interface and the handover.
Third, the full-automation objection. If humans cannot monitor, then stop asking them to. In many domains this is right: anti-lock brakes, protective relays on power grids, the inner control loops of fly-by-wire. Human withdrawal is a net gain. But it has a precondition — the anomalies must be enumerable. As long as a system can still meet situations its designers did not imagine, something must be able to handle the unimagined. Simon offers a related observation in The Sciences of the Artificial: the path of an ant looks complex, but the ant is simple; the complexity resides largely in the environment. By the same logic, an operator looks competent largely because the environment — here, the automation — is carrying most of the work. Remove the environment and the remaining person immediately shows how simple he always was. And handling the unimagined is precisely the one function that cannot be trained in advance. It appears only in people who have been repeatedly shaped by real situations.
V. Write the Practice Into the Design
So the real question is not whether the human belongs in the loop. It is three design parameters that can be checked one at a time.
Criterion one: anomaly frequency. If a class of anomaly occurs once a year, do not design a post for a "skilled takeover operator" — that person does not exist. The design must assume an unskilled taker: the interface should let a stranger build situational awareness in thirty seconds, rather than letting an expert build it faster. These two requirements frequently conflict, and most systems choose the second.
Criterion two: the takeover window. Bainbridge draws a clear line. Failures that unfold in seconds — certain transients in a pressurised water reactor — leave no time for a human, and must be answered automatically whatever the cost. Failures measured in minutes can be met by over-learned manual responses that buy time. Only failures measured in tens of minutes give a person room to get into a state. The window determines whether the human belongs in the loop, not the reverse.
Criterion three: whether the feedback is legible. She identifies an easily overlooked mechanism: automatic control can disguise failure. By adjusting the controlled variables, it absorbs the anomaly, so the trend remains invisible until it is pushed outside the control range. The object of monitoring therefore cannot be "has a limit been crossed" but "is the movement of the variables itself unusual." This holds for software as much as for process plants: a service that quietly retries and quietly degrades will hide its failure until the moment it can hide it no longer.
None of this is aviation archaeology. When an agent takes over retrieval, drafting, and first-pass screening, what remains to the human is judging whether the output is right — the very skill with the fewest opportunities to practise. The pattern Christian documents in The Alignment Problem is worth keeping: systems optimise precisely the objective that was written down, and whatever was not written down quietly disappears. Russell's direction in Human Compatible is correct — a machine should not hold definite beliefs about human preferences but remain uncertain and defer when the human is dissatisfied. But that solution moves the burden to the same place: the human must be able to express dissatisfaction. And by what right does someone who never practises judgment know that he is dissatisfied?
The limits deserve stating too. This argument holds only for settings that are infrequent, high-consequence, and slow to give feedback. Where events are frequent, consequences small, and feedback immediate — spell-check, route planning, recommendation ranking — the net benefit of automation is unambiguous, and human withdrawal is a good thing, not a loss to be mourned. And because that boundary is so clear, this is a design question, not a position on the proper relation between people and machines.
Bainbridge closes her paper with restraint: she hopes it has shown that automation does not necessarily eliminate these difficulties. Forty-three years on, the scope of that sentence has widened from the control room to knowledge work entire. What gets cancelled was never the job. It was the practice. And practice, once cancelled, does not come back at the moment it is needed.
