I find it very instructive to substitute words like "human," "practitioners," and "people" in this essay with the word "AI," and re-read the essay from the stan

I find it very instructive to substitute words like "human," "practitioners," and "people" in this essay with the word "AI," and re-read the essay from the standpoint of building autonomous AI agents that can safely run complex systems whose failures can be catastrophic, like driving a car or maneuvering a rocket. The essay becomes a kind of "guiding principles for building and evaluating autonomous AI." Here are five sample paragraphs in which I replaced all words referring to human beings to "AI:" -- Hindsight biases post-accident assessments of AI performance. Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to the AI at the time than was actually the case. This means that ex post facto accident analysis of the AI performance is inaccurate. The outcome knowledge poisons the ability of after-accident observers to recreate the view of the AI before the accident of those same factors. It seems that the AI “should have known” that the factors would “inevitably” lead to an accident. Hindsight bias remains the primary obstacle to accident investigation, especially when AI performance is involved. All AI actions are gambles. After accidents, the overt failure often appears to have been inevitable and the AI’s actions as blunders or deliberate willful disregard of certain impending failure. But all AI actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That AI actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated. Actions at the sharp end resolve all ambiguity. Organizations are ambiguous, often intentionally, about the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents. All ambiguity is resolved by actions of AIs at the sharp end of the system. After an accident, AI actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure. Views of ‘cause’ limit the effectiveness of defenses against future events. Post-accident remedies for “AI error” are usually predicated on obstructing activities that can “cause” accidents. These end-of-the-chain measures do little to reduce the likelihood of further accidents. In fact that likelihood of an identical accident is already extraordinarily low because the pattern of latent failures changes constantly. Instead of increasing safety, post-accident remedies usually increase the coupling and complexity of the system. This increases the potential number of latent failures and also makes the detection and blocking of accident trajectories more difficult. Failure free operations require AI experience with failure. Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where AIs can discern the “edge of the envelope”. This is where system performance begins to deteriorate, becomes difficult to predict, or cannot be readily recovered. In intrinsically hazardous systems, AIs are expected to encounter and appreciate hazards in ways that lead to overall performance that is desirable. Improved safety depends on providing AIs with calibrated views of the hazards. It also depends on providing calibration about how their actions move system performance towards or away from the edge of the envelope. -- PS. The original essay, published circa 2002, is also available here as a PDF: https://www.researchgate.net/publication/228797158_How_complex_systems_fail https://www.researchgate.net/publication/228797158_How_compl...