We rely on many layers of more or less leaky abstractions. Components have always a set of intended behavior and some, often unknown or poorly understood, set

We rely on many layers of more or less leaky abstractions. Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior. It's very important to understand that abstractions mask both intended and unintended behavior. You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through. Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system. Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs. > So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have. Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake? You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.