This sounds like a case of a system so complex that the humans trying to reason about were unable to do so. As a software engineer I've sometimes had to run a p

This sounds like a case of a system so complex that the humans trying to reason about were unable to do so. As a software engineer I've sometimes had to run a piece of software to properly understand how it works. This doesn't seem like a desirable property for systems that are critical for safety.