Yes but how will they do that if they don't have a clear understanding. When we build software, we have (or should have) a clear understanding of the various co

Yes but how will they do that if they don't have a clear understanding. When we build software, we have (or should have) a clear understanding of the various components and, in some cases, like with distributed and mission-critical/military systems, a formal verification/simulation of the system when needed. When we're dealing with emergent behavior, as we have with these large transformers, but no exact understanding of how the behavior is produced and only a limited way of refining/controlling it, I don't think we're in a position to guaratee that refinements in one area won't lead to regressions in other areas or a change in the global characteristics of the system. I mean... we're dealing with complex emergent behavior, at a different scale of complexity than what we have had to deal with so far (in traditional software development) and no mature verification/analysis tools.