> We should probably be doing modelling and formal verification and stuff like that. This is interesting. Is this a hypothetical notion or can you share any pe

> We should probably be doing modelling and formal verification and stuff like that. This is interesting. Is this a hypothetical notion or can you share any personal experience of having worked on any projects where this is done successfully? I bring this up because there is typically a narrow band of applicability of purely theoretical constructs in most complex systems (AWS & Azure teams both use formal methods [1] to find bugs and to increase confidence in critical algorithms but I don't believe they do this system-wide. Also the verification is only done on the abstraction, not the actual implementation.). Richard Cook gave a talk on "How Complex Systems Fail" [2] which addresses this point, and advocates for designing for resiliency as opposed to designing to exhaustively handle all failure modes (which are impossible to predict). Design people often design from some idealized notion of inputs and/or worst case scenarios. Ops people know what can go wrong will go wrong, which is why ops is there to handle the deviations from the norm. Real world design exposes lift points, reveals controls and supports continuous maintenance. Highly reliable systems are evolved, not just designed. [1] https://en.wikipedia.org/wiki/TLA%2B#Industry_use https://en.wikipedia.org/wiki/TLA%2B#Industry_use [2] https://www.youtube.com/watch?v=2S0k12uZR14 https://www.youtube.com/watch?v=2S0k12uZR14