What if the problem is not diagnosable from logs? In a perfect world production and development would be identical, all production issues would be easy to repr

What if the problem is not diagnosable from logs? In a perfect world production and development would be identical, all production issues would be easy to reproduce in development, and logging and error handling would be sufficient to diagnose any problem. But what do you do when you have a problem that doesn't happen in a perfect world -- that you can't diagnose normally, doesn't occur in dev, shows nothing abnormal in logs, etc.? At some point you need the ability to inspect the actual thing. It's like diagnosing a problem with a bridge by doing experiments on a scale model replica. Doesn't always work.