There’s one particular domain where this question runs deep - tests that verify parts of distributed systems. Note: please do not extrapolate any problem descri

There’s one particular domain where this question runs deep - tests that verify parts of distributed systems. Note: please do not extrapolate any problem described below outside of distributed systems. For instance in Kubernetes we run lots of tests that verify parts of the system actually working together, not just mocked. A fair amount of the time, once tests are in and soaked, the test flake is actually the canary that says things like “no, the kernel regressed networking AGAIN” or “no, your retry failure means that etcd lost quorum - which shouldn’t have happened, what on earth is going on, oh, someone suddenly started hotlooping and now the api server is performing 10k writes a second”. At some point, testing the system in isolation is not enough because all failures are interactions of multiple components. Your flakes are the evidence that the gigantic stack of software known as “modern computing run from HEAD lol” is full bugs. It’s definitely a culture change - once anyone anywhere thinks “oh it’s just flaky” they stop treating it like signal. Once they treat it like noise, it’s very hard to unwind. Today, when I look at one of those test suites and see a flake, I assume that everything in open source is broken, again, and pull out the shovel, because 75% of the time it is.