3 ms·
I have seen that peoples response to randomized testing hinges on the answer to one failing question: Does the test log the input that led to a failure? If the
by mumblemumble 6y ago
I have seen that peoples response to randomized testing hinges on the answer to one failing question: Does the test log the input that led to a failure?
If the answer is no, then it's an immediate slam on the brakes. Tests aren't reproducible, they're flaky, this is impossible, etc.
If the answer is yes, then it's a revelation. The tests are finding me corner and edge cases I never would have thought of on my own, and I can enshrine and document them by creating a fixed-data unit test to cover them even before I start to work the bug (I am still doing TDD, after all), and hey, maybe I don't even have to think that much about the specific data, just its shape, and that will keep me honest and protect me from accidentally writing tautological tests. A short while later, you discover property testing, start achieving ever increasing test coverage with ever decreasing test code, have some sort of satori moment, tell your friends what you've discovered, and they tell you you're insane, randomize tests are flaky by definition, and flaky tests are Taboo. You slink away and change your legal name to Jonathan Livingston Tester.