3 ms·
Randomizing tests in any way is good and useful in theory, but am I the only one finding that it ends-up being counter-productive in practice? What I found ove
by Seb-C 5y ago
Randomizing tests in any way is good and useful in theory, but am I the only one finding that it ends-up being counter-productive in practice?
What I found over time is that is erodes the trust of the team in the test suite. When you work on feature A, the last thing you want is for the tests of unrelated features B and C to randomly fail and prevent merging your work.
So what becomes the norm is re-running the tests until is passes rather than understanding it and fixing it.
Over time, such issues accumulate and it becomes harder and harder to be lucky enough for it to pass.
While the right thing to in theory do would be to always fix it, it's not always possible depending on the organization of the team and tasks, and being bothered by unrelated stuff during a specific task is just annoying.
That said, this package could be useful if pseudo-random (maybe it is, I didn't look so much in details).
- icholy 5y agoWhen a bad input is found, it gets saved to your corpus and gets checked on all subsequent runs. So re-running the test won't make it go away.
- Seb-C 5y agoWhich contravenes a fundamental good practice of testing: Each test should run independently and be stateless. Also, this would not work with most (all?) CI runs (which are inherently stateless), and working around this would be complex and introduce a lot of negative side effects.
- eru 5y agoWell, the alternative point of view is to see that when you have an infinite number of tests, you can't run them all, and need to sample them. (Saving the seeds to the corpus still leaves your tests independent and stateless. You are just sampling from your space of all possible tests a bit more intelligently in future.)