3 ms·
I doubt many people intentionally write flaky tests. In developers I've worked with I've observed a lack of understanding of the language/libraries used (ie. t
by Normal_gaussian 6y ago
I doubt many people intentionally write flaky tests.
In developers I've worked with I've observed a lack of understanding of the language/libraries used (ie. the order of something is coincidental, not guaranteed, leading to flakiness) and a lack of understanding of the test system (eg. shared db for multiple test runners, querying last insert is not deterministic in this case) as well as accidental mistakes (tests timeout after 1000ms, enough I/O is in the test that it can vary from 700ms to (rarely) 1200ms.
Recently I wrote a test that checked the deterministic properties of a build. I forgot that the two test builds could run in parallel and thus end up with the same timestamp (to the ms) and thus the same artifact. It never occurs on my dev laptop where it was a single threaded build, but it occurs maybe one in a hundred times on my desktop. Simple to fix, but easy to write. The fix was of course to mock the timestamp properly.