4 ms·

> One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a dif

> One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a different outcome). Just records the input whenever the test fails, and have an option to re-run a test, with those inputs. The tests are still flaky but it doesn't matter if we can reproduce the results. You don't even have to record all inputs. You could just use a random seed, and record the seed at each run. Test become deterministic again simply by fixing the seed (typically to a value that caused the previous run to fail).

gregmac 6y ago
Using a seed with the "random" is exactly how you get to pseudo-random, which gives the same sequence everytime, and is absolutely fine for tests. If you want to have the build (test) result feedback to save the input, that's an ok goal but I think it's hard to pull off: now you have your build making commits to source control (causing another build), or you have test data stored somewhere else not in source (when really, it should be part of source).