4 ms·
What I want is a tool to make flaky tests fail reliably. They won't be fixed until they start actually preventing commits. If somebody deletes a test, that is
by ncmncm 5y ago
What I want is a tool to make flaky tests fail reliably.
They won't be fixed until they start actually preventing commits. If somebody deletes a test, that is on that person. I don't want a tool automatically suppressing testing.
- ankitdce 5y agoThat's very interesting feedback. We certainly don't have a way to force simulate failure. A related capability we are working on is to also rerun the identified flaky tests X times so they pass. This depends on the capabilities of the test runner, so it will work with specific ones first (cypress, pytest, etc). That way you still make sure that flaky tests pass instead of supressing.
- ncmncm 5y agoIt should run the flaky tests N times, and report failure if any run fails. Hiding failures is not doing us a favor.
- berkay 5y agoWhat if it was easy to see why a test is flaky, compare failed/successful test runs like a code diff? Would that be useful? This is what we're building at Thundra (foresight product), instrument the tests as well as backend services to enable devs to quickly diagnose failing/flaky tests. Would appreciate any feedback you may have, here or privately.
- ncmncm 5y agoIt would be helpful to be able to present diffs of log output between successful and failing runs of a test. This is tricky to implement, for several reasons. Log output is normally timestamped, making every line unique. Those parts of log lines would need to be ignored when comparing between runs. Log output ordering is often indeterminate, particularly when a test has multiple threads, or interacts with an external service. Often the order of events logged is an essential feature of the difference between a successful and failed run. But some or most order differences are just incidental. The number of logged events may vary incidentally, or significantly. Explaining all these differences in detail to the test system would be too hard. So, the system needs to discover as much as possible of this for itself, and represent these discoveries symbolically. Then, allow a test to be annotated to override default judgments about the diagnostic significance of these features.