3 ms·

Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.

• rectang 6y ago
  But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.
  • slaymaker1907 6y ago
    We actually have different classes of tests to allow for some more flaky tests. You definitely don't want to run those flaky tests after every build, but you should be able to eventually get a run where those tests pass before handing it off to customers.
    • marshmellman 6y ago
      Interesting. Doesn’t this imply that the behavior itself is flaky?
  • ddevault 6y ago
    Right - but you fix the bug. You don't add a tolerance.
  • hinkley 6y ago
    They say that contempt is the beginning of the end of the Rule of Law, which is why you should be careful not to pass frivolous laws. Tests I've found are much the same way. You don't write one flaky test and stop. If you write one and everybody is okay with it, you and your coworkers write more, and more, until there are 50, 100. Once the suite flakes out on an interval, nobody takes a failed test seriously, and then broken code doesn't get checked for hours. One broken test? I bet it's the usual. I'll just rerun it a couple of times. The thread you're pulling on here starts to unravel the whole Continuous Integration sweater.