5 ms·

How many times do you run the tests? I've had tests that only fail about 1/200 times or less.

• slavik81 8y ago
  It gets pretty brutal when you have 10,000 tests that each have a 1/1000 chance of failing. The worst was that some tests were actually wrong and only passed due to nondeterminism. Once I made the tests deterministic, I discovered they always failed. Until that point, I found those test failures very confusing, because it seemed like my efforts to improve the reliability of the tests was having no effect on them.
• brianberns 8y ago
  If someone showed me that my tests were not deterministic, I'd be in a hurry to figure out why. Hidden race conditions? Unreliable initial state of the system? Unexpected interactions between tests? I'm just surprised that this article doesn't mention any effort to find the root cause and fix it.
  • SpicyLemonZest 8y ago
    The challenge isn’t when your tests are flaky. The challenge is when a test written by some guy who left the company begins flaking, and your preliminary investigation shows it’s a timing issue two layers below you in the stack, but the people who own that project insist you’re using it wrong and your test shouldn’t pass in the first place. It’s hard to sell why your team should drop everything to untangle that knot.
    • tacostakohashi 8y ago
      These are all just excuses, and common ones at that. Think of it this way: * The test cases written by the some guy who left the company are now your test cases. * Before he left the company, he was able to commit the test cases, passing whatever review / quality processes were put in place by the company (or that the company failed to put in place). * If your team wants to have the benefit of reliable testcases, the knot needs to be untangled. That doesn't entail "dropping everything" (although perhaps thats the only way your organization knows how to operate), there is no reason that improving test cases can't be prioritized alongside and in the context of other work. Unfortunately, in my experience it's usually the last bullet point where things fall down. Product owners consistently pressure development teams to deliver features now, and do test cases later, which means never.
      • JoeAltmaier 8y ago
        Its all about competition. You move fast and deliver rapidly, or get left behind. Time is actually money, for an engineer at least. So unless those tests are needed now, it always seems better to de-prioritize them.
      • SpicyLemonZest 8y ago
        The last bullet point often should fall down. There are a lot of tests that aren’t worth the effort of making them run reliably, but most organizations have (understandably) strong norms against fixing a failing test by removing it.
  • Buge 8y ago
    Any test with a normally non-hit timeout anywhere is nondeterministic, because the operating system could schedule your test very slowly causing it to hit the timeout.
• deathanatos 8y ago
  I once only had a test that failed Saturday evenings. We didn't notice it for the longest time because generally, people don't check in code then, because they're not working. The test dealt with a "week", and on Saturday evening, the current week in UTC differed from that of the local timezone.