4 ms·

Shouldn't the developers be falling over themselves to fix all those flaky unit tests? I can't believe Google just lets them keep running unfixed. In my code, I

Shouldn't the developers be falling over themselves to fix all those flaky unit tests? I can't believe Google just lets them keep running unfixed. In my code, I don't check anything in unless all unit tests pass (without flaking).

Buge 8y ago
How many times do you run the tests? I've had tests that only fail about 1/200 times or less.
- slavik81 8y ago
  It gets pretty brutal when you have 10,000 tests that each have a 1/1000 chance of failing. The worst was that some tests were actually wrong and only passed due to nondeterminism. Once I made the tests deterministic, I discovered they always failed. Until that point, I found those test failures very confusing, because it seemed like my efforts to improve the reliability of the tests was having no effect on them.
- brianberns 8y ago
  If someone showed me that my tests were not deterministic, I'd be in a hurry to figure out why. Hidden race conditions? Unreliable initial state of the system? Unexpected interactions between tests? I'm just surprised that this article doesn't mention any effort to find the root cause and fix it.
  - SpicyLemonZest 8y ago
    The challenge isn’t when your tests are flaky. The challenge is when a test written by some guy who left the company begins flaking, and your preliminary investigation shows it’s a timing issue two layers below you in the stack, but the people who own that project insist you’re using it wrong and your test shouldn’t pass in the first place. It’s hard to sell why your team should drop everything to untangle that knot.
    - tacostakohashi 8y ago
      These are all just excuses, and common ones at that. Think of it this way: * The test cases written by the some guy who left the company are now your test cases. * Before he left the company, he was able to commit the test cases, passing whatever review / quality processes were put in place by the company (or that the company failed to put in place). * If your team wants to have the benefit of reliable testcases, the knot needs to be untangled. That doesn't entail "dropping everything" (although perhaps thats the only way your organization knows how to operate), there is no reason that improving test cases can't be prioritized alongside and in the context of other work. Unfortunately, in my experience it's usually the last bullet point where things fall down. Product owners consistently pressure development teams to deliver features now, and do test cases later, which means never.
      - JoeAltmaier 8y ago
        Its all about competition. You move fast and deliver rapidly, or get left behind. Time is actually money, for an engineer at least. So unless those tests are needed now, it always seems better to de-prioritize them.
      - SpicyLemonZest 8y ago
        The last bullet point often should fall down. There are a lot of tests that aren’t worth the effort of making them run reliably, but most organizations have (understandably) strong norms against fixing a failing test by removing it.
  - Buge 8y ago
    Any test with a normally non-hit timeout anywhere is nondeterministic, because the operating system could schedule your test very slowly causing it to hit the timeout.
- deathanatos 8y ago
  I once only had a test that failed Saturday evenings. We didn't notice it for the longest time because generally, people don't check in code then, because they're not working. The test dealt with a "week", and on Saturday evening, the current week in UTC differed from that of the local timezone.
sqldba 8y ago
A previous workplace has thousands of tens of thousands of tests and about 2-3% fail per build. I don’t know why the developers don’t care but I suspect they’ve learned to ignore specific tests which repeatedly fail. Like in most development places management doesn’t give a flying shit about all of the tests passing if it means a feature will ship. Probably they’re mostly older tests and so fixing them detracts from adding a new fancy feature
idunno246 8y ago
I’ve never worked on a large(ten+ of devs including people that left) project that didn’t have flakes. They’re hard to fix. If you can’t reliably reproduce then you don’t even know if you that it’s fixed or you’re just lucky. They’re hard to find. The change that they first appear tends not to be related to the cause. Leads to a lot of “not my problem” Really that’s just lack of ownership. Ultimately who owns the entire codebase. You say “my code” but I see a lot of people thinking that means the files I edit, not everything someone has to explicitly decide to spend day(s) fixing tests, and then only might fix a couple. The roi is super frustrating and it’s hard to spend whole days on a single test. There’s a ton of broken window theory. Once you get used to hitting rebuild, you just always hit rebuild and don’t even know it’s a new flake. The most effective thing I’ve seen when you hit this point is get all the test data, junit xml say, somewhere queryable and make graphs and explicit parts of okr with time against it.

Buge 8y ago

How many times do you run the tests? I've had tests that only fail about 1/200 times or less.

slavik81 8y ago
It gets pretty brutal when you have 10,000 tests that each have a 1/1000 chance of failing. The worst was that some tests were actually wrong and only passed due to nondeterminism. Once I made the tests deterministic, I discovered they always failed. Until that point, I found those test failures very confusing, because it seemed like my efforts to improve the reliability of the tests was having no effect on them.
brianberns 8y ago
If someone showed me that my tests were not deterministic, I'd be in a hurry to figure out why. Hidden race conditions? Unreliable initial state of the system? Unexpected interactions between tests? I'm just surprised that this article doesn't mention any effort to find the root cause and fix it.
- SpicyLemonZest 8y ago
  The challenge isn’t when your tests are flaky. The challenge is when a test written by some guy who left the company begins flaking, and your preliminary investigation shows it’s a timing issue two layers below you in the stack, but the people who own that project insist you’re using it wrong and your test shouldn’t pass in the first place. It’s hard to sell why your team should drop everything to untangle that knot.
  - tacostakohashi 8y ago
    These are all just excuses, and common ones at that. Think of it this way: * The test cases written by the some guy who left the company are now your test cases. * Before he left the company, he was able to commit the test cases, passing whatever review / quality processes were put in place by the company (or that the company failed to put in place). * If your team wants to have the benefit of reliable testcases, the knot needs to be untangled. That doesn't entail "dropping everything" (although perhaps thats the only way your organization knows how to operate), there is no reason that improving test cases can't be prioritized alongside and in the context of other work. Unfortunately, in my experience it's usually the last bullet point where things fall down. Product owners consistently pressure development teams to deliver features now, and do test cases later, which means never.
    - JoeAltmaier 8y ago
      Its all about competition. You move fast and deliver rapidly, or get left behind. Time is actually money, for an engineer at least. So unless those tests are needed now, it always seems better to de-prioritize them.
    - SpicyLemonZest 8y ago
      The last bullet point often should fall down. There are a lot of tests that aren’t worth the effort of making them run reliably, but most organizations have (understandably) strong norms against fixing a failing test by removing it.
- Buge 8y ago
  Any test with a normally non-hit timeout anywhere is nondeterministic, because the operating system could schedule your test very slowly causing it to hit the timeout.
deathanatos 8y ago
I once only had a test that failed Saturday evenings. We didn't notice it for the longest time because generally, people don't check in code then, because they're not working. The test dealt with a "week", and on Saturday evening, the current week in UTC differed from that of the local timezone.

sqldba 8y ago

A previous workplace has thousands of tens of thousands of tests and about 2-3% fail per build. I don’t know why the developers don’t care but I suspect they’ve learned to ignore specific tests which repeatedly fail. Like in most development places management doesn’t give a flying shit about all of the tests passing if it means a feature will ship. Probably they’re mostly older tests and so fixing them detracts from adding a new fancy feature

idunno246 8y ago

I’ve never worked on a large(ten+ of devs including people that left) project that didn’t have flakes. They’re hard to fix. If you can’t reliably reproduce then you don’t even know if you that it’s fixed or you’re just lucky. They’re hard to find. The change that they first appear tends not to be related to the cause. Leads to a lot of “not my problem” Really that’s just lack of ownership. Ultimately who owns the entire codebase. You say “my code” but I see a lot of people thinking that means the files I edit, not everything someone has to explicitly decide to spend day(s) fixing tests, and then only might fix a couple. The roi is super frustrating and it’s hard to spend whole days on a single test. There’s a ton of broken window theory. Once you get used to hitting rebuild, you just always hit rebuild and don’t even know it’s a new flake. The most effective thing I’ve seen when you hit this point is get all the test data, junit xml say, somewhere queryable and make graphs and explicit parts of okr with time against it.