4 ms·

Skip your flaky tests should be a religion. There's nothing else I feel as strongly about regarding CI optimization. If a test is flaky, it gets immediately ski

by vasco 3y ago

Skip your flaky tests should be a religion. There's nothing else I feel as strongly about regarding CI optimization. If a test is flaky, it gets immediately skipped. Even if you're working on a fix, it's skipped until you solve it, if you ever do. Most of your CI problems can start to be solved by applying this simple rule. How do you know if it's flaky? You keep a count and any time a test fails/recovers 3 times it gets skipped, even if there's weeks between failures. You can make it more complex for little gain, but I've found this system will have teams actually prioritize fixing important tests, but mostly it has proven that many "important tests to keep even if they are flaky" never were actually important or end up getting re-written in different ways later on.

tumetab1 3y ago
I feel the same, I can't understand why people value more having a flaky test than not having it. In both cases, the feature is not validated (the test failure is ignored), but not having the test is transparent (the feature is not being validated), improves pipeline speed and reduces the noise. Maybe people just like the lie, that the feature is being validated.. I don't really understand.
- deleted 3y ago
  [deleted]
johnbellone 3y ago
Fix the test or remove it.
twic 3y ago
> You keep a count and any time a test fails/recovers 3 times it gets skipped, even if there's weeks between failures. Congratulations, you now have an untested feature. Flaky tests should be fixed, of course. But just deleting them isn't fixing them, and it leaves you worse off, not better. May i ask how you feel about TODO comments?
- vasco 3y ago
  If you re-read my comment you'll see I already addressed this. Not having a test is better than having a flaky test. The team should feel the pain of having their shit test skipped without them being able to stop it and it's up to them to either fix it and bring it back or cope.
  - twic 3y ago
    > Not having a test is better than having a flaky test. What i'm saying is that you are completely wrong about that. Sorry if i wasn't clear.
    - vasco 3y ago
      Ok that's more clear, this is fine then we know we disagree (also why i called it a bit of a religion). Have a great day! PS: if the test was flaky, you already had an untested feature
thiht 3y ago
Or just fix them? A test shouldn't ever, ever, ever be flaky. It can happen because we all make assumptions that can be wrong, or forget about non deterministic behaviours, but when it does a flaky test should be immediately fixed, with the highest priority.
- vasco 3y ago
  That is out of your control across many teams. Your only hope is having hard rules on what to do with flakyness so that teams aren't able to spend 3 months telling you "please don't skip because we'll fix it in our next sprint".
jkaptur 3y ago
I've often been tempted by the "delete your flaky tests" extremist sect of that religion :)
kohsuke 3y ago
This is indeed a religion, because in my experience people tend to feel strongly holding very different positions. You can already see it in this thread. I think quantifying and prioritizing is key, like you wrote. Respected engineering organizations like Google and GitHub all came to the same place. Flakiness is often unevenly distributed, so find & tackle ones that are the worst. Don't try to eliminate the flakiness because that's not economically viable. I'm trying to put my money where my mouth is... we'll see how it goes.

by vasco 3y ago

tumetab1 3y ago

I feel the same, I can't understand why people value more having a flaky test than not having it. In both cases, the feature is not validated (the test failure is ignored), but not having the test is transparent (the feature is not being validated), improves pipeline speed and reduces the noise. Maybe people just like the lie, that the feature is being validated.. I don't really understand.

deleted 3y ago
[deleted]

johnbellone 3y ago

Fix the test or remove it.

twic 3y ago

> You keep a count and any time a test fails/recovers 3 times it gets skipped, even if there's weeks between failures. Congratulations, you now have an untested feature. Flaky tests should be fixed, of course. But just deleting them isn't fixing them, and it leaves you worse off, not better. May i ask how you feel about TODO comments?

vasco 3y ago
If you re-read my comment you'll see I already addressed this. Not having a test is better than having a flaky test. The team should feel the pain of having their shit test skipped without them being able to stop it and it's up to them to either fix it and bring it back or cope.
- twic 3y ago
  > Not having a test is better than having a flaky test. What i'm saying is that you are completely wrong about that. Sorry if i wasn't clear.
  - vasco 3y ago
    Ok that's more clear, this is fine then we know we disagree (also why i called it a bit of a religion). Have a great day! PS: if the test was flaky, you already had an untested feature

thiht 3y ago

Or just fix them? A test shouldn't ever, ever, ever be flaky. It can happen because we all make assumptions that can be wrong, or forget about non deterministic behaviours, but when it does a flaky test should be immediately fixed, with the highest priority.

vasco 3y ago
That is out of your control across many teams. Your only hope is having hard rules on what to do with flakyness so that teams aren't able to spend 3 months telling you "please don't skip because we'll fix it in our next sprint".

jkaptur 3y ago

I've often been tempted by the "delete your flaky tests" extremist sect of that religion :)

kohsuke 3y ago

This is indeed a religion, because in my experience people tend to feel strongly holding very different positions. You can already see it in this thread. I think quantifying and prioritizing is key, like you wrote. Respected engineering organizations like Google and GitHub all came to the same place. Flakiness is often unevenly distributed, so find & tackle ones that are the worst. Don't try to eliminate the flakiness because that's not economically viable. I'm trying to put my money where my mouth is... we'll see how it goes.