5 ms·
How does your zero tolerance policy work when you can't find a reason the rest fails? I.e. you have critical functionality you test, but fail one test run in 10
by viraptor 6y ago
How does your zero tolerance policy work when you can't find a reason the rest fails? I.e. you have critical functionality you test, but fail one test run in 1000. You can spend 2 days on the issue and not find a solution.
Sinking more time into it doesn't really help anyone much and won't offset any time saving in a realistic future. On the other hand, if you actually remove the test, you risk messing up a critical transaction.
- maccard 6y agoIf you have a critical function failing testing, you need to figure out the root cause, whether it takes 2 days or 2 months. 2 days is nothing - I've spent more than that on UI polish bugs, to not be able to dedicate 5x that time to fixing critical functionality, something is desperately wrong.
- majkinetor 6y agoViraptor is right. Your "solution" is far from unversally acceptable. If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective. I hate the flaky tests just as much as a next dude but seriously, you need to factor in everything - deadlines, features, CI/CD, metrics etc, while still having constant time frame. So lets not get into such polar opinions, its just not realistic at all. Life is never that ordered.
- mpweiher 6y agoIf you came to me with that attitude, I would fire you immediately. This is, per definition, critical functionality, and we can't figure out whether it is working or not, and you are saying you don't care.
- maccard 6y ago> If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective. This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality. If I were _your_ lead and you told me you fired someone because they spent two months on something, I'd probably fire you for not knowing what your team are doing for two months.
- viraptor 6y ago> This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality. It may be unreliable critical functionality, or unreliable test. In some cases it's obvious which one it is, in some it isn't. If you know it's the test, it becomes much less beneficial to spend days on it.
- majkinetor 6y agoI didn't say fire, didn't I, I said "show you the dor", like, get out from my office :P And seriously, if you fire someone for having a strong opinion, you should remind yourself that even little kids tollerate a far worse behavior.
- pavel_lishin 6y agoIf it's acceptable for your critical functionality to fail 0.1% of the time in production, then sure, leave it in.
- viraptor 6y agoThere's difference between a flakey test and test failing due to randomly failing feature. They're not always the same thing. The test failing does not necessarily mean the feature doesn't work.
- majkinetor 6y agoMeh. Majority of time, in my experience, flaky tests has nothing to do with the software but it is about faulty test infrastructure.