3 ms·
> A test with a pattern of failing intermittently over its entire lifetime is most likely a flaky test. I'm not GP, but I think they meant something more radic
by kubanczyk 6y ago
> A test with a pattern of failing intermittently over its entire lifetime is most likely a flaky test.
I'm not GP, but I think they meant something more radical.
In CI, if a test is seen to fail and then succeed on the same git commit for the first time, that's it. One time is enough to classify it as "flaky". No more lifetime, no more patterns of interesting behavior. The test is removed and thrown into backlog.
- hinkley 6y agoI might paint with a broader brush. If your commit breaks master, we revert the whole change and you try again. When the old stuff breaks, there's a bunch of other decisions built on top of it and there is no clear path to unwind the changes. Between HEAD and HEAD-4? Yank it.
- viraptor 6y agoIf you have a problem with testing infrastructure/code, that will start disabling random tests until either you're left with nothing, or realise that radical moves are rarely practical. (With a side of distrust from any team you send the work to)