3 ms·
If there are enough tests then flakiness is unavoidable. The key is to separate "flaky test" from "actual failure" so that you can react quickly to actual and i
by alkonaut 3y ago
If there are enough tests then flakiness is unavoidable. The key is to separate "flaky test" from "actual failure" so that you can react quickly to actual and ignore flakiness or at least not address it as urgently.
For 99% of software projects this isn't even necessary however. If the number of tests is resonable (say less than a few hours) then you can probably do better with "fix all tests immediately, including addressing flakiness".
But for the large project that e.g. tests several 100k tests across a matrix of different platforms, then you need to resort to the "we'll never reach 100% but we can't dip below 99%" mindset. You need to analyze whether that failure in a PR validation build is actually reasonable to expect from the area that was changed, and if it isn't, then you need to merge it despite the validation not passing. Otherwise it'll never be merged. Modularization helps somewhat, but the small tests that stay within modules aren't the ones that will cause the problems.