4 ms·
Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale. A 1 in 10,000 failure can be a dail
by Swizec 3y ago
Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale.
A 1 in 10,000 failure can be a daily annoyance for your users even with just 100 daily active users who each make 10 actions on your app. At “internet scale” a 1 in 10k error frustrates a user every few seconds.
If your tests are so flaky that you need SLOs … your poor users …
- drewcoo 3y ago> Every flakey test is a production failure that really happens to some users sometimes No. Tests can be flakey when the code they're testing is not. Every flakey test is not a customer problem. Training developers to ignore flakey tests also trains them to ignore real failures. Potentially many of them. Any of those might be real customer problems.
- m3ll0 3y agoThis! Flaky tests become a customer problem when developers learn to ignore them! You gotta weed them out.
- zarroboogs 3y ago> Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale. Not really, the one I’ve seen most often is a shared resource failing - for example a Gitlab Runner not handling a new VM for a DB so tests fail etc
- sixstringtheory 3y agoAlso, dirty test harness state not being cleaned up between tests/suites, and threading issues in the test code
- vlovich123 3y agoDepending on the problem domain, a lot of times the flakiness is in the way the test is written and not the code under test. You have to judge the relative cost of tracking down and solving that.