3 ms·
This note from the article jumped out at me: > While engineers tend to trust passing test results, they often retry failing tests a number of times on the same
by boulos 6y ago
This note from the article jumped out at me:
> While engineers tend to trust passing test results, they often retry failing tests a number of times on the same version of code and consider failures followed by passing results as flaky. We do not have a good theoretical explanation for why this behavior prevails.
which seems to me like standard confirmation bias.
When writing a test or updating code, you clearly assume you got it right. So a passing test “confirms” that.
Personally, I find myself quite susceptible to this bias when updating code (i.e., when I haven’t broken any tests, I assume I updated my code correctly) but I’m much more cautious when writing new tests. I usually prefer to write a failing test, and get it to pass, to mitigate some of that bias (but fully admit that once it turns green, I’m overly confident at the “clearly correct” nature of the output).