4 ms·
> To a significant extent, testing is about sheer volume of tests Well, quantity != quality Just because you have 1E10 test cases does not mean you have a lar
by tveyben 3y ago
> To a significant extent, testing is about sheer volume of tests
Well, quantity != quality
Just because you have 1E10 test cases does not mean you have a large coverage…
One cleverly constructed testcase might return more insights regarding the product than 100 testcases made just to increase the number of test cases.
Goodhart's law: "When a measure becomes a target, it ceases to be a good measure" comes to mind…
- pfdietz 3y agoQuantity has a quality all its own. Time and again, testing complex programs, I've seen this approach find bugs one could not have conceived when creating manual tests. You mention coverage. Interestingly, if you read the Csmith paper on random testing of C compilers, they found that while it found plenty of bugs, it didn't increase coverage (over previous tests) more than a tiny amount. Coverage is a great example of Goodhart's law.