4 ms·
The article presents two forms of testing, but it implies checking is only possible with a complete specification of the program. But that's not true: one can
by pfdietz 3y ago
The article presents two forms of testing, but it implies checking is only possible with a complete specification of the program.
But that's not true: one can check incomplete specifications, both positive and negative.
To the extent one has any specification, one can automate testing, and in doing so execute far more test cases than any manual exploration ever could. To a significant extent, testing is about sheer volume of tests, not cleverness in constructing the tests.
- tveyben 3y ago> To a significant extent, testing is about sheer volume of tests Well, quantity != quality Just because you have 1E10 test cases does not mean you have a large coverage… One cleverly constructed testcase might return more insights regarding the product than 100 testcases made just to increase the number of test cases. Goodhart's law: "When a measure becomes a target, it ceases to be a good measure" comes to mind…
- pfdietz 3y agoQuantity has a quality all its own. Time and again, testing complex programs, I've seen this approach find bugs one could not have conceived when creating manual tests. You mention coverage. Interestingly, if you read the Csmith paper on random testing of C compilers, they found that while it found plenty of bugs, it didn't increase coverage (over previous tests) more than a tiny amount. Coverage is a great example of Goodhart's law.
- drewcoo 3y ago> it implies checking is only possible with a complete specification of the program No, it doesn't. If exploratory testing has any value at all, that implies that there cannot be complete test coverage with checks, thus there cannot be checks matching a complete specification. > incomplete specifications, both positive and negative There's more to testing than positive and negative test cases. > execute far more test cases than any manual exploration ever could Exploratory testing does not "execute test cases," so you have a tautology here. > testing is about sheer volume of tests The purpose of testing is open to wildly contradictory opinions. I usually claim that it's to find user-impacting bugs before users do. Maybe that can happen with lots of checks and maybe those can be automated. Maybe you're dealing with a system with lots of hidden interdependencies and some "I'm not gonna touch that" dev code and even if you want to check everything you just can't. Maybe the user experience matters more and that goes far beyond positive/negative test cases. There's no simple one-size-fits-all solution.
- omgwtfbyobbq 3y agoIn my experience, manual and automated testing are necessary together, and neither is sufficient alone. Manual testing alone usually can't cover enough ground, and automated testing alone usually can't generate inputs with sufficient cardinality to cover enough of what the software does.
- Jtsummers 3y agoProperty-based and fuzz testing can generate a lot of inputs. You may have to run them overnight or longer sometimes though. And if you pair property-based testing with model-based testing you can detect a lot of errors that manual testing will likely never discover, just due to time. https://hypothesis.readthedocs.io/en/latest/stateful.html https://hypothesis.readthedocs.io/en/latest/stateful.html - one example
- drewcoo 3y agoThere are ways to avoid running all night, which I assume is a side-effect of combinatoric explosion of the test space. Check out pairwise (a.k.a. "all pairs") testing: https://en.wikipedia.org/wiki/All-pairs_testing https://en.wikipedia.org/wiki/All-pairs_testing https://www.pairwise.org https://www.pairwise.org And there are also ways to (at runtime) prune the set of cases being run based on system behavior. As in "aha - these similar things failed to meet expectations in the same way, let's flag it as the same bug and stop running anything with the failure characteristic," if that makes sense. Say, for example, you have a zip code field that takes a string and after several instances of tests with a zip code your test determines that you're getting a NaN error (for a string field?) . . . stop running any tests with zip code strings and treat all of those as the same failure. This can greatly increase test speed and make triage of failures easier.
- omgwtfbyobbq 3y agoNice! That would perfect for fully automated testing if a user didn't need to specify any primitive actions. I'm sure it still saves a ton of time/catches many bugs.