4 ms·

In my experience, manual and automated testing are necessary together, and neither is sufficient alone. Manual testing alone usually can't cover enough ground,

In my experience, manual and automated testing are necessary together, and neither is sufficient alone. Manual testing alone usually can't cover enough ground, and automated testing alone usually can't generate inputs with sufficient cardinality to cover enough of what the software does.

Jtsummers 3y ago
Property-based and fuzz testing can generate a lot of inputs. You may have to run them overnight or longer sometimes though. And if you pair property-based testing with model-based testing you can detect a lot of errors that manual testing will likely never discover, just due to time. https://hypothesis.readthedocs.io/en/latest/stateful.html https://hypothesis.readthedocs.io/en/latest/stateful.html - one example
- drewcoo 3y ago
  There are ways to avoid running all night, which I assume is a side-effect of combinatoric explosion of the test space. Check out pairwise (a.k.a. "all pairs") testing: https://en.wikipedia.org/wiki/All-pairs_testing https://en.wikipedia.org/wiki/All-pairs_testing https://www.pairwise.org https://www.pairwise.org And there are also ways to (at runtime) prune the set of cases being run based on system behavior. As in "aha - these similar things failed to meet expectations in the same way, let's flag it as the same bug and stop running anything with the failure characteristic," if that makes sense. Say, for example, you have a zip code field that takes a string and after several instances of tests with a zip code your test determines that you're getting a NaN error (for a string field?) . . . stop running any tests with zip code strings and treat all of those as the same failure. This can greatly increase test speed and make triage of failures easier.
- omgwtfbyobbq 3y ago
  Nice! That would perfect for fully automated testing if a user didn't need to specify any primitive actions. I'm sure it still saves a ton of time/catches many bugs.

Jtsummers 3y ago

Property-based and fuzz testing can generate a lot of inputs. You may have to run them overnight or longer sometimes though. And if you pair property-based testing with model-based testing you can detect a lot of errors that manual testing will likely never discover, just due to time. https://hypothesis.readthedocs.io/en/latest/stateful.html https://hypothesis.readthedocs.io/en/latest/stateful.html - one example

drewcoo 3y ago
There are ways to avoid running all night, which I assume is a side-effect of combinatoric explosion of the test space. Check out pairwise (a.k.a. "all pairs") testing: https://en.wikipedia.org/wiki/All-pairs_testing https://en.wikipedia.org/wiki/All-pairs_testing https://www.pairwise.org https://www.pairwise.org And there are also ways to (at runtime) prune the set of cases being run based on system behavior. As in "aha - these similar things failed to meet expectations in the same way, let's flag it as the same bug and stop running anything with the failure characteristic," if that makes sense. Say, for example, you have a zip code field that takes a string and after several instances of tests with a zip code your test determines that you're getting a NaN error (for a string field?) . . . stop running any tests with zip code strings and treat all of those as the same failure. This can greatly increase test speed and make triage of failures easier.
omgwtfbyobbq 3y ago
Nice! That would perfect for fully automated testing if a user didn't need to specify any primitive actions. I'm sure it still saves a ton of time/catches many bugs.