3 ms·

> In my opinion, an ideal test case only terminates if a mistake is found. If not, it will simply keep on searching, potentially for months, years, ... You need

> In my opinion, an ideal test case only terminates if a mistake is found. If not, it will simply keep on searching, potentially for months, years, ... You need to be able to start the search for mistakes and keep it running for as long as you want. That's a good point, but the fundamental problem is that any sufficiently complex space takes way more than a practical amount of time to be completely traversed (easily reaching or surpassing age-of-the-universe magnitudes), so isn't that where fuzzy testing comes in? If you use something like Prolog, isn't it going to perform the search in an ordered manner, drastically reducing the chances that a hard-to-spot mistake is found? And at least in the general case, don't you have to restart the search from scratch every time logic changes?

zhd 5y ago
These are basically the problems that https://hypofuzz.com https://hypofuzz.com is designed to solve. - You write standard property-based tests with Hypothesis, which fit into a traditional unit-test workflow (run with pytest, or unittest, or whatever) - then HypoFuzz runs indefinitely, using feedback-guided fuzzing to find bugs that random generation is too slow for - restarting the search on each new commit (or daily, etc) begins by replaying every distinguishable input found earlier so you don't have to start from scratch. There is _some_ loss of context, but that's inherent in testing different code. Plus other nice stuff like allocating more compute time to tests which are finding new behaviour faster, for maximum efficiency. The roadmap includes exploiting SAT solvers to find very very rare behaviour, git history to focus on recently changed code, statistical risk analysis ("what's the expected time to next bug"), and also a budget-mode ("my compute costs $. 08/hr, stop when you expect a bug to cost $100 or more to find"). It's a good time to be working on testing!

zhd 5y ago

These are basically the problems that https://hypofuzz.com https://hypofuzz.com is designed to solve. - You write standard property-based tests with Hypothesis, which fit into a traditional unit-test workflow (run with pytest, or unittest, or whatever) - then HypoFuzz runs indefinitely, using feedback-guided fuzzing to find bugs that random generation is too slow for - restarting the search on each new commit (or daily, etc) begins by replaying every distinguishable input found earlier so you don't have to start from scratch. There is _some_ loss of context, but that's inherent in testing different code. Plus other nice stuff like allocating more compute time to tests which are finding new behaviour faster, for maximum efficiency. The roadmap includes exploiting SAT solvers to find very very rare behaviour, git history to focus on recently changed code, statistical risk analysis ("what's the expected time to next bug"), and also a budget-mode ("my compute costs $. 08/hr, stop when you expect a bug to cost $100 or more to find"). It's a good time to be working on testing!