3 ms·
There are two improvements that I think can be made to quick-check style testing. One is replacing the random source with bytes from a buffer and having a way t
by dan-robertson 4y ago
There are two improvements that I think can be made to quick-check style testing. One is replacing the random source with bytes from a buffer and having a way to go from test cases to bytes. Hypothesis (python) does this. This means that you can connect a fuzzer like afl instead of generating purely random inputs. Another is setting up test-runners. If you have automated tests as part of CI, you want a fixed seed and deterministic ‘random’ inputs and a small number of runs so tests are fast and reliable. You can still catch obvious bugs but less likely bugs are harder to find. But if you can easily find all the tests and run them all the time with many more different inputs, hopefully you’ll have a much higher chance of finding rare bugs.
One problem though is that being better at finding bugs isn’t always great: if you find bugs you might feel the desire to fix them but for many software teams, having rare bugs is acceptable, even if they aren’t rare in absolute terms (ie a 1-in-10,000,000 when you have a billion chances for it to happen a day)