3 ms·
I'm not intimately familiar with "fuzzing" techniques, and by "randomized inputs", I am indeed referring to tests that could be flaky in the presence of bugs. O
by whack 6y ago
I'm not intimately familiar with "fuzzing" techniques, and by "randomized inputs", I am indeed referring to tests that could be flaky in the presence of bugs. Organizational discipline is indeed needed here, in order to not ignore the occasional flaky failure, and to investigate the cause of the flaky error. I believe the benefits outweigh the drawbacks.
Here's a link that talks about the above in more detail - https://software.rajivprab.com/2019/04/28/rethinking-software-testing-perspectives-from-the-world-of-hardware/ https://software.rajivprab.com/2019/04/28/rethinking-softwar...
- josephg 6y agoI use fuzzers extensively in software I write. I find it quite enjoyably humbling how quickly a newly minted fuzzer finds bugs. I use a seeded random number generator. During development I seed it randomly. When crashes happen I output both the seed and iteration so I can hardcode that seed and reproduce the problem. And then in the CI I’ll often fix the seed so the test suite is stable. But it’s also good practice to run fuzzers outside the context of a CI. You want your CI tests to be fast and stable. Fuzzer tests can also be slow and thorough. Foundationdb has dedicated hardware that runs their fuzzer 24/7 and spits out any errors it finds. If that fuzzer finds problems it doesn’t break the build. But someone will take a look, reproduce the problem, hopefully convert the bug into a unit test and fix the issue.