3 ms·
Well, there’s ups and downs. In theory, we should be able to automatically generate tests for all of our software. There are methods that include path-based, co
by nickpsecurity 3y ago
Well, there’s ups and downs. In theory, we should be able to automatically generate tests for all of our software. There are methods that include path-based, combinatorial, symbolic, model-driven, etc. While some options are generic (eg range test on primitive types), most tests need a good understanding of what is correct vs incorrect behavior.
That brings us to the other problem: software specification. We can’t even know if software is correct if we don’t precisely define what correct means. Same with “secure.” So, we need specifications of each property we want to check. Then, we can use a variety of methods, including testing, to check those. Developers usually don’t specify the correctness conditions. The tools for doing so aren’t great for developers either.
Enter fuzzing. It can do something similar to path-based and combinatorial testing with no user guidance. Many failures cause crashes or other obviously bad behavior. That gets value on software without specifications.
Even if we specify things, we might not get all the conditions right or every module covered. Maybe a developer updates code without updating the specification. While model-based testing is good due to human effort, fuzzing will catch what they missed by not depending on human efforts.
So, there’s better methods to test software from both an efficiency and accuracy standpoint. Yet, the labor involved plus room for human error makes fuzzing a valuable tool in the toolbox. There’s no shame in using it.
- pfdietz 3y ago> That brings us to the other problem: software specification. We can’t even know if software is correct if we don’t precisely define what correct means. However if we have software that is mostly correct, and if the bugs are unpredictable and specific to particular implementations, we can still find bugs without a specification by using differential testing: using two or more separate implementations, identify inputs on which their behavior differs. (This requires the software be deterministic.) Places where they differ will not tell us which of the two is wrong, but that can be determined manually once the discrepancy is found.