3 ms·
To me, the real insight is the fuzz testing of compilers by automatically generating test cases that don't compare the compiled binaries directly but the result
by tuukkah 7y ago
To me, the real insight is the fuzz testing of compilers by automatically generating test cases that don't compare the compiled binaries directly but the results of executing those binaries instead. This way, they can find bugs in optimizing compilers as well. Also, one of the compilers is formally verified so they have an oracle (a golden reference). EDIT: Plus of course, their method is able to generate test cases where the results are free of any allowed undefined behaviour.
- pfdietz 7y agoThere's a related technique called metamorphic testing where the insight is that the only part of the program being compiled that matters is the part that will be executed to compute that differentiating output. So, the other parts can be mutated (even with undefined behaving code, as those parts are not executed) and differently mutated versions can be tested against one another. This works even when one just has a single compiler.
- kazinator 7y agoAnyone who has bootstrapped a compiler using the interpreter for the same language, and then validated the original interpreted test suite under compilation, has done this. Likewise anyone who boostrapped a new compiler using an existing one. There is nothing earth shattering here. Fuzzing is also not new. The idea of feeding the same input to multiple implementations of the same program side by side is not new, either. Just maybe the combination of all three is what is new.
- tuukkah 7y ago> Just maybe the combination of all three is what is new. Precisely.