3 ms·
He isn't using black box testing, he's using white box testing (knowing Greg, either KLEE [1] or AFL [2]... probably both... and probably also black box testing
by derf_ 12y ago
He isn't using black box testing, he's using white box testing (knowing Greg, either KLEE [1] or AFL [2]... probably both... and probably also black box testing). The test does incorporate a great deal of knowledge of the algorithms (in fact, complete knowledge), but the point is that it's the machine that does the analysis of the implementation, not a human.
[1] https://klee.github.io/ https://klee.github.io/
[2] http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
- nullc 12y agoThough I use both klee and AFL, this particular instance was found with blackbox randomized tests; using numbers with low transition probabilities. E.g. 00000111111111000111000000000000. Klee or AFL would be somewhat unlikely to find this particular problem because the error was an omitted branch, so they wouldn't know there was a case left to be satisfied.
- derf_ 12y agoRight, one of the weaknesses of tools that rely on coverage analysis... and highlights the importance of using more than one approach to testing. No single tool can do everything.