3 ms·
I don't understand why we are getting LLMs to generate code to create fuzzing data as a 'thing' Logically LLMs should be quite good at creating the fuzzing dat
by aaron695 3y ago
I don't understand why we are getting LLMs to generate code to create fuzzing data as a 'thing'
Logically LLMs should be quite good at creating the fuzzing data.
To state the obvious why, it's too expensive to use LLMs directly and this way works since they found "4 memory safety bugs and one hang"
But the future we are heading to should be LLMs will directly pentest/test the code. This is where it's interesting and new.
- moyix 3y agoI don't think using a language model to generate inputs directly is ever going to be as efficient as writing a little bit of code to do the generation; it's really hard to beat an input generator that can craft thousands of inputs/second.
- yencabulator 3y agoFor one, it'd be really hard for an LLM to get the CRC32 right, especially when it's in a header before the data it covers. Then again, this whole approach to fuzzing comes across as kinda naive, at the very least you'd want to use an API of a coverage-guided fuzzer for generating the randomness (and then almost always fixing up CRC32 on top of that, like a human-written wrapper function would).
- dmazzoni 3y agoExactly. If I actually wanted to fuzz this I'd use libfuzzer and manually fix the crc32. An LLM would be useful in helping me write the libfuzzer glue code.