2 ms·

Not contradicting this (I am sure it's true), but why is using an LLM for this qualitatively better than using an actual fuzzer?

• saagarjha 7mo ago
  Presumably because people have used actual fuzzers and not found these bugs.
• utopiah 7mo ago
  I didn't even read the piece but my bet is that fuzzers are typically limited to inputs whereas relying on LLMs is also about find text patterns, and a bit more loosely than before while still being statistically relevant, in the code base.
• azakai 7mo ago
  1. This is a kind of fuzzer. In general it's just great to have many different fuzzers that work in different ways, to get more coverage. 2. I wouldn't say LLMs are "better" than other fuzzers. Someone would need to measure findings/cost for that. But many LLMs do work at a higher level than most fuzzers, as they can generate plausible-looking source code.
• mmis1000 7mo ago
  It's not really bad or not though. It's a more directed than the rest fuzzer. While being able to craft a payload that trigger flaw in deep flow path. It could also miss some obvious pattern that normal people don't think it will have problem (this is what most fuzzer currently tests)
• hrmtst93837 7mo ago
  [flagged]
• bvisness 7mo ago
  As someone on the SpiderMonkey team who had to evaluate some of Anthropic's bugs, I can definitely say that Anthropic's test cases were definitely far easier to assess than those generated by traditional fuzzers. Instead of extremely random and mostly superfluous gibberish, we received test cases that actually resembled a coherent program.