2 ms·
That is a point. It might even be true. But showing a small model an example of vulnerable code and asking to confirm that it is vulnerable code isn't evidence
by tptacek 7mo ago
That is a point. It might even be true. But showing a small model an example of vulnerable code and asking to confirm that it is vulnerable code isn't evidence for that point!
- danieltanfh95 7mo agoNo, it is evidence for that point. You could just rattle off every possible vulnerability and have the cheap model scan for it in the harness through a loop. Note that I say cheap, not small, because small models may lack the reasoning needed, but some models are cheap enough but retain enough reasoning (ala Sonnet 3.7+)
- tptacek 7mo agoThat's not what they did.
- roywiggins 7mo agoThey could write a post demonstrating that you can do that and surface the same bugs in the same codebases. It would be way more informative than this one, which didn't do that.