3 ms·

I've had mixed results. I find that agents can be great for: 1. Producing new tests to increase coverage. Migrating you to property testing. Setting up fuzzing

I've had mixed results. I find that agents can be great for: 1. Producing new tests to increase coverage. Migrating you to property testing. Setting up fuzzing. Setting up more static analysis tooling. All of that would normally take "time" but now it's a background task. 2. They can find some vulnerabilities. They are "okay" at this, but if you are willing to burn tokens then it's fine. 3. They are absolutely wrong sometimes about something being safe. I have had Claude very explicitly state that a security boundary existed when it didn't. That is, it appeared to exist in the same way that a chroot appears to confine, and it was intended to be a security boundary, but it was not a sufficient boundary whatsoever. Multiple models not only identified the boundary and stated it exists but referred to it as "extremely safe" or other such things. This has happened to me a number of times and it required a lot of nudging for it to see the problems. 4. They often seem to do better with "local" bugs. Often something that has the very obvious pattern of an unsafe thing. Sort of like "that's a pointer deref" or "that's an array access" or "that's `unsafe {}`" etc. They do far, far worse the less "local" a vulnerability is. Product features that interact in unsafe ways when combined, that's something I have yet to have an AI be able to pick up on. This is unsurprising - if we trivialize agents as "pattern matchers", well, spotting some unsafe patterns and then validating the known properties of that pattern to validate is not so surprising, but "your product has multiple completely unrelated features, bugs, and deployment properties, which all combine into a vulnerability" is not something they'll notice easily. It's important to remain skeptical of safety claims by models. Finding vulns is huge, but you need to be able to spot the mistakes.

mozdeco 7mo ago
[work at Mozilla] I agree that LLMs are sometimes wrong, which is why this new method here is so valuable - it provides us with easily verifiable testcases rather than just some kind of analysis that could be right or wrong. Purely triaging through vulnerability reports that are static (i.e. no actual PoC) is very time consuming and false-positive prone (same issue with pure static analysis). I can't really confirm the part about "local" bugs anymore though, but that might also be a model thing. When I did experiments longer ago, this was certainly true, esp. for the "one shot" approaches where you basically prompt it once with source code and want some analysis back. But this actually changed with agentic SDKs where more context can be pulled together automatically.
- kwanbix 7mo ago
  Please, implement "name window" natively in Firefox. I have to use chrome because the lack of it.
  - janpio 7mo ago
    https://addons.mozilla.org/en-GB/firefox/addon/window-titler/ https://addons.mozilla.org/en-GB/firefox/addon/window-titler... works great.
    - kwanbix 7mo ago
      Sort of. It won't be save between machines, for example, as chrome's implementation does. If Firefox crashes, most of th time it is lost. It is also not as clean as chrome's native implementation. I have tried it. This has been requested since 2022: https://connect.mozilla.org/t5/ideas/user-defined-name-for-each-firefox-window-on-my-desktop/idi-p/6736#comments https://connect.mozilla.org/t5/ideas/user-defined-name-for-e...
- staticassertion 7mo ago
  My point is that "verifiable testcases" works great for proving "this is vulnerable" but LLMs are still risky if you believe "this is safe", which you can't easily prove. My point is that you need to be very skeptical of when they decide that something isn't vulnerable. I completely agree that LLMs are great when instructed to provide provable, repeatable exploits. I have done this multiple times and uncovered some neat bugs. > I can't really confirm the part about "local" bugs anymore though, but that might also be a model thing. I don't think it's a model thing, it's just a sort of basic limitation of the technology. We shouldn't expect LLMs to perform novel tasks so we shouldn't expect LLMs to find novel vulnerabilities. Agents help, human in the loop is critical for "injecting novelty" as I put it. The LLM becomes great at producing POCs to test out.
octoclaw 7mo ago
[dead]
rithdmc 7mo ago
Security has had pattern matching in traditional static analysis for a while. It wasn't great. I've personally used two AI-first static analysis security tools and found great results, including interesting business logic issues, across my employers SaaS tech stack. We integrated one of the tools. I look forward to getting employer approval to say which, but that hasn't happened yet, sadly.
StilesCrisis 7mo ago
This description is also pretty accurate for a lot of real-world SWEs, too. Local bugs are just easier to spot. Imperfect security boundaries often seem sufficient at first glance.
nitwit005 7mo ago
I've seen fairly poor results from people asking AI agents to fill in coverage holes. Too many tests that either don't make sense, or add coverage without meaningfully testing anything. If you're already at a very high coverage, the remaining bits are presumably just inherently difficult.
- staticassertion 7mo ago
  I suppose it's mixed results but a coverage report should give you "these exact lines are uncovered" and it becomes pretty straightforward to see "ah yeah that error condition isn't tracked, the behavior should be X, go write that test".
  - nitwit005 7mo ago
    That's what people tried right? It'd be great if the AI never failed at tasks, but they clearly do sometimes.
delaminator 7mo ago
But you're not a member of Anthropic's Red Team, with access to a specialist version of Claude.
- staticassertion 7mo ago
  I don't think that matters at all.
  - delaminator 7mo ago
    I think that Anthropic's own version of Claude will give them different results than the ones you get. "Find zero-day exploits in this popular software." I haven't tried it I suspect that the guardrails will make a difference.
    - staticassertion 7mo ago
      I don't think so. I've never had Claude reject the idea of finding a vulnerability (unlike ChatGPT). The issue is that it's limited by its training set. It'll be trained on things like UAF, it won't be trained on things like "the way your secrets are injected + the way you make HTTP requests + the way you deploy means that an SSRF can expose your private key" or whatever, and that's a technology limitation.

mozdeco 7mo ago

[work at Mozilla] I agree that LLMs are sometimes wrong, which is why this new method here is so valuable - it provides us with easily verifiable testcases rather than just some kind of analysis that could be right or wrong. Purely triaging through vulnerability reports that are static (i.e. no actual PoC) is very time consuming and false-positive prone (same issue with pure static analysis). I can't really confirm the part about "local" bugs anymore though, but that might also be a model thing. When I did experiments longer ago, this was certainly true, esp. for the "one shot" approaches where you basically prompt it once with source code and want some analysis back. But this actually changed with agentic SDKs where more context can be pulled together automatically.

kwanbix 7mo ago
Please, implement "name window" natively in Firefox. I have to use chrome because the lack of it.
- janpio 7mo ago
  https://addons.mozilla.org/en-GB/firefox/addon/window-titler/ https://addons.mozilla.org/en-GB/firefox/addon/window-titler... works great.
  - kwanbix 7mo ago
    Sort of. It won't be save between machines, for example, as chrome's implementation does. If Firefox crashes, most of th time it is lost. It is also not as clean as chrome's native implementation. I have tried it. This has been requested since 2022: https://connect.mozilla.org/t5/ideas/user-defined-name-for-each-firefox-window-on-my-desktop/idi-p/6736#comments https://connect.mozilla.org/t5/ideas/user-defined-name-for-e...
staticassertion 7mo ago
My point is that "verifiable testcases" works great for proving "this is vulnerable" but LLMs are still risky if you believe "this is safe", which you can't easily prove. My point is that you need to be very skeptical of when they decide that something isn't vulnerable. I completely agree that LLMs are great when instructed to provide provable, repeatable exploits. I have done this multiple times and uncovered some neat bugs. > I can't really confirm the part about "local" bugs anymore though, but that might also be a model thing. I don't think it's a model thing, it's just a sort of basic limitation of the technology. We shouldn't expect LLMs to perform novel tasks so we shouldn't expect LLMs to find novel vulnerabilities. Agents help, human in the loop is critical for "injecting novelty" as I put it. The LLM becomes great at producing POCs to test out.

octoclaw 7mo ago

[dead]

rithdmc 7mo ago

Security has had pattern matching in traditional static analysis for a while. It wasn't great. I've personally used two AI-first static analysis security tools and found great results, including interesting business logic issues, across my employers SaaS tech stack. We integrated one of the tools. I look forward to getting employer approval to say which, but that hasn't happened yet, sadly.

StilesCrisis 7mo ago

This description is also pretty accurate for a lot of real-world SWEs, too. Local bugs are just easier to spot. Imperfect security boundaries often seem sufficient at first glance.

nitwit005 7mo ago

I've seen fairly poor results from people asking AI agents to fill in coverage holes. Too many tests that either don't make sense, or add coverage without meaningfully testing anything. If you're already at a very high coverage, the remaining bits are presumably just inherently difficult.

staticassertion 7mo ago
I suppose it's mixed results but a coverage report should give you "these exact lines are uncovered" and it becomes pretty straightforward to see "ah yeah that error condition isn't tracked, the behavior should be X, go write that test".
- nitwit005 7mo ago
  That's what people tried right? It'd be great if the AI never failed at tasks, but they clearly do sometimes.

delaminator 7mo ago

But you're not a member of Anthropic's Red Team, with access to a specialist version of Claude.

staticassertion 7mo ago
I don't think that matters at all.
- delaminator 7mo ago
  I think that Anthropic's own version of Claude will give them different results than the ones you get. "Find zero-day exploits in this popular software." I haven't tried it I suspect that the guardrails will make a difference.
  - staticassertion 7mo ago
    I don't think so. I've never had Claude reject the idea of finding a vulnerability (unlike ChatGPT). The issue is that it's limited by its training set. It'll be trained on things like UAF, it won't be trained on things like "the way your secrets are injected + the way you make HTTP requests + the way you deploy means that an SSRF can expose your private key" or whatever, and that's a technology limitation.