6 ms·

The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult. But, on th

The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult. But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.

ymir_e 3mo ago
Edits for clarity: seems like this was a controversial take, which I find pretty interesting. I believe people took this comment as LLMs being better than security aware engineers who have the time to spend building solid systems. This wasn't the point, the reality outside "established" tech companies is that software security can be lackluster. There often simply isn't enough resources to check old software for basic vulnerabilities, outdated packages with known issues, there might be a manager who insists on a certain solution. Or that certain services are on "maintenance mode", but rarely get checked since they're on a certain part of the internal network. If they are able to properly scan their full software stack for CVE issues, they cannot deal with a flood of CVEs. ---- We're in a transition period where AI will eventually make software much more secure than it ever was. These noisy CVEs will probably lead to agents verifying vulnerabilities before humans review them. The problem with agent reviews from what I can think of is: - cost to use LLMs to review things - not necessarily easy to plug-and-play in repos: (domain knowledge + vulnerability knowledge) - especially with anthropic: able to use models defensively, without hitting guardrails The last one is the most interesting one to me. How does the AI providers know if you're a "good or bad" guy? And does it matter if open source models is catching up? We're in a kind of cyber arms race wether we like it or not.
- rghammt 3mo ago
  Currently we either get AI promo vulnerability dumps like from Chrome with pretty graphs that no one checks or false positives. Where is this one now that was hyped everywhere? https://news.ycombinator.com/item?id=49133889 https://news.ycombinator.com/item?id=49133889 The GitHub submitter could no longer reproduce the issue and the LKML post has no replies: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-ZHY5T_=4QY5+Fn8BM4A@mail.gmail.com/ https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
  - bluGill 3mo ago
    All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.
    - ymir_e 3mo ago
      Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability. The reality is that humans aren't perfect, so we do not make "perfect programs". It is also possible to test it yourself to see how good LLMs are at finding software issues: Find any open source project that isn't huge, then have Kimi 3 review it for potential issues. You will find code "issues". Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.
      - CodesInChaos 3mo ago
        I'm sure if somebody hired him for a pentest it'd cost more per vulnerability found.
        ymir_e 3mo ago
        I think his point was more that he didn't believe it was difficult to find these issues. Could be Dunning-Kruger, but if AI models continue to get better this trend will surely continue.
        telotortium 3mo ago
        1. George Hotz exhibits strong signs of Dunning-Kruger in most things he does nowadays. 2. Even in exploit development, his original field, he might be able to find vulnerabilities in mature, well-maintained software with security programs. But would he charge $20k or more for that service? Say it with me, Claude: “Absolutely!”
tcp_handshaker 3mo ago
[dead]
flerchin 3mo ago
The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.
- bluGill 3mo ago
  The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don't know if you are a good or bad - they assume bad. However if I was writing this response just one year ago I would instead be saying: the majority off LLM CVS are noise where the code is correct, and often they are writing up for code that doesn't even exist. Which is to say I suspect the repo in question was generated with a year-old LLM, since they act like that. The new ones [mostly?] are much better. Still, if a modern LLM points out something you should fix it. Even if we can't figure out how to exploit it today that doesn't mean we won't figure it out in the future.
  - thayne 3mo ago
    > I suspect the repo in question was generated with a year-old LLM I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.
    - bluGill 3mo ago
      Most bounty programs are aware of this. Many projects have closed their bounty program in response to that type of attack.
- cogman10 3mo ago
  This is what I truly don't like about the CVE system. One such example is CVE-2023-45853 [1]. Zlib included in it's source an extra set of utilities and add-ons. One such utility, MiniZip, had a buffer overflow vulnerability. BAM, 8.8 CVE (was a 9, looks like they pulled it back a bit). But not one that the 99% of applications using zlib would ever be vulnerable to because almost nobody used the MiniZip utility. It was so unused that the solution for zlib was to simply remove it. I know about this one particularly because our security policy required us to do a BUNCH of pointless updates for it since zlib is in just about everything. [1] https://app.opencve.io/cve/CVE-2023-45853 https://app.opencve.io/cve/CVE-2023-45853
gedy 3mo ago
I think in both cases, the LLM use needs to be handled by someone experienced with the domain. We are seeing this over and over with product development, writing, images, etc. Without good people driving this it's just noise
jodacola 3mo ago
I'm sending all my condolences out to everyone who is in a corporate environment using tools like Veracode or Snyk to automatically assess for CVEs and block merges or deployments based on "findings". My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI" is probably going to be difficult to get accepted, and when it does, starts to become its own avenue for exploitation and adds even more noise and confusion to the mix.
- javcasas 3mo ago
  Snyk routinely tells us that we need to upgrade urgently because they found a vulnerability in the Windows version of a package. We run everything in Linux.
- MattPalmer1086 3mo ago
  In our organisation, my security team spend the majority of our effort weeding out unexploitable CVEs - which are the majority of them. But the volume is growing year on year - I don't know how sustainable it's going to be.
  - rowls66 3mo ago
    Sounds like a good job for AI. I do fear that the future will involve deploying AI do deal with the mountain of slop produced by other AI's.
    - Versipelle 3mo ago
      I agree, this is a trend I'm seeing all over the place with generative AI: first, break a functioning, human-based workflow by flooding it with AI generated content. Blame the persons maintaining the workflow for being the bottleneck and hindering innovation. Then, implement generative AI in the workflow review or approval process to keep up the pace, irrespective of how valuable the extra generated stuff has been.
- 3mo ago
pornel 3mo ago
CVEs sucked even before LLMs. CVSS is an ass-covering exercise, not an importance signal. Everything is "critical" all the time.
hoppp 3mo ago
Basically nothing is real nowadays unless you verify it for yourself.
stellamariesays 3mo ago
[flagged]
zuzululu 3mo ago
This is an issue with CVEs not LLMs
VCFundedGenYer 3mo ago
Which is precisely why I do not celebrate when Microsoft or other companies say they released "their biggest set of security vulnerability fixes ever" because there's a good chance that most of them are hallucinations or non-critical issues erroneously considered zero-days or critical. What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.
- pixl97 3mo ago
  Eh, when we see this from Microsoft, Linux kernel, and Apple all at the same time just thinking it's everybody being dumb becomes risky. Now, if you're an F100 that's currently not using Anthropic it's pretty likely you can go to them now and ask for a non-guardrailed unlimited Mythos trial for a month you can get an agent that will not only find bugs for you, but also write exploits in a framework that will demonstrate them in action. Once you see an AI chain a bunch of shit together, get a foothold and then expand around laterally in your system you start getting concerned about how you're going to keep up. The time has come to pay for our sins of shit security.
stronglikedan 3mo ago
> I do know that LLMs have been discovering a lot of legit CVEs, And that's great! Now the human in the loop can verify the accuracy of each finding and take responsibility for moving it forward. > and I will lay odds that the blackhats are leveraging them to the max. We just need more humans in the loop to do that more quickly, each putting their own reputation on the line with each CVE they determine to be legit. AI doesn't stand for "accountability irrelevant", even though most people act like it does.
- pixl97 3mo ago
  We can shoot for 100% accountability by demanding 100% accuracy. Of course what will really happen is anyone looking for problems will stop going to the source for fixes and will start gaining cred in the dark markets of exploits and problems, which is better for everyone, right? Just because someone is using AI doesn't mean they didn't find a bug, it's more likely they just have no means of determining if it's a real bug or not, so to the maintainers tickets seem random. The problem comes in when clever attackers can take these untrustworthy tickets and run them through another LLM that determines if an exploit is possible and then uses the attacks it finds before anyone has a chance to patch.

ymir_e 3mo ago

Edits for clarity: seems like this was a controversial take, which I find pretty interesting. I believe people took this comment as LLMs being better than security aware engineers who have the time to spend building solid systems. This wasn't the point, the reality outside "established" tech companies is that software security can be lackluster. There often simply isn't enough resources to check old software for basic vulnerabilities, outdated packages with known issues, there might be a manager who insists on a certain solution. Or that certain services are on "maintenance mode", but rarely get checked since they're on a certain part of the internal network. If they are able to properly scan their full software stack for CVE issues, they cannot deal with a flood of CVEs. ---- We're in a transition period where AI will eventually make software much more secure than it ever was. These noisy CVEs will probably lead to agents verifying vulnerabilities before humans review them. The problem with agent reviews from what I can think of is: - cost to use LLMs to review things - not necessarily easy to plug-and-play in repos: (domain knowledge + vulnerability knowledge) - especially with anthropic: able to use models defensively, without hitting guardrails The last one is the most interesting one to me. How does the AI providers know if you're a "good or bad" guy? And does it matter if open source models is catching up? We're in a kind of cyber arms race wether we like it or not.

rghammt 3mo ago
Currently we either get AI promo vulnerability dumps like from Chrome with pretty graphs that no one checks or false positives. Where is this one now that was hyped everywhere? https://news.ycombinator.com/item?id=49133889 https://news.ycombinator.com/item?id=49133889 The GitHub submitter could no longer reproduce the issue and the LKML post has no replies: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-ZHY5T_=4QY5+Fn8BM4A@mail.gmail.com/ https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
- bluGill 3mo ago
  All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.
  - ymir_e 3mo ago
    Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability. The reality is that humans aren't perfect, so we do not make "perfect programs". It is also possible to test it yourself to see how good LLMs are at finding software issues: Find any open source project that isn't huge, then have Kimi 3 review it for potential issues. You will find code "issues". Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.
    - CodesInChaos 3mo ago
      I'm sure if somebody hired him for a pentest it'd cost more per vulnerability found.
      - ymir_e 3mo ago
        I think his point was more that he didn't believe it was difficult to find these issues. Could be Dunning-Kruger, but if AI models continue to get better this trend will surely continue.
        telotortium 3mo ago
        1. George Hotz exhibits strong signs of Dunning-Kruger in most things he does nowadays. 2. Even in exploit development, his original field, he might be able to find vulnerabilities in mature, well-maintained software with security programs. But would he charge $20k or more for that service? Say it with me, Claude: “Absolutely!”

tcp_handshaker 3mo ago

[dead]

flerchin 3mo ago

The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.

bluGill 3mo ago
The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don't know if you are a good or bad - they assume bad. However if I was writing this response just one year ago I would instead be saying: the majority off LLM CVS are noise where the code is correct, and often they are writing up for code that doesn't even exist. Which is to say I suspect the repo in question was generated with a year-old LLM, since they act like that. The new ones [mostly?] are much better. Still, if a modern LLM points out something you should fix it. Even if we can't figure out how to exploit it today that doesn't mean we won't figure it out in the future.
- thayne 3mo ago
  > I suspect the repo in question was generated with a year-old LLM I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.
  - bluGill 3mo ago
    Most bounty programs are aware of this. Many projects have closed their bounty program in response to that type of attack.
cogman10 3mo ago
This is what I truly don't like about the CVE system. One such example is CVE-2023-45853 [1]. Zlib included in it's source an extra set of utilities and add-ons. One such utility, MiniZip, had a buffer overflow vulnerability. BAM, 8.8 CVE (was a 9, looks like they pulled it back a bit). But not one that the 99% of applications using zlib would ever be vulnerable to because almost nobody used the MiniZip utility. It was so unused that the solution for zlib was to simply remove it. I know about this one particularly because our security policy required us to do a BUNCH of pointless updates for it since zlib is in just about everything. [1] https://app.opencve.io/cve/CVE-2023-45853 https://app.opencve.io/cve/CVE-2023-45853

gedy 3mo ago

I think in both cases, the LLM use needs to be handled by someone experienced with the domain. We are seeing this over and over with product development, writing, images, etc. Without good people driving this it's just noise

jodacola 3mo ago

I'm sending all my condolences out to everyone who is in a corporate environment using tools like Veracode or Snyk to automatically assess for CVEs and block merges or deployments based on "findings". My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI" is probably going to be difficult to get accepted, and when it does, starts to become its own avenue for exploitation and adds even more noise and confusion to the mix.

javcasas 3mo ago
Snyk routinely tells us that we need to upgrade urgently because they found a vulnerability in the Windows version of a package. We run everything in Linux.
MattPalmer1086 3mo ago
In our organisation, my security team spend the majority of our effort weeding out unexploitable CVEs - which are the majority of them. But the volume is growing year on year - I don't know how sustainable it's going to be.
- rowls66 3mo ago
  Sounds like a good job for AI. I do fear that the future will involve deploying AI do deal with the mountain of slop produced by other AI's.
  - Versipelle 3mo ago
    I agree, this is a trend I'm seeing all over the place with generative AI: first, break a functioning, human-based workflow by flooding it with AI generated content. Blame the persons maintaining the workflow for being the bottleneck and hindering innovation. Then, implement generative AI in the workflow review or approval process to keep up the pace, irrespective of how valuable the extra generated stuff has been.
3mo ago

pornel 3mo ago

CVEs sucked even before LLMs. CVSS is an ass-covering exercise, not an importance signal. Everything is "critical" all the time.

hoppp 3mo ago

Basically nothing is real nowadays unless you verify it for yourself.

stellamariesays 3mo ago

[flagged]

zuzululu 3mo ago

This is an issue with CVEs not LLMs

VCFundedGenYer 3mo ago

Which is precisely why I do not celebrate when Microsoft or other companies say they released "their biggest set of security vulnerability fixes ever" because there's a good chance that most of them are hallucinations or non-critical issues erroneously considered zero-days or critical. What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.

pixl97 3mo ago
Eh, when we see this from Microsoft, Linux kernel, and Apple all at the same time just thinking it's everybody being dumb becomes risky. Now, if you're an F100 that's currently not using Anthropic it's pretty likely you can go to them now and ask for a non-guardrailed unlimited Mythos trial for a month you can get an agent that will not only find bugs for you, but also write exploits in a framework that will demonstrate them in action. Once you see an AI chain a bunch of shit together, get a foothold and then expand around laterally in your system you start getting concerned about how you're going to keep up. The time has come to pay for our sins of shit security.

stronglikedan 3mo ago

> I do know that LLMs have been discovering a lot of legit CVEs, And that's great! Now the human in the loop can verify the accuracy of each finding and take responsibility for moving it forward. > and I will lay odds that the blackhats are leveraging them to the max. We just need more humans in the loop to do that more quickly, each putting their own reputation on the line with each CVE they determine to be legit. AI doesn't stand for "accountability irrelevant", even though most people act like it does.

pixl97 3mo ago
We can shoot for 100% accountability by demanding 100% accuracy. Of course what will really happen is anyone looking for problems will stop going to the source for fixes and will start gaining cred in the dark markets of exploits and problems, which is better for everyone, right? Just because someone is using AI doesn't mean they didn't find a bug, it's more likely they just have no means of determining if it's a real bug or not, so to the maintainers tickets seem random. The problem comes in when clever attackers can take these untrustworthy tickets and run them through another LLM that determines if an exploit is possible and then uses the attacks it finds before anyone has a chance to patch.