3 ms·

> However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities) Only a matter

by nomilk 2y ago

> However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities) Only a matter of time until LLMs are brilliant enough to find sophisticated real-world vulnerabilities. Perhaps not every novelty, but certainly vulnerabilities that resemble any seen in the past. The good news is in this arms race devs will be armed with similar tools, and the solution will be a few lines of CI code to run our own vulnerability-seeking LLM and prevent merging when (not if) vulnerabilities are detected.

DelightOne 2y ago
The open source maintainer gonna pay this?
- nomilk 2y ago
  > pay For a few API calls?
  - DelightOne 2y ago
    The proposed approach used multiple LLM chains. Then you don't know how much context you need. I imagine it can get expensive. Per Commit.
    - nomilk 2y ago
      OpenAI's latest (GPT-4o) model is US$5.00 / 1M tokens [1] For large repos it might get expensive, but many will have orgs footing the bill. To satiate curiosity, I made a fresh rails app and it has 9141023 characters, let's /5 to estimate 'tokens' (wild guess), so to scan an entire rails app, that's about $10. Not nothing, but not back-breakingly expensive. Scanning could be reserved for non-trivial PRs and important applications where vulnerabilities are especially likely (e.g. perhaps not static sites, demos, or apps not in prod or a live environment). Scanning only new or edited files, along with those they interact with (where sensible patterns could emerge over time) could decrease the total volume of files needing scans, thereby reducing costs. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
      - simonw 2y ago
        Or $2.50 / million tokens if you run it in batch mode (results in up to 24 hours, though in practice much faster than that): https://platform.openai.com/docs/guides/batch/getting-started https://platform.openai.com/docs/guides/batch/getting-starte...
    - vlovich123 2y ago
      You don’t need this per commit unless you’re especially paranoid. Can easily do it just on a per release basis. The problem is that right now it’s still all wasted cost - this thing can’t really do the thing needed.
  - woodruffw 2y ago
    Most OSS maintainers don't even pay for GitHub, so you're competing with free. I for one certainly won't pay for a service for the OSS projects I maintain for free. (Or another framing: I might pay for such a service, if the service could demonstrate that it would save me N hours of work fixing bugs instead of costing me N hours of work triaging false positives. Nobody has presented such a demonstration yet, which is also why turning non-LLM program analysis tooling into paid products is such a struggle!)
    - TeMPOraL 2y ago
      There's a breed of OSS maintainers that really do care about whether their project is used and satisfying to the users, vs. building stuff just because. In a situation where having your project used vs. deemed insecure is a matter of forking out $10 for an exhaustive pass with GPT-4+-based tool, such maintainers may actually care to pay. Or Github will fund it for them, like they always do.
      - woodruffw 2y ago
        The implication that I'm building stuff "just because" isn't appreciated. What I really care about in these instances is protecting my time: I have only so many hours in the day, including time that I rightfully reserve for not doing unpaid maintenance. If the X hours that I previously spent doing OSS maintenance is now partially occupied by triaging false positives, my projects are both worse off and I'm less inclined to work on them, because I equate that work with mindless churn rather than helping my users. You've pointed out that GitHub will fund it, which is potentially true! But observe that this hasn't gone all that well with CodeQL, which they also fund for public repositories: I've had to disable it on a lot of my projects, because the FP/FN ratio simply wasn't worth it. I did, and do, better for my users dedicating my time to actual issues filed.
        TeMPOraL 2y ago
        > The implication that I'm building stuff "just because" isn't appreciated. It seems this came across not the way I intended; I apologize - I meant that very positively, as a set of possible motivations related to the code/project, problem domain, and/or author themselves, as opposed to product thinking. Like, e.g. making for fun, as part of learning, or because it's useful to author - where other users are at best a secondary concern.
tdullien 2y ago
I've worked on zero days for decades and friends of mine with great academic pedigrees and deep experience in both AI and security are uniformly frustrated with the inability of even cutting-edge LLMs to reason in the way required for security bug detection.
- rfoo 2y ago
  > to reason Should we expect current LLMs to be able to reason at all? As the paper said GPT-4 maybe make a better scanner. The problem is too many people think all we need is better scanners, which is not new.
  - mattgreenrocks 2y ago
    Yup. If you believe vulnerability research is a problem solved by throwing more bodies at it, I can see why LLMs are appealing. I remain skeptical about their efficacy. You need a lot of nonlocal contextual information and very good reasoning skills to be an effective reverse engineer.
- TeMPOraL 2y ago
  Give it time. GPT-4 is barely 14 months old. Things might look different in a year or two.
mattgreenrocks 2y ago
> Only a matter of time until LLMs are brilliant enough to find sophisticated real-world vulnerabilities. There sure seems to be a lot of “hopes and prayers” around AI’s future. :) My money is still on SMEs using specialized tooling backed by large funding vehicles. Aka, how things work right now.

by nomilk 2y ago

DelightOne 2y ago

The open source maintainer gonna pay this?

nomilk 2y ago
> pay For a few API calls?
- DelightOne 2y ago
  The proposed approach used multiple LLM chains. Then you don't know how much context you need. I imagine it can get expensive. Per Commit.
  - nomilk 2y ago
    OpenAI's latest (GPT-4o) model is US$5.00 / 1M tokens [1] For large repos it might get expensive, but many will have orgs footing the bill. To satiate curiosity, I made a fresh rails app and it has 9141023 characters, let's /5 to estimate 'tokens' (wild guess), so to scan an entire rails app, that's about $10. Not nothing, but not back-breakingly expensive. Scanning could be reserved for non-trivial PRs and important applications where vulnerabilities are especially likely (e.g. perhaps not static sites, demos, or apps not in prod or a live environment). Scanning only new or edited files, along with those they interact with (where sensible patterns could emerge over time) could decrease the total volume of files needing scans, thereby reducing costs. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
    - simonw 2y ago
      Or $2.50 / million tokens if you run it in batch mode (results in up to 24 hours, though in practice much faster than that): https://platform.openai.com/docs/guides/batch/getting-started https://platform.openai.com/docs/guides/batch/getting-starte...
  - vlovich123 2y ago
    You don’t need this per commit unless you’re especially paranoid. Can easily do it just on a per release basis. The problem is that right now it’s still all wasted cost - this thing can’t really do the thing needed.
- woodruffw 2y ago
  Most OSS maintainers don't even pay for GitHub, so you're competing with free. I for one certainly won't pay for a service for the OSS projects I maintain for free. (Or another framing: I might pay for such a service, if the service could demonstrate that it would save me N hours of work fixing bugs instead of costing me N hours of work triaging false positives. Nobody has presented such a demonstration yet, which is also why turning non-LLM program analysis tooling into paid products is such a struggle!)
  - TeMPOraL 2y ago
    There's a breed of OSS maintainers that really do care about whether their project is used and satisfying to the users, vs. building stuff just because. In a situation where having your project used vs. deemed insecure is a matter of forking out $10 for an exhaustive pass with GPT-4+-based tool, such maintainers may actually care to pay. Or Github will fund it for them, like they always do.
    - woodruffw 2y ago
      The implication that I'm building stuff "just because" isn't appreciated. What I really care about in these instances is protecting my time: I have only so many hours in the day, including time that I rightfully reserve for not doing unpaid maintenance. If the X hours that I previously spent doing OSS maintenance is now partially occupied by triaging false positives, my projects are both worse off and I'm less inclined to work on them, because I equate that work with mindless churn rather than helping my users. You've pointed out that GitHub will fund it, which is potentially true! But observe that this hasn't gone all that well with CodeQL, which they also fund for public repositories: I've had to disable it on a lot of my projects, because the FP/FN ratio simply wasn't worth it. I did, and do, better for my users dedicating my time to actual issues filed.
      - TeMPOraL 2y ago
        > The implication that I'm building stuff "just because" isn't appreciated. It seems this came across not the way I intended; I apologize - I meant that very positively, as a set of possible motivations related to the code/project, problem domain, and/or author themselves, as opposed to product thinking. Like, e.g. making for fun, as part of learning, or because it's useful to author - where other users are at best a secondary concern.

tdullien 2y ago

I've worked on zero days for decades and friends of mine with great academic pedigrees and deep experience in both AI and security are uniformly frustrated with the inability of even cutting-edge LLMs to reason in the way required for security bug detection.

rfoo 2y ago
> to reason Should we expect current LLMs to be able to reason at all? As the paper said GPT-4 maybe make a better scanner. The problem is too many people think all we need is better scanners, which is not new.
- mattgreenrocks 2y ago
  Yup. If you believe vulnerability research is a problem solved by throwing more bodies at it, I can see why LLMs are appealing. I remain skeptical about their efficacy. You need a lot of nonlocal contextual information and very good reasoning skills to be an effective reverse engineer.
TeMPOraL 2y ago
Give it time. GPT-4 is barely 14 months old. Things might look different in a year or two.

mattgreenrocks 2y ago

> Only a matter of time until LLMs are brilliant enough to find sophisticated real-world vulnerabilities. There sure seems to be a lot of “hopes and prayers” around AI’s future. :) My money is still on SMEs using specialized tooling backed by large funding vehicles. Aka, how things work right now.