5 ms·

The open source maintainer gonna pay this?

by DelightOne 2y ago

nomilk 2y ago
> pay For a few API calls?
- DelightOne 2y ago
  The proposed approach used multiple LLM chains. Then you don't know how much context you need. I imagine it can get expensive. Per Commit.
  - nomilk 2y ago
    OpenAI's latest (GPT-4o) model is US$5.00 / 1M tokens [1] For large repos it might get expensive, but many will have orgs footing the bill. To satiate curiosity, I made a fresh rails app and it has 9141023 characters, let's /5 to estimate 'tokens' (wild guess), so to scan an entire rails app, that's about $10. Not nothing, but not back-breakingly expensive. Scanning could be reserved for non-trivial PRs and important applications where vulnerabilities are especially likely (e.g. perhaps not static sites, demos, or apps not in prod or a live environment). Scanning only new or edited files, along with those they interact with (where sensible patterns could emerge over time) could decrease the total volume of files needing scans, thereby reducing costs. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
    - simonw 2y ago
      Or $2.50 / million tokens if you run it in batch mode (results in up to 24 hours, though in practice much faster than that): https://platform.openai.com/docs/guides/batch/getting-started https://platform.openai.com/docs/guides/batch/getting-starte...
  - vlovich123 2y ago
    You don’t need this per commit unless you’re especially paranoid. Can easily do it just on a per release basis. The problem is that right now it’s still all wasted cost - this thing can’t really do the thing needed.
- woodruffw 2y ago
  Most OSS maintainers don't even pay for GitHub, so you're competing with free. I for one certainly won't pay for a service for the OSS projects I maintain for free. (Or another framing: I might pay for such a service, if the service could demonstrate that it would save me N hours of work fixing bugs instead of costing me N hours of work triaging false positives. Nobody has presented such a demonstration yet, which is also why turning non-LLM program analysis tooling into paid products is such a struggle!)
  - TeMPOraL 2y ago
    There's a breed of OSS maintainers that really do care about whether their project is used and satisfying to the users, vs. building stuff just because. In a situation where having your project used vs. deemed insecure is a matter of forking out $10 for an exhaustive pass with GPT-4+-based tool, such maintainers may actually care to pay. Or Github will fund it for them, like they always do.
    - woodruffw 2y ago
      The implication that I'm building stuff "just because" isn't appreciated. What I really care about in these instances is protecting my time: I have only so many hours in the day, including time that I rightfully reserve for not doing unpaid maintenance. If the X hours that I previously spent doing OSS maintenance is now partially occupied by triaging false positives, my projects are both worse off and I'm less inclined to work on them, because I equate that work with mindless churn rather than helping my users. You've pointed out that GitHub will fund it, which is potentially true! But observe that this hasn't gone all that well with CodeQL, which they also fund for public repositories: I've had to disable it on a lot of my projects, because the FP/FN ratio simply wasn't worth it. I did, and do, better for my users dedicating my time to actual issues filed.
      - TeMPOraL 2y ago
        > The implication that I'm building stuff "just because" isn't appreciated. It seems this came across not the way I intended; I apologize - I meant that very positively, as a set of possible motivations related to the code/project, problem domain, and/or author themselves, as opposed to product thinking. Like, e.g. making for fun, as part of learning, or because it's useful to author - where other users are at best a secondary concern.

The open source maintainer gonna pay this?

nomilk 2y ago

> pay For a few API calls?

DelightOne 2y ago
The proposed approach used multiple LLM chains. Then you don't know how much context you need. I imagine it can get expensive. Per Commit.
- nomilk 2y ago
  OpenAI's latest (GPT-4o) model is US$5.00 / 1M tokens [1] For large repos it might get expensive, but many will have orgs footing the bill. To satiate curiosity, I made a fresh rails app and it has 9141023 characters, let's /5 to estimate 'tokens' (wild guess), so to scan an entire rails app, that's about $10. Not nothing, but not back-breakingly expensive. Scanning could be reserved for non-trivial PRs and important applications where vulnerabilities are especially likely (e.g. perhaps not static sites, demos, or apps not in prod or a live environment). Scanning only new or edited files, along with those they interact with (where sensible patterns could emerge over time) could decrease the total volume of files needing scans, thereby reducing costs. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
  - simonw 2y ago
    Or $2.50 / million tokens if you run it in batch mode (results in up to 24 hours, though in practice much faster than that): https://platform.openai.com/docs/guides/batch/getting-started https://platform.openai.com/docs/guides/batch/getting-starte...
- vlovich123 2y ago
  You don’t need this per commit unless you’re especially paranoid. Can easily do it just on a per release basis. The problem is that right now it’s still all wasted cost - this thing can’t really do the thing needed.
woodruffw 2y ago
Most OSS maintainers don't even pay for GitHub, so you're competing with free. I for one certainly won't pay for a service for the OSS projects I maintain for free. (Or another framing: I might pay for such a service, if the service could demonstrate that it would save me N hours of work fixing bugs instead of costing me N hours of work triaging false positives. Nobody has presented such a demonstration yet, which is also why turning non-LLM program analysis tooling into paid products is such a struggle!)
- TeMPOraL 2y ago
  There's a breed of OSS maintainers that really do care about whether their project is used and satisfying to the users, vs. building stuff just because. In a situation where having your project used vs. deemed insecure is a matter of forking out $10 for an exhaustive pass with GPT-4+-based tool, such maintainers may actually care to pay. Or Github will fund it for them, like they always do.
  - woodruffw 2y ago
    The implication that I'm building stuff "just because" isn't appreciated. What I really care about in these instances is protecting my time: I have only so many hours in the day, including time that I rightfully reserve for not doing unpaid maintenance. If the X hours that I previously spent doing OSS maintenance is now partially occupied by triaging false positives, my projects are both worse off and I'm less inclined to work on them, because I equate that work with mindless churn rather than helping my users. You've pointed out that GitHub will fund it, which is potentially true! But observe that this hasn't gone all that well with CodeQL, which they also fund for public repositories: I've had to disable it on a lot of my projects, because the FP/FN ratio simply wasn't worth it. I did, and do, better for my users dedicating my time to actual issues filed.
    - TeMPOraL 2y ago
      > The implication that I'm building stuff "just because" isn't appreciated. It seems this came across not the way I intended; I apologize - I meant that very positively, as a set of possible motivations related to the code/project, problem domain, and/or author themselves, as opposed to product thinking. Like, e.g. making for fun, as part of learning, or because it's useful to author - where other users are at best a secondary concern.