8 ms·

Context: [1, 2] > Open source code library cURL is removing the possibility to earn money by reporting bugs, hoping that this will reduce the volume of AI slop

by jraph 9mo ago

Context: [1, 2] > Open source code library cURL is removing the possibility to earn money by reporting bugs, hoping that this will reduce the volume of AI slop reports. > cURL has been flooded with AI-generated error reports. Now one of the incentives to create them will go away. [1] https://news.ycombinator.com/item?id=46701733 https://news.ycombinator.com/item?id=46701733 [2] https://etn.se/index.php/nyheter/72808-curl-removes-bug-bounties.html https://etn.se/index.php/nyheter/72808-curl-removes-bug-boun...

dotancohen 9mo ago
Money for a report and a patch, with convincing test cases, might be worthwhile. Even if a machine generates them.
- hobs 9mo ago
  To be clear, no, it is not, because of the opportunity cost of all the other slop. That's what this is all about.
  - johnisgood 9mo ago
    Then no bug reports and no fixes. Sounds good enough.
    - mikkupikku 9mo ago
      They'll still get bug reports and fixes from people who actually give a shit and aren't just trying to get some quick money.
    - latexr 9mo ago
      Of course there are still bug reports and fixes without financial compensation. The proof is all of open-source, including cURL.
- creata 9mo ago
  > Even if a machine generates them. That sounds wonderfully meritocratic, but in the real world, a machine generating it is a very strong signal that it's bullshit, and the people are flooding maintainers using the machines. Maintainers don't have infinite time.
- jraph 9mo ago
  I've read this idea that we could make people pay for security reports a few times here on HN (and you get back the money if the report is deemed good). That feels very wrong. If I find a security issue, I'm willing to responsibly disclose it, but if you make me pay, I don't think I will bother. Punishing bad behavior to disincentivize it seems more sensible.
  - ezst 9mo ago
    Punishing bad behaviour does close to nothing, because the problem at hand is one of high asymmetry between the low effort to submit vs the high effort to review. I do agree that paying for reports isn't ideal, and we should find other ways to level the playing field, but in the meantime I haven't heard of anything as effective.
    - jraph 9mo ago
      > the problem at hand is one of high asymmetry between the low effort to submit vs the high effort to review Hence the threat to shame publicly I suppose. Actually, Daniel Stenberg previously responded to this proposal the same way as me [1] (and maybe would still do). Coincidentally, I was reading your answer at about the same time as this part of the talk. [1] https://www.youtube.com/watch?v=6n2eDcRjSsk&t=1823s https://www.youtube.com/watch?v=6n2eDcRjSsk&t=1823s (via https://news.ycombinator.com/item?id=46717556#46717822 https://news.ycombinator.com/item?id=46717556#46717822)
      - ezst 9mo ago
        Doesn't work when using throwaway accounts, the low effort gets only marginally higher.
  - yorwba 9mo ago
    For a person finding bugs for a living, an up-front fee to have their report reviewed by a maintainer would amount to an investment towards receiving a bug bounty if their report is valid and valuable. Just the cost of doing business. It would discourage drive-by reports by people who just happened to notice a bug and want to let the maintainers know, but I think for a project that's high-profile enough to be flooded by bogus bug reports, bugs that random users just happen to notice will probably also get found by professional bug hunters at some point.
- josefx 9mo ago
  > Even if a machine generates them. Why? If it is a purely machine generated report there is no need to have dozens of third parties that throw them around blindly. A project could run it internally without having to deal with the kind of complications third parties introduce, like duplicates, copy paste errors or nonsensical assertions that they deserve money for unrelated bugfixes. A purely machine generated report without any meaningfull contribution by the submitter seems to be the first thing you would want to exclude from a bug bounty program.
- TheDong 9mo ago
  Not necessarily. Reviewing an issue report is already enough time. Reviewing a patch is even more developer time. The problem they had before was a financial incentive to sending reports, leading to crap reports that wasted time to review. Incentivizing sending reports + patches has the same failure mode, but they now have to waste even more time to review the larger quantity of input. Anyway, for most cases I'd bet that Daniel can produce and get reviewed a correct patch for a given security bug quicker than the curl team can review a third-party patch for the same, especially if it's "correct, but ai-written".
- MBCook 9mo ago
  What was a kind design to thank good contributors is now a lottery. Throw enough AI crap at enough projects and you may get a payout. The incentives fail in the face of no-effort flooding. They accidentally encourage it.

by jraph 9mo ago

dotancohen 9mo ago

Money for a report and a patch, with convincing test cases, might be worthwhile. Even if a machine generates them.

hobs 9mo ago
To be clear, no, it is not, because of the opportunity cost of all the other slop. That's what this is all about.
- johnisgood 9mo ago
  Then no bug reports and no fixes. Sounds good enough.
  - mikkupikku 9mo ago
    They'll still get bug reports and fixes from people who actually give a shit and aren't just trying to get some quick money.
  - latexr 9mo ago
    Of course there are still bug reports and fixes without financial compensation. The proof is all of open-source, including cURL.
creata 9mo ago
> Even if a machine generates them. That sounds wonderfully meritocratic, but in the real world, a machine generating it is a very strong signal that it's bullshit, and the people are flooding maintainers using the machines. Maintainers don't have infinite time.
jraph 9mo ago
I've read this idea that we could make people pay for security reports a few times here on HN (and you get back the money if the report is deemed good). That feels very wrong. If I find a security issue, I'm willing to responsibly disclose it, but if you make me pay, I don't think I will bother. Punishing bad behavior to disincentivize it seems more sensible.
- ezst 9mo ago
  Punishing bad behaviour does close to nothing, because the problem at hand is one of high asymmetry between the low effort to submit vs the high effort to review. I do agree that paying for reports isn't ideal, and we should find other ways to level the playing field, but in the meantime I haven't heard of anything as effective.
  - jraph 9mo ago
    > the problem at hand is one of high asymmetry between the low effort to submit vs the high effort to review Hence the threat to shame publicly I suppose. Actually, Daniel Stenberg previously responded to this proposal the same way as me [1] (and maybe would still do). Coincidentally, I was reading your answer at about the same time as this part of the talk. [1] https://www.youtube.com/watch?v=6n2eDcRjSsk&t=1823s https://www.youtube.com/watch?v=6n2eDcRjSsk&t=1823s (via https://news.ycombinator.com/item?id=46717556#46717822 https://news.ycombinator.com/item?id=46717556#46717822)
    - ezst 9mo ago
      Doesn't work when using throwaway accounts, the low effort gets only marginally higher.
- yorwba 9mo ago
  For a person finding bugs for a living, an up-front fee to have their report reviewed by a maintainer would amount to an investment towards receiving a bug bounty if their report is valid and valuable. Just the cost of doing business. It would discourage drive-by reports by people who just happened to notice a bug and want to let the maintainers know, but I think for a project that's high-profile enough to be flooded by bogus bug reports, bugs that random users just happen to notice will probably also get found by professional bug hunters at some point.
josefx 9mo ago
> Even if a machine generates them. Why? If it is a purely machine generated report there is no need to have dozens of third parties that throw them around blindly. A project could run it internally without having to deal with the kind of complications third parties introduce, like duplicates, copy paste errors or nonsensical assertions that they deserve money for unrelated bugfixes. A purely machine generated report without any meaningfull contribution by the submitter seems to be the first thing you would want to exclude from a bug bounty program.
TheDong 9mo ago
Not necessarily. Reviewing an issue report is already enough time. Reviewing a patch is even more developer time. The problem they had before was a financial incentive to sending reports, leading to crap reports that wasted time to review. Incentivizing sending reports + patches has the same failure mode, but they now have to waste even more time to review the larger quantity of input. Anyway, for most cases I'd bet that Daniel can produce and get reviewed a correct patch for a given security bug quicker than the curl team can review a third-party patch for the same, especially if it's "correct, but ai-written".
MBCook 9mo ago
What was a kind design to thank good contributors is now a lottery. Throw enough AI crap at enough projects and you may get a payout. The incentives fail in the face of no-effort flooding. They accidentally encourage it.