3 ms·

Receiving hundreds of AI generated bug reports would be so demoralizing and probably turn me off from maintaining an open source project forever. I think develo

Receiving hundreds of AI generated bug reports would be so demoralizing and probably turn me off from maintaining an open source project forever. I think developers are going to eventually need tools to filter out slop. If you didn’t take the time to write it, why should I take the time to read it?

triknomeister 1y ago
Eventually projects who can afford the smugness are going to charge people to be able to talk to open source developers.
- tough 1y ago
  isnt that called enterprise support / consulting
  - deleted 1y ago
    [deleted]
  - triknomeister 1y ago
    This is without the enterprise.
    - tough 1y ago
      gotchu, maybe i could see github donations enabling issue creation or wahtever in the future idk but foss is foss, i guess source available doesnt mean we have to read your messages see sqlite (wont even take PR's lol)
jgalt212 1y ago
One would think if AI can generate the slop it could also triage the slop.
- err4nt 1y ago
  How does it know the difference?
  - scubbo 1y ago
    I'm still on the AI-skeptic side of the spectrum (though shifting more towards "it has some useful applications"), but, I think the easy answer is - if different models/prompts are used in generation than in quality-/correctness-checking.
  - jgalt212 1y ago
    I think Claude, given enough time to mull it over, could probably come up with some sort of bug severity score.
  - beng-nl 1y ago
    This might not always work, but whenever possible, a working exploit could be demanded, working in a form that can be automatically verified to work.
teeray 1y ago
You see, the dream is another AI that reads the report and writes the issue in the bug tracker. Then another AI implements the fix. A third AI then reviews the code and approves and merges it. All without human interaction! Once CI releases the fix, the first AI can then find the same vulnerability plus a few new and exciting ones.
- dingnuts 1y ago
  This is completely absurd. If generating code is reliable, you can have one generator make the change, and then merge and release it with traditional software. If it's not reliable, how can you rely on the written issue to be correct, or the review, and so how does that benefit you over just blindly merging whatever changes are created by the model?
  - tempodox 1y ago
    Making sense is not required as long as “AI” vendors sell subscriptions.
  - croes 1y ago
    That’s why parent wrote it’s a dream. It’s not real. But you can bet someone will sell that as the solution.
Nicook 1y ago
Open source maintainers have been complaining about this for a while. https://sethmlarson.dev/slop-security-reports https://sethmlarson.dev/slop-security-reports. I'm assuming the proliferation of AI will have some significant changes on/already has had for open source projects.
- nestorD 1y ago
  Yes! I recently had to manually answer and close a Github issue telling me I might have pushed an API key to github. No, "API_KEY=put-your-key-here;" is a placeholder and I should not have to waste time writing that.
tptacek 1y ago
These aren't like Github Issues reports; they're bug bounty programs, specifically stood up to soak up incoming reports from anonymous strangers looking to make money on their submissions, with the premise being that enough of those reports will drive specific security goals (the scope of each program is, for smart vendors, tailored to engineering goals they have internally) to make it worthwhile.
- ryandrake 1y ago
  Got it! The financial incentive will probably turn out to be a double edged sword. Maybe in the pre-AI age, it’s By Design to drive those goals, but I bet the ability to automate submissions will inevitably alter the rules of these programs. I think within the next 5 years or so, we are going to see a societal pattern repeating: any program that rewards human ingenuity and input will become industrialized by AI to the point where it becomes a cottage industry of companies flooding every program with 99% AI submissions. What used to be lone wolves or small groups of humans working on bounties will become truckloads of AI generated “stuff” trying to maximize revenue.
  - dcminter 1y ago
    I'm wary of a lot of AI stuff, but here: > What used to be lone wolves or small groups of humans working on bounties will become truckloads of AI generated “stuff” trying to maximize revenue. You're objecting to the wrong thing. The purpose of a bug bounty programme is not to provide a cottage industry for security artisans - it's to flush out security vulnerabilities. There are reasonable objections to AI automation in this space, but this is not one of them.
  - t0mas88 1y ago
    Might be fixable by adding a $ 100 submission fee that is returned when you're proving working exploit code. Would make the Curl team a lot of money.
    - billy99k 1y ago
      I've been on Hackerone for almost 8 years and I think the problem with this is that too many companies won't pay for legitimate bugs, even when you have a working exploit. I had one critical bug take 3 years to get a pay out. I had a full walkthrough with videos and report. The company kept stalling and at one point told me that because they completely had the app remade, they weren't going to pay me anything. Hackerone doesn't really protect the researcher either. I was told multiple times that there was 'nothing they could do'. I eventually got paid, but this is pretty normal behavior with regards to bug bounty. Too many companies use it for free security work.
moyix 1y ago
All of these reports came with executable proof of the vulnerabilities – otherwise, as you say, you get flooded with hallucinated junk like the poor curl dev. This is one of the things that makes offensive security an actually good use case for AI – exploits serve as hard evidence that the LLM can't fake.
- eeeeeeehio 1y ago
  Is "proof of vulnerability" a marketing term, or do you actually claim that XBOW has a 0% false positive rate? (i.e. "all" reports come with a PoV, and this PoV "proves" there is a vulnerability?)
bawolff 1y ago
If you think the AI slop is demoralizing, you should see the human submissions bug bounties get. There is a reason companies like hackerone exist - its because dealing with the submissions is terrible.

triknomeister 1y ago

Eventually projects who can afford the smugness are going to charge people to be able to talk to open source developers.

tough 1y ago
isnt that called enterprise support / consulting
- deleted 1y ago
  [deleted]
- triknomeister 1y ago
  This is without the enterprise.
  - tough 1y ago
    gotchu, maybe i could see github donations enabling issue creation or wahtever in the future idk but foss is foss, i guess source available doesnt mean we have to read your messages see sqlite (wont even take PR's lol)

jgalt212 1y ago

One would think if AI can generate the slop it could also triage the slop.

err4nt 1y ago
How does it know the difference?
- scubbo 1y ago
  I'm still on the AI-skeptic side of the spectrum (though shifting more towards "it has some useful applications"), but, I think the easy answer is - if different models/prompts are used in generation than in quality-/correctness-checking.
- jgalt212 1y ago
  I think Claude, given enough time to mull it over, could probably come up with some sort of bug severity score.
- beng-nl 1y ago
  This might not always work, but whenever possible, a working exploit could be demanded, working in a form that can be automatically verified to work.

teeray 1y ago

You see, the dream is another AI that reads the report and writes the issue in the bug tracker. Then another AI implements the fix. A third AI then reviews the code and approves and merges it. All without human interaction! Once CI releases the fix, the first AI can then find the same vulnerability plus a few new and exciting ones.

dingnuts 1y ago
This is completely absurd. If generating code is reliable, you can have one generator make the change, and then merge and release it with traditional software. If it's not reliable, how can you rely on the written issue to be correct, or the review, and so how does that benefit you over just blindly merging whatever changes are created by the model?
- tempodox 1y ago
  Making sense is not required as long as “AI” vendors sell subscriptions.
- croes 1y ago
  That’s why parent wrote it’s a dream. It’s not real. But you can bet someone will sell that as the solution.

Nicook 1y ago

Open source maintainers have been complaining about this for a while. https://sethmlarson.dev/slop-security-reports https://sethmlarson.dev/slop-security-reports. I'm assuming the proliferation of AI will have some significant changes on/already has had for open source projects.

nestorD 1y ago
Yes! I recently had to manually answer and close a Github issue telling me I might have pushed an API key to github. No, "API_KEY=put-your-key-here;" is a placeholder and I should not have to waste time writing that.

tptacek 1y ago

These aren't like Github Issues reports; they're bug bounty programs, specifically stood up to soak up incoming reports from anonymous strangers looking to make money on their submissions, with the premise being that enough of those reports will drive specific security goals (the scope of each program is, for smart vendors, tailored to engineering goals they have internally) to make it worthwhile.

ryandrake 1y ago
Got it! The financial incentive will probably turn out to be a double edged sword. Maybe in the pre-AI age, it’s By Design to drive those goals, but I bet the ability to automate submissions will inevitably alter the rules of these programs. I think within the next 5 years or so, we are going to see a societal pattern repeating: any program that rewards human ingenuity and input will become industrialized by AI to the point where it becomes a cottage industry of companies flooding every program with 99% AI submissions. What used to be lone wolves or small groups of humans working on bounties will become truckloads of AI generated “stuff” trying to maximize revenue.
- dcminter 1y ago
  I'm wary of a lot of AI stuff, but here: > What used to be lone wolves or small groups of humans working on bounties will become truckloads of AI generated “stuff” trying to maximize revenue. You're objecting to the wrong thing. The purpose of a bug bounty programme is not to provide a cottage industry for security artisans - it's to flush out security vulnerabilities. There are reasonable objections to AI automation in this space, but this is not one of them.
- t0mas88 1y ago
  Might be fixable by adding a $ 100 submission fee that is returned when you're proving working exploit code. Would make the Curl team a lot of money.
  - billy99k 1y ago
    I've been on Hackerone for almost 8 years and I think the problem with this is that too many companies won't pay for legitimate bugs, even when you have a working exploit. I had one critical bug take 3 years to get a pay out. I had a full walkthrough with videos and report. The company kept stalling and at one point told me that because they completely had the app remade, they weren't going to pay me anything. Hackerone doesn't really protect the researcher either. I was told multiple times that there was 'nothing they could do'. I eventually got paid, but this is pretty normal behavior with regards to bug bounty. Too many companies use it for free security work.

moyix 1y ago

All of these reports came with executable proof of the vulnerabilities – otherwise, as you say, you get flooded with hallucinated junk like the poor curl dev. This is one of the things that makes offensive security an actually good use case for AI – exploits serve as hard evidence that the LLM can't fake.

eeeeeeehio 1y ago
Is "proof of vulnerability" a marketing term, or do you actually claim that XBOW has a 0% false positive rate? (i.e. "all" reports come with a PoV, and this PoV "proves" there is a vulnerability?)

bawolff 1y ago

If you think the AI slop is demoralizing, you should see the human submissions bug bounties get. There is a reason companies like hackerone exist - its because dealing with the submissions is terrible.