6 ms·

AI Slop vs. OSS Security

193 points by mooreds 1y ago

pksebben 1y ago
> The model has no concept of truth—only of plausibility. This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear. Hats off to the curl maintainers. You are the xkcd jenga block at the base.
- jcattle 1y ago
  I am assuming that millions of dollars have already been spent trying to get LLMs to hallucinate less. Even if Problems feel soluble, they often aren't. You might have to invent an entirely new paradigm of text generation to solve the hallucination problem. Or it could be the Collatz Conjecture of LLMs, that it "feels" so possible, but you never really get there.
  - big-and-small 1y ago
    Nuclear fusion was always 30 years away (c)
    - quikoa 1y ago
      It would be nice if nuclear fusion had the AI budget.
      - Cthulhu_ 1y ago
        Fusion will at best have a few dozen sales once it's commercially viable and then take decades to realise, but you can sell AI stuff to millions of customers for $20 / month each and do it today.
- pjc50 1y ago
  The "fact database" is the old AI solution, e.g. Cycorp; it doesn't quite work either. Knowing what is true is a really hard, unsolved problem in philosophy, see e.g. https://en.wikipedia.org/wiki/Gettier_problem https://en.wikipedia.org/wiki/Gettier_problem . The secret to modern AI is just to skip that and replace unsolvable epistemology with "LGTM", then sell it to investors.
  - pksebben 1y ago
    There are some things that we can define as "definitely true as close as makes no difference" in the context of an LLM: - dictionary definitions - stable apis for specific versions of software - mathematical proofs - anything else that is true by definition rather than evidence-based (i realize that some of these are not actually as stable over time as they might seem, but they ought to do good enough with the pace that we train new models at). If you even just had an MOE component whose only job was verifying validity against this dataset in chain-of-thought I bet you'd get some mileage out of it.
- wongarsu 1y ago
  Truth comes from being able to test your assertions. Without that they remain in the realm of plausibility. You can't get from plausibility to truth with better training data, you need to give LLMs better tools to test the truth of their plausible statements before spewing them to the user (and train the models to use them, obviously. But that's not the hard part).
wwfn 1y ago
Wealth generated on top of underpaid labor is a reoccurring theme -- and in this case maybe surprisingly exacerbated by LLMs. Would this be different if the underlying code had a viral license? If google's infrastructure was built on a GPL'ed libcurl [0], would they have investment in the code/a team with resources to evaluate security reports (slop or otherwise)? Ditto for libxml. Does GPL help the linux kernel get investment from it's corporate users? [0] Perhaps an impossible hypothetical. Would google have skipped over the imaginary GPL'ed libcurl or libxml for a more permissively licensed library? And even if they didn't, would a big company's involvement in an openly developed ecosystem create asymmetric funding/goals, a la XMPP or Nix?
- big-and-small 1y ago
  Copyleft licenses are made to support freedom for everyone and particularly end-users. They only limit freedom of developers / maintainers to exploit the code and users. > Does GPL help the linux kernel get investment from it's corporate users? GPL has helped "linux kernel the project" greatly, but companies invest in it out of their self-interest. They want to benefit from upstream improvements and playing nicely by upstreaming changes is just much cheaper than maintaining own kernel fork. On other side you have companies like Sony that used BSD OS code for their game consoles for decades and contributed shit. So... Two unrelated things.
  - wwfn 1y ago
    I would have thought supporting libcurl and libxml would also be in a company's self-interest. Is that companies do this for GPL'ed linux kernel but not BSD evidence that strong copyleft licensing limits the extent to which OSS projects are exploited/under-resourced?
    - big-and-small 1y ago
      > I would have thought supporting libcurl and libxml would also be in a company's self-interest. Unfortunately majority of companies don't have something special they really need to add to cURL. They okay using it as is - so they have no reason to pay salary to cURL developers regardless of licensing. Yes they want it to be secure, but as always nobody except few very large orgs care about security for real. > Is that companies do this for GPL'ed linux kernel but not BSD evidence that strong copyleft licensing limits the extent to which OSS projects are exploited/under-resourced? It certainly helped with "under-resourced" part. Whatever you considered "exploited" is up to discussion. From project perspective ofc copyleft licensing benefited the project. Linus Torvalds end up with a good amount of publicity and is now somewhat well set-off, but almost all other kernel developers live in obscurity earning somewhat average salaries. I pretty sure we can all agree that Linux Kernel made a massive positive impact on whole humanity and compared to that payoff to stakeholders is rather small IMO.
dvt 1y ago
> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop. If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if you find some function, that for example doesn't do bounds-checking on some array, you can trivially write some unit testing code that's able to break it.
- noirscape 1y ago
  The problem that is that a lot of CVEs often don't represent "real" vulnerabilities, but merely theoretical ones that could hypothetically be combined to make a real exploit. Regex exploitation is the forever example to bring up here, as it's generally the main reason that "autofail the CI system the moment an auditing command fails" doesn't work on certain codebases. The reason this happens is because it's trivial to make a string that can waste significant resources to try and do a regex match against, and the moment you have a function that accepts a user-supplied regex pattern, that's suddenly an exploit... which gets a CVE. A lot of projects then have CVEs filed against them because internal functions rely on Regex calls as arguments, even if they're in code the user is flat-out never going to be able interact with (ie. Several dozen layers deep in framework soup there's a regex call somewhere, in a way the user won't be able to access unless a developer several layers up starts breaking the framework they're using in really weird ways on purpose). The CVE system is just completely broken and barely serves as an indicator of much of anything really. The approval system from what I can tell favors acceptance over rejection, since the people reviewing the initial CVE filing aren't the same people that actively investigate if the CVE is bogus or not and the incentive for the CVE system is literally to encourage companies to give a shit about software security (at the same time, this fact is also often exploited to create beg bounties). CVEs have been filed against software for what amounts to "a computer allows a user to do things on it" even before AI slop made everything worse; the system was questionable in quality 7 years ago at the very least, and is even worse these days. The only indicator it really gives is that a real security exploit can feel more legitimate if it gets a CVE assigned to it.
Jean-Papoulos 1y ago
The solution isn't to block aggressively or to allow everything, but to prioritize. Put accounts older than the AI boom at the top, and allow them to give "referrals", ie stake a part of their own credibility to boost another account on the priority ladder. Referral systems are very efficient at filtering noise.
- skydhash 1y ago
  It only works when it’s a true stake. Like loosing privileges when a referral is a dunce. The downside is tribalism.
- nixpulvis 1y ago
  This is very short sighted.
- bluGill 1y ago
  Who will refer me when I find my first issue. this might be a critical thing that should be top priority, but because nobody knows me and I don't know anyone else you just put me to the bottom. Meanwhile someone evil figures out the same problem and a week latter you discover the "zero day" that was causing so many problems for you was in fact reported earlier. There is also the possibility that in trying to get someone to refer me I give enough details that the trusted person can submit instead of me and claim credit.
- postexitus 1y ago
  Works only for old projects. What happens with new ones?
goalieca 1y ago
> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.
- jsheard 1y ago
  The other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.
  - gdulli 1y ago
    That's an important one. Another fundamental problem with plausible output is that it makes a manager, or a junior, or some other unsophisticated end user think the technology is almost there, and a reliably correct version is just around the corner.
  - jacquesm 1y ago
    Indeed. The elderly in my family are seeing a substantial uptick of AI generated stuff that looks extremely plausible. Fortunately they're old but not stupid, so far nobody has fallen for any of these but I have to admit: they look good enough to pass a first casual inspection.
- Sharlin 1y ago
  Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.
  - dafelst 1y ago
    I agree wholeheartedly, and this is the core problem - many of the people evangelizing LLMs for a particular task (especially investors and AI gold rush "entrepreneurs") do not have enough expertise in that particular field to effectively evaluate the quality of the output. It sure looks the part though, and for those with a shallow understanding, it is often enough.
beeburrt 1y ago
I didn't realize how bleak the future looks, wrt CVE infastructure both MITRE and at National Vuln. Database. What do other countries do for their stuff like this?
Maro 1y ago
Manufacturing vulnerability submissions that look like real vulnerability submissions, but the vulnerability isn't there and the submitter doesn't understand what it's saying. It's a cargo cult. Maybe the airplanes will land and bring the goodies! https://en.wikipedia.org/wiki/Cargo_cult https://en.wikipedia.org/wiki/Cargo_cult
supriyo-biswas 1y ago
Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"
- nixpulvis 1y ago
  I'm so sick of people claiming things sound like AI, when it's so easily not true. Between this and the flip side of AI-slop it's getting really frustrating out here online.
  - chemotaxis 1y ago
    I think people sometimes jump the gun over small things (emdashes, etc). That said, in this instance, your anger is very likely misdirected. The article is almost certainly substantially AI-generated.
    - nixpulvis 1y ago
      I mean, I guess I just don't care as much, as long as the author proof read it and it says what they want it to say. Just like I don't care if an AI submits a vulnerability report that is actually real.
  - progbits 1y ago
    From the article itself (presumably added later): Disclosure: Certain sections of this content were grammatically refined/updated using AI assistance
  - bluGill 1y ago
    Problem is AI sounds like normal people. There are a few clues, but good writers have always sounded like AI - except good writers are not making things up.
hacb 1y ago
I don't know why submitting a vulnerability on those platforms is still free. If reporters had to pay a little amount of money (let's say, 20-50$, or indexed to the maximum gain of a vulnerability in a given category) when submitting their report, maybe those would be of better quality. I know that this poses new problems (some people can't afford to spend this money), but it would be better than just wasting people's time.
- Chilinot 1y ago
  This point is brought up and discussed in the linked article.
nixpulvis 1y ago
I believe "Economic Friction" is truly the best and potentially strongest way to deal with this. It's good for the site collecting the fee, it's good for the projects being reported on and it doesn't negatively affect valid reports. It does exactly what we want by disincentivizing bad reports, either AI generated or not.
mmsc 1y ago
>First, the typical AI-powered reporter, especially one just pasting GPT output into a submission form, neither knows enough about the actual codebase being examined nor understands the security implications well enough to provide insight that projects need. How ironic, considering every time I've reported a complicated issue to a program on HackerOne, the triggers have completely rejected them because they do not understand the complicated codebase that they are triaging for. Also the curl examples given in TFA completely ignore recent developments, where curl's maintainers welcomed and fixed literally hundred of AI-found bugs: https://www.theregister.com/2025/10/02/curl_project_swamped_with_ai/ https://www.theregister.com/2025/10/02/curl_project_swamped_...
heymax054 1y ago
Semi-related: I posted this yesterday: https://news.ycombinator.com/item?id=45828917 https://news.ycombinator.com/item?id=45828917 Different models perform differently when it comes to catching/fixing security vulnerabilities.
samlinnfer 1y ago
Just add a country IP ban, we all know who is submitting these reports. Remember Hacktoberfest?
- Cthulhu_ 1y ago
  That's a game of whack-a-mole, they'd just use a VPN. And besides, no we don't "all know" who is submitting these reports, that's a generalization.
  - samlinnfer 1y ago
    It doesn’t have to be perfect, it just has add friction so they stop spamming.
- mschuster91 1y ago
  > Just add a country IP ban, we all know who is submitting these reports. As much as I'd like to see Russia, China and India disconnected off of the wide Internet until they clean up shop with abusive actors, the Hacktoberfest stuff you're likely referring to doesn't have anything to do with your implication - that was just a chance at a free t-shirt [1] that caused all the noise. In ye olde times, you'd need to take care how you behaved in public because pulling off a stunt like that could reasonably lead to your company going out of business - but even a "small" company like DO is too big to fail from FAFO, much less ultra large corporations like Google that just run on sheer moat. IMHO, that is where we have to start - break up the giants, maybe that's enough of a warning signal to also alert "smaller" large companies to behave like citizens again. [1] https://domenic.me/hacktoberfest/ https://domenic.me/hacktoberfest/
  - samlinnfer 1y ago
    IP banning India would have solved hacktoberfest spam 100% and maybe 99% of fake AI bug reports.
Cthulhu_ 1y ago
Very blunt maybe, but if individuals try to get internet points by doing frivolous security reports under their own name, should they be loudly pinned to a Wall of Shame to discourage the practice?
cheesecompiler 1y ago
> The problem isn't knowledge—it's incentives. > When you're volunteering out of love in a market society, you're setting yourself up to be exploited. I sound like a broken record but there's unifying causes to most issues I observe in the world. None of the proposed solutions address the cause (and they can't of course): public scrutiny doesn't do anything if account creation is zero-effort; monetary penalization will kill the submissions entirely. In a perfect world OSS maintainers would get paid properly. But, we've been doing this since the 90s, and all that's happened is OSS got deployed by private companies, concentrating the wealth and the economic benefits. When every hour is paid labour, you pick the AWS Kafka over spinning up your own cluster, or you run Linux in the cloud instead of your own metal. This will always keep happening so long as the incentives are what they are and survival hinges on capital. That people still put in their free time speaks to the beautiful nature of humans, but it's in spite of the current systems.
progbits 1y ago
Certain sections of this content were grammatically refined/updated using AI assistance, as English is not my first language. OP: I sympathize, but I would much rather read your original text, with typos and grammatical errors. By feeding it through the LLM you fix issues that are not really important but remove your own voice and get a bland slop identical to 90% of these slopblogs (which your's isn't!)
danieltk76 1y ago
At Vulnetic ai we deal with this by having a separate validator agent attack the vulnerability reported using a novel perspective separate of the hacking agent.
Sharlin 1y ago
> The incentive is to submit as many reports as possible and see what sticks, because even a 5% hit rate on a hundred submissions is better than the effort of manually verifying five findings. As I just commented in the other AI trust thread on the front page, this dynamic is funnily enough what any woman using online dating services has always been very familiar with. With the exact same tragedy of the commons that results. Except for the important difference that terrible profiles and intro messages have traditionally usually been very short and easily red-flagged. But that is, of course, now also changing or already changed due to LLMs. (Someone I follow on a certain social media platform just remarked that she got no less than fifty messages within a single hour of marking herself as "single". And she's just some average person, not a "star" of any sort.)
samuelknight 1y ago
You can address the issue by putting the report and the code base in a sandbox with an agent that tries to reproduce it. If it can't reproduce it then that should be a strike against the reporter. OSS projects should absolutely ban accounts that repetitively create reports that are of such low quality that it can't be recreated. IMO the Hacker One reputation mechanism is a good idea because it incentives users who operate in good faith and can serially produce findings.
- mayhemducks 1y ago
  And who pays for the tokens?
  - noduerme 1y ago
    Sandbox a third AI that just bets on AI stocks and crypto. Add a fourth AI to check the third AI's bets, and a fifth one to go on forums and pump the relevant equities. A sixth AI can short sell when the fourth AI gets overheated.
bklosky 1y ago
Companies soliciting big bounties should charge a fee to submit, making slop costly. Steam, the video game publisher, does this. Developers need to pay a fee to list their game, and if their game sells sufficient volume, the fee is returned. Creating a separating equilibrium here is not hard, the hand wringing is weird to me.
- 8note 1y ago
  adding cost to accounts is handy, but its not like that has solved steam's scam/slop problems. fake games by fake studios played by fake players is still a thing
theptip 1y ago
It seems pretty obvious that the bar needs to be raised. > A security report lands in your inbox. It claims there's a buffer overflow in a specific function. The report is well-formatted, includes CVE-style nomenclature, and uses appropriate technical language. Given how easy it is to generate a POC these days, I wonder if HackerOne needs to be pivoting hard into scaffolding to help bug hunters prove their vulns. - Claude skills/MCP for OSS projects - Attested logging/monitoring for API investigations (eg hosted BURP)
noduerme 1y ago
>> some actors with misaligned incentives interpret high submission volume as achievement Welcome to the Internet.
ok123456 1y ago
Require a docker/script that provides for the necessary conditions for the exploit, along with a POC. If something is impossible to provide a POC for, as it's more of a speculative attack, require vetting like arxiv. Most people's initial contributions are going to be more concrete exploits.
ceroxylon 1y ago
Surely there can be a workflow created to "fight fire with fire" and have an AI that reads reports, trained on the code base with explicit instructions to verify all of the telltale signs of slop...? If AI services can handle the nightmare of parsing emails and understanding the psychology of phishing, I am optimistic it can be done for OSS reports. It doesn't have to make the final judgement, just some sort of filter that automatically flags things like function calls that don't exist in the code.
stackedinserter 1y ago
So use AI to check AI reports, problem solved.
- aidenn0 1y ago
  I cannot tell if you are being serious or sarcastic.
  - stackedinserter 1y ago
    50/50
  - 8note 1y ago
    i dunno about problem solved, but id expect to be able to put together an agent that helps requesters format their reports well, and that could run the reproduction steps in a sandbox.
fleventynine 1y ago
"My attention is a limited resource. In order to prove that you're a serious human, please donate exactly $157.42 to the Rust maintainers fund and paste the receipt link here".
stuxf 1y ago
I agree with a lot of things said in this article, I also think some sort of centralized trust system for OSS bug bounty would be a really good solution to this problem > The downside is that it makes it harder for new researchers to enter the field, and it risks creating an insider club. I also think this concern can be largely mitigated or reduced to a nonissue. New researchers would have a trust score of zero for example, but people who consistently submit AI slop will have a very low score and can be filtered out fairly easily.
HiPhish 1y ago
We need a name for the type of attack where maintainers get overwhelmed by a massive amount of low-effort and false bug reports and contributions. I propose "Slop Overflow". That's right, you heard it here first, I came up with it :)
DrewADesign 1y ago
> By flooding bug bounty programs with AI-generated reports, they feel productive and entrepreneurial. Some genuinely believe the AI has found something real. Others know it's questionable but figure they'll let the maintainers sort it out. I’m not saying that AI hasn’t already given us useful things, but this is a symptom of one very negative change that’s drowned a lot of the positive out for many people: the competence gap used to be an automatic barrier for many things. For example, to get hired as a freelance developer, you had to have at least cargo-culted something together once or twice, and even if you were way overconfident in your capability, you probably knew you weren’t the real thing. However, the AI tools industry essentially markets free competence, and out of context for any given topic, that little disclaimer is meaningless. It’s essentially given people climbing the Dunning-Krueger Mt. Stupid the agency to produce garbage at a damaging volume that’s too plausible looking for them (or laypeople, for that matter) to realize it’s garbage. I also think somewhat nihilist people prone to get-rich-quick schemes (e.g. drop-shipping, NFTs) play these workflows like lottery tickets while remaining deliberately ignorant of their dubious value.
breakpointalpha 1y ago
Open Source maintainers should require a $10 deposit to submit a pull request. Even more so when there is a bounty payout. Refundable if the PR/report is accepted.
jruohonen 1y ago
I think this piece was a good summary of the state of affairs. If I would have written it, I would have perhaps mentioned that similar problems exist also in other domains, including science, arts, and media. Maybe the solutions might be similar too? I am particularly pointing toward the following quote that wasn't yet discussed here: "New reporters could be required to have established community members vouch for them, creating a web-of-trust model. This mirrors how the world worked before bug bounty platforms commodified security research. The only downside is, it risks creating an insider club."

193 points by mooreds 1y ago

> The model has no concept of truth—only of plausibility. This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear. Hats off to the curl maintainers. You are the xkcd jenga block at the base.

jcattle 1y ago
I am assuming that millions of dollars have already been spent trying to get LLMs to hallucinate less. Even if Problems feel soluble, they often aren't. You might have to invent an entirely new paradigm of text generation to solve the hallucination problem. Or it could be the Collatz Conjecture of LLMs, that it "feels" so possible, but you never really get there.
- big-and-small 1y ago
  Nuclear fusion was always 30 years away (c)
  - quikoa 1y ago
    It would be nice if nuclear fusion had the AI budget.
    - Cthulhu_ 1y ago
      Fusion will at best have a few dozen sales once it's commercially viable and then take decades to realise, but you can sell AI stuff to millions of customers for $20 / month each and do it today.
pjc50 1y ago
The "fact database" is the old AI solution, e.g. Cycorp; it doesn't quite work either. Knowing what is true is a really hard, unsolved problem in philosophy, see e.g. https://en.wikipedia.org/wiki/Gettier_problem https://en.wikipedia.org/wiki/Gettier_problem . The secret to modern AI is just to skip that and replace unsolvable epistemology with "LGTM", then sell it to investors.
- pksebben 1y ago
  There are some things that we can define as "definitely true as close as makes no difference" in the context of an LLM: - dictionary definitions - stable apis for specific versions of software - mathematical proofs - anything else that is true by definition rather than evidence-based (i realize that some of these are not actually as stable over time as they might seem, but they ought to do good enough with the pace that we train new models at). If you even just had an MOE component whose only job was verifying validity against this dataset in chain-of-thought I bet you'd get some mileage out of it.
wongarsu 1y ago
Truth comes from being able to test your assertions. Without that they remain in the realm of plausibility. You can't get from plausibility to truth with better training data, you need to give LLMs better tools to test the truth of their plausible statements before spewing them to the user (and train the models to use them, obviously. But that's not the hard part).

wwfn 1y ago

Wealth generated on top of underpaid labor is a reoccurring theme -- and in this case maybe surprisingly exacerbated by LLMs. Would this be different if the underlying code had a viral license? If google's infrastructure was built on a GPL'ed libcurl [0], would they have investment in the code/a team with resources to evaluate security reports (slop or otherwise)? Ditto for libxml. Does GPL help the linux kernel get investment from it's corporate users? [0] Perhaps an impossible hypothetical. Would google have skipped over the imaginary GPL'ed libcurl or libxml for a more permissively licensed library? And even if they didn't, would a big company's involvement in an openly developed ecosystem create asymmetric funding/goals, a la XMPP or Nix?

big-and-small 1y ago
Copyleft licenses are made to support freedom for everyone and particularly end-users. They only limit freedom of developers / maintainers to exploit the code and users. > Does GPL help the linux kernel get investment from it's corporate users? GPL has helped "linux kernel the project" greatly, but companies invest in it out of their self-interest. They want to benefit from upstream improvements and playing nicely by upstreaming changes is just much cheaper than maintaining own kernel fork. On other side you have companies like Sony that used BSD OS code for their game consoles for decades and contributed shit. So... Two unrelated things.
- wwfn 1y ago
  I would have thought supporting libcurl and libxml would also be in a company's self-interest. Is that companies do this for GPL'ed linux kernel but not BSD evidence that strong copyleft licensing limits the extent to which OSS projects are exploited/under-resourced?
  - big-and-small 1y ago
    > I would have thought supporting libcurl and libxml would also be in a company's self-interest. Unfortunately majority of companies don't have something special they really need to add to cURL. They okay using it as is - so they have no reason to pay salary to cURL developers regardless of licensing. Yes they want it to be secure, but as always nobody except few very large orgs care about security for real. > Is that companies do this for GPL'ed linux kernel but not BSD evidence that strong copyleft licensing limits the extent to which OSS projects are exploited/under-resourced? It certainly helped with "under-resourced" part. Whatever you considered "exploited" is up to discussion. From project perspective ofc copyleft licensing benefited the project. Linus Torvalds end up with a good amount of publicity and is now somewhat well set-off, but almost all other kernel developers live in obscurity earning somewhat average salaries. I pretty sure we can all agree that Linux Kernel made a massive positive impact on whole humanity and compared to that payoff to stakeholders is rather small IMO.

dvt 1y ago

> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop. If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if you find some function, that for example doesn't do bounds-checking on some array, you can trivially write some unit testing code that's able to break it.

noirscape 1y ago
The problem that is that a lot of CVEs often don't represent "real" vulnerabilities, but merely theoretical ones that could hypothetically be combined to make a real exploit. Regex exploitation is the forever example to bring up here, as it's generally the main reason that "autofail the CI system the moment an auditing command fails" doesn't work on certain codebases. The reason this happens is because it's trivial to make a string that can waste significant resources to try and do a regex match against, and the moment you have a function that accepts a user-supplied regex pattern, that's suddenly an exploit... which gets a CVE. A lot of projects then have CVEs filed against them because internal functions rely on Regex calls as arguments, even if they're in code the user is flat-out never going to be able interact with (ie. Several dozen layers deep in framework soup there's a regex call somewhere, in a way the user won't be able to access unless a developer several layers up starts breaking the framework they're using in really weird ways on purpose). The CVE system is just completely broken and barely serves as an indicator of much of anything really. The approval system from what I can tell favors acceptance over rejection, since the people reviewing the initial CVE filing aren't the same people that actively investigate if the CVE is bogus or not and the incentive for the CVE system is literally to encourage companies to give a shit about software security (at the same time, this fact is also often exploited to create beg bounties). CVEs have been filed against software for what amounts to "a computer allows a user to do things on it" even before AI slop made everything worse; the system was questionable in quality 7 years ago at the very least, and is even worse these days. The only indicator it really gives is that a real security exploit can feel more legitimate if it gets a CVE assigned to it.

Jean-Papoulos 1y ago

The solution isn't to block aggressively or to allow everything, but to prioritize. Put accounts older than the AI boom at the top, and allow them to give "referrals", ie stake a part of their own credibility to boost another account on the priority ladder. Referral systems are very efficient at filtering noise.

skydhash 1y ago
It only works when it’s a true stake. Like loosing privileges when a referral is a dunce. The downside is tribalism.
nixpulvis 1y ago
This is very short sighted.
bluGill 1y ago
Who will refer me when I find my first issue. this might be a critical thing that should be top priority, but because nobody knows me and I don't know anyone else you just put me to the bottom. Meanwhile someone evil figures out the same problem and a week latter you discover the "zero day" that was causing so many problems for you was in fact reported earlier. There is also the possibility that in trying to get someone to refer me I give enough details that the trusted person can submit instead of me and claim credit.
postexitus 1y ago
Works only for old projects. What happens with new ones?

goalieca 1y ago

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

jsheard 1y ago
The other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.
- gdulli 1y ago
  That's an important one. Another fundamental problem with plausible output is that it makes a manager, or a junior, or some other unsophisticated end user think the technology is almost there, and a reliably correct version is just around the corner.
- jacquesm 1y ago
  Indeed. The elderly in my family are seeing a substantial uptick of AI generated stuff that looks extremely plausible. Fortunately they're old but not stupid, so far nobody has fallen for any of these but I have to admit: they look good enough to pass a first casual inspection.
Sharlin 1y ago
Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.
- dafelst 1y ago
  I agree wholeheartedly, and this is the core problem - many of the people evangelizing LLMs for a particular task (especially investors and AI gold rush "entrepreneurs") do not have enough expertise in that particular field to effectively evaluate the quality of the output. It sure looks the part though, and for those with a shallow understanding, it is often enough.

beeburrt 1y ago

I didn't realize how bleak the future looks, wrt CVE infastructure both MITRE and at National Vuln. Database. What do other countries do for their stuff like this?

Maro 1y ago

Manufacturing vulnerability submissions that look like real vulnerability submissions, but the vulnerability isn't there and the submitter doesn't understand what it's saying. It's a cargo cult. Maybe the airplanes will land and bring the goodies! https://en.wikipedia.org/wiki/Cargo_cult https://en.wikipedia.org/wiki/Cargo_cult

supriyo-biswas 1y ago

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

nixpulvis 1y ago
I'm so sick of people claiming things sound like AI, when it's so easily not true. Between this and the flip side of AI-slop it's getting really frustrating out here online.
- chemotaxis 1y ago
  I think people sometimes jump the gun over small things (emdashes, etc). That said, in this instance, your anger is very likely misdirected. The article is almost certainly substantially AI-generated.
  - nixpulvis 1y ago
    I mean, I guess I just don't care as much, as long as the author proof read it and it says what they want it to say. Just like I don't care if an AI submits a vulnerability report that is actually real.
- progbits 1y ago
  From the article itself (presumably added later): Disclosure: Certain sections of this content were grammatically refined/updated using AI assistance
- bluGill 1y ago
  Problem is AI sounds like normal people. There are a few clues, but good writers have always sounded like AI - except good writers are not making things up.

hacb 1y ago

I don't know why submitting a vulnerability on those platforms is still free. If reporters had to pay a little amount of money (let's say, 20-50$, or indexed to the maximum gain of a vulnerability in a given category) when submitting their report, maybe those would be of better quality. I know that this poses new problems (some people can't afford to spend this money), but it would be better than just wasting people's time.

Chilinot 1y ago
This point is brought up and discussed in the linked article.

nixpulvis 1y ago

I believe "Economic Friction" is truly the best and potentially strongest way to deal with this. It's good for the site collecting the fee, it's good for the projects being reported on and it doesn't negatively affect valid reports. It does exactly what we want by disincentivizing bad reports, either AI generated or not.

mmsc 1y ago

>First, the typical AI-powered reporter, especially one just pasting GPT output into a submission form, neither knows enough about the actual codebase being examined nor understands the security implications well enough to provide insight that projects need. How ironic, considering every time I've reported a complicated issue to a program on HackerOne, the triggers have completely rejected them because they do not understand the complicated codebase that they are triaging for. Also the curl examples given in TFA completely ignore recent developments, where curl's maintainers welcomed and fixed literally hundred of AI-found bugs: https://www.theregister.com/2025/10/02/curl_project_swamped_with_ai/ https://www.theregister.com/2025/10/02/curl_project_swamped_...

heymax054 1y ago

Semi-related: I posted this yesterday: https://news.ycombinator.com/item?id=45828917 https://news.ycombinator.com/item?id=45828917 Different models perform differently when it comes to catching/fixing security vulnerabilities.

samlinnfer 1y ago

Just add a country IP ban, we all know who is submitting these reports. Remember Hacktoberfest?

Cthulhu_ 1y ago
That's a game of whack-a-mole, they'd just use a VPN. And besides, no we don't "all know" who is submitting these reports, that's a generalization.
- samlinnfer 1y ago
  It doesn’t have to be perfect, it just has add friction so they stop spamming.
mschuster91 1y ago
> Just add a country IP ban, we all know who is submitting these reports. As much as I'd like to see Russia, China and India disconnected off of the wide Internet until they clean up shop with abusive actors, the Hacktoberfest stuff you're likely referring to doesn't have anything to do with your implication - that was just a chance at a free t-shirt [1] that caused all the noise. In ye olde times, you'd need to take care how you behaved in public because pulling off a stunt like that could reasonably lead to your company going out of business - but even a "small" company like DO is too big to fail from FAFO, much less ultra large corporations like Google that just run on sheer moat. IMHO, that is where we have to start - break up the giants, maybe that's enough of a warning signal to also alert "smaller" large companies to behave like citizens again. [1] https://domenic.me/hacktoberfest/ https://domenic.me/hacktoberfest/
- samlinnfer 1y ago
  IP banning India would have solved hacktoberfest spam 100% and maybe 99% of fake AI bug reports.

Cthulhu_ 1y ago

Very blunt maybe, but if individuals try to get internet points by doing frivolous security reports under their own name, should they be loudly pinned to a Wall of Shame to discourage the practice?

cheesecompiler 1y ago

> The problem isn't knowledge—it's incentives. > When you're volunteering out of love in a market society, you're setting yourself up to be exploited. I sound like a broken record but there's unifying causes to most issues I observe in the world. None of the proposed solutions address the cause (and they can't of course): public scrutiny doesn't do anything if account creation is zero-effort; monetary penalization will kill the submissions entirely. In a perfect world OSS maintainers would get paid properly. But, we've been doing this since the 90s, and all that's happened is OSS got deployed by private companies, concentrating the wealth and the economic benefits. When every hour is paid labour, you pick the AWS Kafka over spinning up your own cluster, or you run Linux in the cloud instead of your own metal. This will always keep happening so long as the incentives are what they are and survival hinges on capital. That people still put in their free time speaks to the beautiful nature of humans, but it's in spite of the current systems.

progbits 1y ago

Certain sections of this content were grammatically refined/updated using AI assistance, as English is not my first language. OP: I sympathize, but I would much rather read your original text, with typos and grammatical errors. By feeding it through the LLM you fix issues that are not really important but remove your own voice and get a bland slop identical to 90% of these slopblogs (which your's isn't!)

danieltk76 1y ago

At Vulnetic ai we deal with this by having a separate validator agent attack the vulnerability reported using a novel perspective separate of the hacking agent.

Sharlin 1y ago

> The incentive is to submit as many reports as possible and see what sticks, because even a 5% hit rate on a hundred submissions is better than the effort of manually verifying five findings. As I just commented in the other AI trust thread on the front page, this dynamic is funnily enough what any woman using online dating services has always been very familiar with. With the exact same tragedy of the commons that results. Except for the important difference that terrible profiles and intro messages have traditionally usually been very short and easily red-flagged. But that is, of course, now also changing or already changed due to LLMs. (Someone I follow on a certain social media platform just remarked that she got no less than fifty messages within a single hour of marking herself as "single". And she's just some average person, not a "star" of any sort.)

samuelknight 1y ago

You can address the issue by putting the report and the code base in a sandbox with an agent that tries to reproduce it. If it can't reproduce it then that should be a strike against the reporter. OSS projects should absolutely ban accounts that repetitively create reports that are of such low quality that it can't be recreated. IMO the Hacker One reputation mechanism is a good idea because it incentives users who operate in good faith and can serially produce findings.

mayhemducks 1y ago
And who pays for the tokens?
- noduerme 1y ago
  Sandbox a third AI that just bets on AI stocks and crypto. Add a fourth AI to check the third AI's bets, and a fifth one to go on forums and pump the relevant equities. A sixth AI can short sell when the fourth AI gets overheated.

bklosky 1y ago

Companies soliciting big bounties should charge a fee to submit, making slop costly. Steam, the video game publisher, does this. Developers need to pay a fee to list their game, and if their game sells sufficient volume, the fee is returned. Creating a separating equilibrium here is not hard, the hand wringing is weird to me.

8note 1y ago
adding cost to accounts is handy, but its not like that has solved steam's scam/slop problems. fake games by fake studios played by fake players is still a thing

theptip 1y ago

It seems pretty obvious that the bar needs to be raised. > A security report lands in your inbox. It claims there's a buffer overflow in a specific function. The report is well-formatted, includes CVE-style nomenclature, and uses appropriate technical language. Given how easy it is to generate a POC these days, I wonder if HackerOne needs to be pivoting hard into scaffolding to help bug hunters prove their vulns. - Claude skills/MCP for OSS projects - Attested logging/monitoring for API investigations (eg hosted BURP)

noduerme 1y ago

>> some actors with misaligned incentives interpret high submission volume as achievement Welcome to the Internet.

ok123456 1y ago

Require a docker/script that provides for the necessary conditions for the exploit, along with a POC. If something is impossible to provide a POC for, as it's more of a speculative attack, require vetting like arxiv. Most people's initial contributions are going to be more concrete exploits.

ceroxylon 1y ago

Surely there can be a workflow created to "fight fire with fire" and have an AI that reads reports, trained on the code base with explicit instructions to verify all of the telltale signs of slop...? If AI services can handle the nightmare of parsing emails and understanding the psychology of phishing, I am optimistic it can be done for OSS reports. It doesn't have to make the final judgement, just some sort of filter that automatically flags things like function calls that don't exist in the code.

stackedinserter 1y ago

So use AI to check AI reports, problem solved.

aidenn0 1y ago
I cannot tell if you are being serious or sarcastic.
- stackedinserter 1y ago
  50/50
- 8note 1y ago
  i dunno about problem solved, but id expect to be able to put together an agent that helps requesters format their reports well, and that could run the reproduction steps in a sandbox.

fleventynine 1y ago

"My attention is a limited resource. In order to prove that you're a serious human, please donate exactly $157.42 to the Rust maintainers fund and paste the receipt link here".

stuxf 1y ago

I agree with a lot of things said in this article, I also think some sort of centralized trust system for OSS bug bounty would be a really good solution to this problem > The downside is that it makes it harder for new researchers to enter the field, and it risks creating an insider club. I also think this concern can be largely mitigated or reduced to a nonissue. New researchers would have a trust score of zero for example, but people who consistently submit AI slop will have a very low score and can be filtered out fairly easily.

HiPhish 1y ago

We need a name for the type of attack where maintainers get overwhelmed by a massive amount of low-effort and false bug reports and contributions. I propose "Slop Overflow". That's right, you heard it here first, I came up with it :)

DrewADesign 1y ago

> By flooding bug bounty programs with AI-generated reports, they feel productive and entrepreneurial. Some genuinely believe the AI has found something real. Others know it's questionable but figure they'll let the maintainers sort it out. I’m not saying that AI hasn’t already given us useful things, but this is a symptom of one very negative change that’s drowned a lot of the positive out for many people: the competence gap used to be an automatic barrier for many things. For example, to get hired as a freelance developer, you had to have at least cargo-culted something together once or twice, and even if you were way overconfident in your capability, you probably knew you weren’t the real thing. However, the AI tools industry essentially markets free competence, and out of context for any given topic, that little disclaimer is meaningless. It’s essentially given people climbing the Dunning-Krueger Mt. Stupid the agency to produce garbage at a damaging volume that’s too plausible looking for them (or laypeople, for that matter) to realize it’s garbage. I also think somewhat nihilist people prone to get-rich-quick schemes (e.g. drop-shipping, NFTs) play these workflows like lottery tickets while remaining deliberately ignorant of their dubious value.

breakpointalpha 1y ago

Open Source maintainers should require a $10 deposit to submit a pull request. Even more so when there is a bounty payout. Refundable if the PR/report is accepted.

jruohonen 1y ago

I think this piece was a good summary of the state of affairs. If I would have written it, I would have perhaps mentioned that similar problems exist also in other domains, including science, arts, and media. Maybe the solutions might be similar too? I am particularly pointing toward the following quote that wasn't yet discussed here: "New reporters could be required to have established community members vouch for them, creating a web-of-trust model. This mirrors how the world worked before bug bounty platforms commodified security research. The only downside is, it risks creating an insider club."