10 ms·

AI generated security reports about curl

371 points by skeptrune 3y ago

minimaxir 3y ago
I'm more curious why and how this report got filed. Does curl provide financial incentives for filing CVEs? Or is it a misguided attempt at being helpful? (EDIT: from the blog post, there are indeed bug bounties for curl) What do you prompt a LLM with to get a more humanized output and code intermix for a CVE like this, as it doesn't have the typical ChatGPT tropes? There isn't an obvious indicator that it was LLM-generated until it hallucinated a user name and went into the third-person "raised by the triager".
- rwmj 3y ago
  curl does offer a bug bounty. Unfortunately there's a more fundamental issue: being someone who has filed CVEs can provide some credibility for your resume, that's the financial incentive here, and also the root cause of many many issues with the CVE system.
- xcdzvyn 3y ago
  > Does curl provide financial incentives for filing CVEs? Up to $10,000 according to their HackerOne page. I could also see this being a misguided attempt at being helpful, given the submitter clearly cannot read C. Though I'm not sure where you'd lie in the computer competency spectrum if you can use an LLM to find code bugs withoug realising they'll probably be hallucinated.
- richardwhiuk 3y ago
  Yes Curl offers a Bug Bounty - https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
- technion 3y ago
  I've seen people file cve requests 20 years ago with the output of "grep strcpy" as the entire report. All an llm bought to this modern version is the apologising, arguing and hallucinated code sample.
  - itishappy 3y ago
    Ah, the time wasting bits.
  - shadowgovt 3y ago
    It does make one wonder why anyone still uses strcpy in an open source project if it's going to be a perpetual source of having to flag down grep-identified bug reports forever.
    - flohofwoe 3y ago
      The recommended strncpy() is just as bad as strcpy() because you'll end up with a string that's not zero-terminated if the length of the copied string is >= the target buffer size. strlcpy() is a better option but is neither in the C nor POSIX standard. snprintf() is also a good option since C99 but a bit overkill.
      - nneonneo 3y ago
        The submitter’s proposed fix does add null termination, so it does work as expected. Still, strncpy is a horrible API in general. It even has the incredibly stupid behaviour of zeroing out the entire buffer past your string, which is almost never necessary and therefore just a waste of cycles. If zeroing is needed e.g. for security, it’s always better to include an obvious and explicit memset.
  - arp242 3y ago
    At least those could be dismissed with almost zero effort. "You clearly didn't spend any effort in your report, so I'm not going to spend a lot of effort replying". But with this it at least looks like they spent the effort, and even though you can suspect LLM chicanery, you can never be entirely sure, especially not from the initial message.
deleted 3y ago
[deleted]
philk10 3y ago
Seems related to this - https://news.ycombinator.com/item?id=38840907 https://news.ycombinator.com/item?id=38840907
jsnell 3y ago
I first thought this was a dupe of https://news.ycombinator.com/item?id=37904047 https://news.ycombinator.com/item?id=37904047, but turns out it's a different LLM-generated bogus vulnerability report about curl on hackerone.
- ecshafer 3y ago
  Great so I am not crazy. I read this and I coulve sworn I had seen this before. This is so similar to the last incident its weird. Are popular projects like Curl just going to have constant LLM written incidents opened against them as people try and get some line items on their resume?
  - shadowgovt 3y ago
    Probably. Eventually it might even make sense to accept some of the edits so that the auto-sniffers generating them stop complaining (not unlike making changes to working code that make a linter happy).
  - bombcar 3y ago
    Yes. As anyone who has an email associated with a website knows, you'll get random "vulnerability detection spam" requesting bounties. CVE is going to have to up their game.
- barryrandall 3y ago
  HackerOne's customers are the companies running bug bounty programs. You'd think they'd be a bit more careful about who they allow to spam their customers with clout-seeking LLM garbage.
  - nneonneo 3y ago
    First time contributors can have perfectly valid reports, and this particular report was initially written in a reasonably professional style that doesn’t immediately seem out of place (it’s the replies that make it obvious an LLM is being used). Unfortunately, this issue isn’t likely to go away any time soon, and will probably just get worse as the LLMs get used more widely for this type of work. What probably will happen is that maintainers will have to get better at identifying and screening out this kind of nonsense early, and for platforms to get better at banning people who submit bogus reports (but that’s really going to be a whack-a-mole game).
shadowgovt 3y ago
An important lesson they taught us in high school was how to tell the difference between elegantly-stated falsehoods and inelegantly-stated truths. ... but it can be difficult. People tend to use proper grammar and style as a first-pass filter for intelligent discourse, and getting the shape of language right is something that LLMs are very, very good at.
- OmarShehata 3y ago
  Wow, how well did this land for you at the time, and others in your class? This is a very tricky thing, and I think most adults aren't equipped to deal with this. (I think the average person is totally capable in telling these apart, the problem is that you kind of need to get used to reading things in a systematic way/put in some effort. It's very difficult to do this when scrolling through your phone late at night etc.)
jiveturkey 3y ago
interesting DoS! consuming the valuable time of the analyst. this could become a real problem.
CoastalCoder 3y ago
I'm trying to put my finger on why this pisses me off so much. Maybe because I'm guessing (I could be wrong) that this is an utterly selfish act, damaging to the common good, on the part of whoever submitted the CVE. I.e., it's like vandalizing a Habitat for Humanity office.
- shadowgovt 3y ago
  I can come up with an explanation that doesn't assume ill intent. It's not too terribly hard to run an auto-scanner for a common vulnerability pattern and then hook that scanner up to an LLM to generate English explanations (because, let's call the tech world what it is: well-formed English is likelier to pass the first-pass filter of not being BS than an English-as-a-second-language attempt to explain a problem). Looking at the reporter, they apparently have an undisclosed thank you from Adobe and Toyota. And when they interjected to try and explain an error in their machine mis-stating the name of the reviewer, the text they injected was likely not English-as-first-language. So I can imagine someone basically trying to run an auto-scanner for common vulns to highlight them for various parties to address, because addressing them increases software health globally. Problem is that it's false-positiving on some (admittedly very fragile, in the "only guarded by every human being writing the right code all the time" sense, i.e. the detector's not wrong that if the called function violates contract stuff will break) working code.
  - wrs 3y ago
    As I found out when I first put something up for review on HackerOne, there is a whole cottage industry of people just running scanners and blindly reporting whatever comes out. They are doing this to get something on their resume and maybe receive a bounty, not to improve global software health. It’s basically spam, because all the effort is on the part of the receiver to figure out if these reports make any sense. (And of course you can pay H1 to do that for you.)
  - flohofwoe 3y ago
    All that person had to do is just read the frigging code and check if the strcpy() can indeed lead to a memory corruption problem (and ideally create a reproducer) instead of wasting the curl maintainer's time with AI-generated bullshit. In this case it's quite obvious that it's an AI generating the bullshit, but we really need a mandatory disclaimer that something was generated by an AI so that a human can immediately break off any interaction instead of wasting time.
unglaublich 3y ago
The most worrying is that a few cents of LLM cost wasted a lot of expensive, significant engineering time. Imagine the amount of effort that we need to put into defusing all the bogus info generated right now. A bit like Brandolini's law.
- rurp 3y ago
  Yup, LLMs have the potential to ruin huge swaths of the internet, and I'm not sure it's a solvable problem. There are tells with the current models, but future models will be different and better. Detection and blocking could become an arms race that ends up being untenable to keep up with for many productive people and platforms.
rdtsc 3y ago
If it wasn't the frustration and time wasted it would be funny how dineshsec / dinesh_b is teaching Daniel how to use strncpy. First, they use some random handle to tag Daniel, then invent non-existing code as the "The code in question is:" snippet.
- juujian 3y ago
  This is a classic problem with LLM (misuse). The users wants to analyze something, but it's too long, so they chop it up into multiple requests. And by the time they get to the point, the original code snipped is long out the window and the model confidently spews up something that was never there (though it looks plausible).
  - rdtsc 3y ago
    It seems at some point a human (Dinesh B?) had reviewed and posted it. So, it's fine to use an LLM to summarize and analyze some piece of code, but then after Daniel responded that the analysis was junk, they persist at it, assuming Daniel was wrong, and just re-prompted LLM to paraphrase the crap again. Let's assume this person is doing it for clout, it's surprising they don't see how this behavior would hurt their own reputation.
    - tamimio 3y ago
      > but then after Daniel responded that the analysis was junk, they persist at it, assuming Daniel was wrong, and just re-prompted LLM to paraphrase the crap again. Lack of self awareness, you will be surprised by the amount of people that have that, that’s why I prefer to work with smart emotionally intelligent individuals than ones who supposedly are “smart” with higher GPA/degree and such, because that behavior will be their everyday job and eventually turning the none-sense argument into a political one.
aftbit 3y ago
This is an especially weird place to complain about length bounds checking, as no user-provided data is used here, and all the sizes are static at compile time. curl is putting a base64-encoded 16 byte random string (which turns into 25 ASCII bytes plus the null terminator \0) into a static 40 byte buffer. https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f4c066db6a9ea88/lib/ws.c#L575 https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f... Also, just because I'm curious... can anyone who groks C better than me explain why they're using this `keyval` local variable in the first place? Why not just set `heads[3].val = randstr` then `free()` it after the header data is processed? And why is `keyval` 40 bytes instead of 26 or 32?
- tom_ 3y ago
  It's presumably to reduce the number of places where you might need to call free, and then fail to do so? (This might already be happening in line 580? Though in practice perhaps that case never occurs.)
  - hnarn 3y ago
    How to get your open source project peer reviewed: post an LLM bug report, post it to HN and scour the comments for people reviewing your code ;-)
  - rstuart4133 3y ago
    I doubt it's to reduce the number of calls to free. After all, there must be exactly one call to free() for every call to malloc(). It looks more like a coding habit this particular programmer uses to avoid forgetting to call free(). He puts it as close to the call to malloc() as possible, which reduces the changes of forgetting. It's interesting code to look at, if you are a C programmer. As you noted, his heroic efforts to always call free fails if line 580 gets executed. It never will of course, so him adding 580 reveals a certain paranoia. Perfectly understandable in a C programmer, but for most line 578 would be enough to assuage the paranoia. Others (me, for instance) would make 578 an assert, and get rid of 580. At the 1000ft level the result is the presumably the same in both cases: the program exits after printing an error on stderr, but the error message will be different. The assert makes it plain it's an internal failure, but the current method may be mistaken for a externally triggered error. Or he could have sidestepped the overflow checks and free() by using salloc() - but maybe it doesn't exist on platforms he ports to. I would do a lot of other things differently too. I guess it just shows the way old C programmers deal with the repeated beatings inflicted by the language varies a lot.
yashap 3y ago
“Beg bounty” ppl have already made it a major pain in the ass to have a bug bounty program, even when they were real people who had to take the time to craft “bug reports” about basically nothing. But if you mix in LLMs, generating bogus reports at essentially no cost, this could get really out of control. Could be the end of bug bounty programs IMO. Or, maybe we just need to lock them down more? Like you need to apply to become part of the bug bounty program, which involves some sort of cheap-to-perform check that you’re a real person and actual security researcher, looking to find real, impactful security bugs. And only ppl admitted into the program can submit bugs and collect financial rewards.
- masklinn 3y ago
  There are platforms which do that yeah, they have "pools" of known researchers with tracked statuses, and you can tune how open your program is. Some also have triage staff, but depending how typical your project is that can be very hit or miss.
  - yashap 3y ago
    Yeah, a decently locked down program seems like a good idea! At the startups I’ve worked at, the programs have been very open (basically “email us”), and it does lead to a fair bit of beg bounty spam, even without LLMs in the mix.
- donmcronald 3y ago
  Maybe have a submission fee? I don't know if that helps, but it would be a deterrent for massive amounts of machine generated garbage. The worst case in my opinion is massive amounts of AI garbage being submitted which will require equally bad AI filtering to "solve" the problem with the result being an overall reduction on quality for everyone trying to participate in good faith.
  - yashap 3y ago
    That’s an interesting idea, though I’m sure you’d be inviting some scathing blog posts! But you could maybe mitigate that by donating all fees to a charity? Or even a hybrid: - If you’re a “trusted security researcher”, you can submit for free - Else, there’s a small fee (on the order of a few dollars per submission)
dexwiz 3y ago
Certainly! Let me elaborate on the concerns raised by the triager This is typical LLM speak, it sounds like a robot butler. I don’t think I have encountered a single person who writes likes this. But it’s also got a weird 3rd person reference that indicates there is another party that is promoting for a response. I am okay with LLMs having a specific voice that makes them identifiable. My worry is that people will start talking like LLMs instead of LLMs sounding like people.
- nicklecompte 3y ago
  Daniel Stenberg[1] brought up a good point: the complexity here is that curl is used all over the world, and there's certainly nothing wrong with a non-English speaker using an LLM to help write their bug report. So the superficial giveaways that the English text is LLM-generated doesn't necessarily mean the report's content was LLM-generated. [1] https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
  - ericbarrett 3y ago
    He makes a great point, but I think writing in an authentic, un-augmented voice will very quickly become the only way to be noticed. Which is a shame for otherwise benign uses of the tool.
  - fnordpiglet 3y ago
    That’s absolutely fair IMO, but they aren’t use it right if that’s what’s happening. LLMs can do a decent job translating, and they should have written their bug report in their native language and asked for a translation. Then translated responses, wrote their response, and translated back This appears to be “write a bug report about X” then “write a response to triager for their reply Y” without intermediating, let alone factually checking the output. That use doesn’t fall prey to LLM voice because the translation is of your text and phrasing. God bless people who use LLMs to improve their life, translate, etc. But using them to think isn’t acceptable.
  - skepticATX 3y ago
ranting-moth 3y ago
Welcome to the early days of the internet post AI.
OhMeadhbh 3y ago
I think this is a person imitating an LLM. For instance, it seems to "understand" that 15 is greater than 10. I've yet to find an LLM that can internalize that.
- pests 3y ago
  What? I have had no issues ever comparing numbers in an LLM context. > Me: you have the numbers 3, 15, 10, and 72. can you put them in order, from least to greatest? > GPT3.5Turbo: Sure! The numbers in order from least to greatest are: 3, 10, 15, 72. > Me: now add 15, 13, 0, 293, and 1023 to the list > GPT3.5Turbo: If you add 15, 13, 0, 293, and 1023 to the list of numbers we had before (3, 15, 10, and 72), the resulting list would be: 0, 3, 10, 13, 15, 15, 72, 293, 1023 Seems to handle duplicates, larger factors having a lesser leading digit, and making changes to it. > Me: now add -5 and 5 > GPT3.5Turbo: Adding -5 and 5 to the previous list of numbers in order, we have: -5, 0, 3, 5, 10, 13, 15, 15, 72, 293, 1023 Negatives are fine too.
  - OhMeadhbh 3y ago
    Hmm... that's not been my experience. But I wasn't using GPT3.5Turbo.
    - dexwiz 3y ago
      ChatGPT does seem to have gotten better at math. It was pretty garbage at it about a year ago, and really easy to trick it so you could get some sweet "haha, LLM dumb content." Choose a random number between 100 and 200, divide it by 5, multiple it by the original number, add 10, and then multiple by pi. What is the final number? ChatGPT3.5: Certainly, let's choose a random number between 100 and 200. Let's say the randomly chosen number is 150: Chosen number: 150 150 / 5 = 30 30 \* 150 = 4500 4500 + 10 = 4510 4510 \* π ≈ 14171.03 (rounded to 2 decimal places) So, the final number is approximately 14171.03 when using the randomly chosen number of 150. If you were to choose a different random number within the specified range, you would get a different result. ChatGPT4: analyzing import random import math # Choose a random number between 100 and 200 random_number = random.randint(100, 200) # Perform the calculations result = (((random_number / 5) \* random_number) + 10) \* math.pi result, random_number Result (10487.264596213448, 129) The final number, after choosing 129 as the random number between 100 and 200, is approximately 10487.26. If you ask ChatGPT 4 for a multiple step equation it appears to first translate the equation into python, run the script, and then give you the output.
scopeh 3y ago
LLMs being LLMs. Reading this makes me a feel a little more secure in my job. They have a long way to go.
- samjanis 3y ago
  Certainly!
airstrike 3y ago
Why are we calling the subject "LLM" as if it had agency? Clickbaity title
- richbell 3y ago
  I wouldn't consider it clickbait any more than "pedestrian struck by vehcile." It's implied that LLMs are tools.
  - airstrike 3y ago
    More like "gun kills person"... "Passenger struck by vehicle" is in the passive voice, which makes it very different. I imagine you agree "vehicle strikes passenger" would sound very strange, would it not?
    - richbell 3y ago
      I think "gun kills person" sounds strange, but "vehicle hits person" does not. I've seen countless articles with a headline of that nature. http://www.nbcconnecticut.com/news/local/metro-north-service-suspended-after-train-hits-person-near-ansonia/3156837/%3famp=1 http://www.nbcconnecticut.com/news/local/metro-north-service... Chalk it up to English being weird, I guess.
      - airstrike 3y ago
        Fair enough and thanks for the example! Not to move goalposts too much, but I still think the LLM scenario is a bit different precisely because it is not necessarily implied that LLM are tools given they are closely related to the concept of "intelligence"
- deleted 3y ago
  [deleted]
layer8 3y ago
I believe they have a bit of a point, because it isn’t locally obvious that randlen will match the zero-terminated length of randstr, at https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f4c066db6a9ea88/lib/ws.c#L579 https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f.... It depends on the exact behavior and interface contract of the external function Curl_base64_encode(). A static analyzer might rightfully flag this. The LLM does a bad job of explaining this though.
deleted 3y ago
[deleted]
esozbek 3y ago
I wonder if or when someone will also open a PR to "fix" this, at least for the lulzs
- skeptrune 3y ago
  That would be a fantastic meme PR lmao
lsy 3y ago
It's kind of fascinating that we've taken the lowest-bandwidth way of proving agency and effort - writing - and made it tremendously more labor-intensive to deduce whether any agency or effort were involved at all. I think the knock-on effects are probably huge - not only are the reporter and the maintainer both wasting time here that could be better spent on something useful, but the whole bug-bounty-crowdsourced-CVE process is now compromised by a lower signal-to-noise ratio and the probable outcome of raising barriers to submission as a means of dealing with the spam, meaning fewer bugs discovered and fixed, more security vulnerabilities, and everything that results from that. The same dynamic is at play in other sectors, meaning we find ourselves increasingly unable to assign any trust to product reviews, court filings, recipes, how-to guides, medical advice, and so forth. One of the promises of the internet was the rapid expansion of content through democratization of publishing, but I think we're witnessing the gutting of any benefit that had left.
leejo 3y ago
Spurious CVEs have been around for a long time, it seems that people feeding open source codebases into LLMs and asking it to point out potential issues is just the next step in that approach. LLMs don't understand the code they're being fed, and it seems the people feeding the LLMs don't understand it either...
Arch-TK 3y ago
Should just be using memcpy and not strcpy, or strncpy (which is definitely the worst option of the three). The code already knows the size of the source buffer and has checked if it will fit in the destination buffer, there is no reason to call strcpy and have it strlen the string unnecessarily. Genuinely the LLM recommendation is something I would actively discourage. If you don't know the size, and don't care about silent truncation, and don't care about performance (strncpy will unnecessarily zero the rest of the buffer), just use snprintf. Unless you're dealing with some UI or something, you probably SHOULD care about truncation and in that case strncpy won't save you.
smsm42 3y ago
Oh my. So now in addition to bounty beggars we'll have bogus generated security report that can waste hours of valuable dev time chasing after LLM hallucinations.
bananapub 3y ago
it really is fascinating to find out that there's a bunch of people eager to make the internet worse but have been held back by being unwilling to write a bunch of text / draw a lot of pictures / etc, but now can be lazy and harmful using LLM/diffusion etc.
pavel_lishin 3y ago
My favorite part is that the submitter of the Bard-based faux report got miffed that people weren't communicating with them in the way that they preferred.
feverzsj 3y ago
Today's AI is still bs generator, but lots of jobs are bs, so it has a value for generating bs.
gumballindie 3y ago
I am genuinely convinced that people who think llms are intelligent and rely on them for everything are idiots.
tamimio 3y ago
Fantastic.. up until recently cybersecurity was somehow immune from that junk and it was primarily in code development.. I believe the real problem in here is the ethics rather than AI itself, although AI is making it easier for those “scammers”, used to be in crafting an email for phishing, to a site UI, all the way to bug security report, as long as it looks “legit”, then it should pass!!

371 points by skeptrune 3y ago

I'm more curious why and how this report got filed. Does curl provide financial incentives for filing CVEs? Or is it a misguided attempt at being helpful? (EDIT: from the blog post, there are indeed bug bounties for curl) What do you prompt a LLM with to get a more humanized output and code intermix for a CVE like this, as it doesn't have the typical ChatGPT tropes? There isn't an obvious indicator that it was LLM-generated until it hallucinated a user name and went into the third-person "raised by the triager".

rwmj 3y ago
curl does offer a bug bounty. Unfortunately there's a more fundamental issue: being someone who has filed CVEs can provide some credibility for your resume, that's the financial incentive here, and also the root cause of many many issues with the CVE system.
xcdzvyn 3y ago
> Does curl provide financial incentives for filing CVEs? Up to $10,000 according to their HackerOne page. I could also see this being a misguided attempt at being helpful, given the submitter clearly cannot read C. Though I'm not sure where you'd lie in the computer competency spectrum if you can use an LLM to find code bugs withoug realising they'll probably be hallucinated.
richardwhiuk 3y ago
Yes Curl offers a Bug Bounty - https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
technion 3y ago
I've seen people file cve requests 20 years ago with the output of "grep strcpy" as the entire report. All an llm bought to this modern version is the apologising, arguing and hallucinated code sample.
- itishappy 3y ago
  Ah, the time wasting bits.
- shadowgovt 3y ago
  It does make one wonder why anyone still uses strcpy in an open source project if it's going to be a perpetual source of having to flag down grep-identified bug reports forever.
  - flohofwoe 3y ago
    The recommended strncpy() is just as bad as strcpy() because you'll end up with a string that's not zero-terminated if the length of the copied string is >= the target buffer size. strlcpy() is a better option but is neither in the C nor POSIX standard. snprintf() is also a good option since C99 but a bit overkill.
    - nneonneo 3y ago
      The submitter’s proposed fix does add null termination, so it does work as expected. Still, strncpy is a horrible API in general. It even has the incredibly stupid behaviour of zeroing out the entire buffer past your string, which is almost never necessary and therefore just a waste of cycles. If zeroing is needed e.g. for security, it’s always better to include an obvious and explicit memset.
- arp242 3y ago
  At least those could be dismissed with almost zero effort. "You clearly didn't spend any effort in your report, so I'm not going to spend a lot of effort replying". But with this it at least looks like they spent the effort, and even though you can suspect LLM chicanery, you can never be entirely sure, especially not from the initial message.

deleted 3y ago

[deleted]

philk10 3y ago

Seems related to this - https://news.ycombinator.com/item?id=38840907 https://news.ycombinator.com/item?id=38840907

jsnell 3y ago

I first thought this was a dupe of https://news.ycombinator.com/item?id=37904047 https://news.ycombinator.com/item?id=37904047, but turns out it's a different LLM-generated bogus vulnerability report about curl on hackerone.

ecshafer 3y ago
Great so I am not crazy. I read this and I coulve sworn I had seen this before. This is so similar to the last incident its weird. Are popular projects like Curl just going to have constant LLM written incidents opened against them as people try and get some line items on their resume?
- shadowgovt 3y ago
  Probably. Eventually it might even make sense to accept some of the edits so that the auto-sniffers generating them stop complaining (not unlike making changes to working code that make a linter happy).
- bombcar 3y ago
  Yes. As anyone who has an email associated with a website knows, you'll get random "vulnerability detection spam" requesting bounties. CVE is going to have to up their game.
barryrandall 3y ago
HackerOne's customers are the companies running bug bounty programs. You'd think they'd be a bit more careful about who they allow to spam their customers with clout-seeking LLM garbage.
- nneonneo 3y ago
  First time contributors can have perfectly valid reports, and this particular report was initially written in a reasonably professional style that doesn’t immediately seem out of place (it’s the replies that make it obvious an LLM is being used). Unfortunately, this issue isn’t likely to go away any time soon, and will probably just get worse as the LLMs get used more widely for this type of work. What probably will happen is that maintainers will have to get better at identifying and screening out this kind of nonsense early, and for platforms to get better at banning people who submit bogus reports (but that’s really going to be a whack-a-mole game).

shadowgovt 3y ago

An important lesson they taught us in high school was how to tell the difference between elegantly-stated falsehoods and inelegantly-stated truths. ... but it can be difficult. People tend to use proper grammar and style as a first-pass filter for intelligent discourse, and getting the shape of language right is something that LLMs are very, very good at.

OmarShehata 3y ago
Wow, how well did this land for you at the time, and others in your class? This is a very tricky thing, and I think most adults aren't equipped to deal with this. (I think the average person is totally capable in telling these apart, the problem is that you kind of need to get used to reading things in a systematic way/put in some effort. It's very difficult to do this when scrolling through your phone late at night etc.)

jiveturkey 3y ago

interesting DoS! consuming the valuable time of the analyst. this could become a real problem.

CoastalCoder 3y ago

I'm trying to put my finger on why this pisses me off so much. Maybe because I'm guessing (I could be wrong) that this is an utterly selfish act, damaging to the common good, on the part of whoever submitted the CVE. I.e., it's like vandalizing a Habitat for Humanity office.

shadowgovt 3y ago
I can come up with an explanation that doesn't assume ill intent. It's not too terribly hard to run an auto-scanner for a common vulnerability pattern and then hook that scanner up to an LLM to generate English explanations (because, let's call the tech world what it is: well-formed English is likelier to pass the first-pass filter of not being BS than an English-as-a-second-language attempt to explain a problem). Looking at the reporter, they apparently have an undisclosed thank you from Adobe and Toyota. And when they interjected to try and explain an error in their machine mis-stating the name of the reviewer, the text they injected was likely not English-as-first-language. So I can imagine someone basically trying to run an auto-scanner for common vulns to highlight them for various parties to address, because addressing them increases software health globally. Problem is that it's false-positiving on some (admittedly very fragile, in the "only guarded by every human being writing the right code all the time" sense, i.e. the detector's not wrong that if the called function violates contract stuff will break) working code.
- wrs 3y ago
  As I found out when I first put something up for review on HackerOne, there is a whole cottage industry of people just running scanners and blindly reporting whatever comes out. They are doing this to get something on their resume and maybe receive a bounty, not to improve global software health. It’s basically spam, because all the effort is on the part of the receiver to figure out if these reports make any sense. (And of course you can pay H1 to do that for you.)
- flohofwoe 3y ago
  All that person had to do is just read the frigging code and check if the strcpy() can indeed lead to a memory corruption problem (and ideally create a reproducer) instead of wasting the curl maintainer's time with AI-generated bullshit. In this case it's quite obvious that it's an AI generating the bullshit, but we really need a mandatory disclaimer that something was generated by an AI so that a human can immediately break off any interaction instead of wasting time.

unglaublich 3y ago

The most worrying is that a few cents of LLM cost wasted a lot of expensive, significant engineering time. Imagine the amount of effort that we need to put into defusing all the bogus info generated right now. A bit like Brandolini's law.

rurp 3y ago
Yup, LLMs have the potential to ruin huge swaths of the internet, and I'm not sure it's a solvable problem. There are tells with the current models, but future models will be different and better. Detection and blocking could become an arms race that ends up being untenable to keep up with for many productive people and platforms.

rdtsc 3y ago

If it wasn't the frustration and time wasted it would be funny how dineshsec / dinesh_b is teaching Daniel how to use strncpy. First, they use some random handle to tag Daniel, then invent non-existing code as the "The code in question is:" snippet.

juujian 3y ago
This is a classic problem with LLM (misuse). The users wants to analyze something, but it's too long, so they chop it up into multiple requests. And by the time they get to the point, the original code snipped is long out the window and the model confidently spews up something that was never there (though it looks plausible).
- rdtsc 3y ago
  It seems at some point a human (Dinesh B?) had reviewed and posted it. So, it's fine to use an LLM to summarize and analyze some piece of code, but then after Daniel responded that the analysis was junk, they persist at it, assuming Daniel was wrong, and just re-prompted LLM to paraphrase the crap again. Let's assume this person is doing it for clout, it's surprising they don't see how this behavior would hurt their own reputation.
  - tamimio 3y ago
    > but then after Daniel responded that the analysis was junk, they persist at it, assuming Daniel was wrong, and just re-prompted LLM to paraphrase the crap again. Lack of self awareness, you will be surprised by the amount of people that have that, that’s why I prefer to work with smart emotionally intelligent individuals than ones who supposedly are “smart” with higher GPA/degree and such, because that behavior will be their everyday job and eventually turning the none-sense argument into a political one.

aftbit 3y ago

This is an especially weird place to complain about length bounds checking, as no user-provided data is used here, and all the sizes are static at compile time. curl is putting a base64-encoded 16 byte random string (which turns into 25 ASCII bytes plus the null terminator \0) into a static 40 byte buffer. https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f4c066db6a9ea88/lib/ws.c#L575 https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f... Also, just because I'm curious... can anyone who groks C better than me explain why they're using this `keyval` local variable in the first place? Why not just set `heads[3].val = randstr` then `free()` it after the header data is processed? And why is `keyval` 40 bytes instead of 26 or 32?

tom_ 3y ago
It's presumably to reduce the number of places where you might need to call free, and then fail to do so? (This might already be happening in line 580? Though in practice perhaps that case never occurs.)
- hnarn 3y ago
  How to get your open source project peer reviewed: post an LLM bug report, post it to HN and scour the comments for people reviewing your code ;-)
- rstuart4133 3y ago
  I doubt it's to reduce the number of calls to free. After all, there must be exactly one call to free() for every call to malloc(). It looks more like a coding habit this particular programmer uses to avoid forgetting to call free(). He puts it as close to the call to malloc() as possible, which reduces the changes of forgetting. It's interesting code to look at, if you are a C programmer. As you noted, his heroic efforts to always call free fails if line 580 gets executed. It never will of course, so him adding 580 reveals a certain paranoia. Perfectly understandable in a C programmer, but for most line 578 would be enough to assuage the paranoia. Others (me, for instance) would make 578 an assert, and get rid of 580. At the 1000ft level the result is the presumably the same in both cases: the program exits after printing an error on stderr, but the error message will be different. The assert makes it plain it's an internal failure, but the current method may be mistaken for a externally triggered error. Or he could have sidestepped the overflow checks and free() by using salloc() - but maybe it doesn't exist on platforms he ports to. I would do a lot of other things differently too. I guess it just shows the way old C programmers deal with the repeated beatings inflicted by the language varies a lot.

yashap 3y ago

“Beg bounty” ppl have already made it a major pain in the ass to have a bug bounty program, even when they were real people who had to take the time to craft “bug reports” about basically nothing. But if you mix in LLMs, generating bogus reports at essentially no cost, this could get really out of control. Could be the end of bug bounty programs IMO. Or, maybe we just need to lock them down more? Like you need to apply to become part of the bug bounty program, which involves some sort of cheap-to-perform check that you’re a real person and actual security researcher, looking to find real, impactful security bugs. And only ppl admitted into the program can submit bugs and collect financial rewards.

masklinn 3y ago
There are platforms which do that yeah, they have "pools" of known researchers with tracked statuses, and you can tune how open your program is. Some also have triage staff, but depending how typical your project is that can be very hit or miss.
- yashap 3y ago
  Yeah, a decently locked down program seems like a good idea! At the startups I’ve worked at, the programs have been very open (basically “email us”), and it does lead to a fair bit of beg bounty spam, even without LLMs in the mix.
donmcronald 3y ago
Maybe have a submission fee? I don't know if that helps, but it would be a deterrent for massive amounts of machine generated garbage. The worst case in my opinion is massive amounts of AI garbage being submitted which will require equally bad AI filtering to "solve" the problem with the result being an overall reduction on quality for everyone trying to participate in good faith.
- yashap 3y ago
  That’s an interesting idea, though I’m sure you’d be inviting some scathing blog posts! But you could maybe mitigate that by donating all fees to a charity? Or even a hybrid: - If you’re a “trusted security researcher”, you can submit for free - Else, there’s a small fee (on the order of a few dollars per submission)

dexwiz 3y ago

Certainly! Let me elaborate on the concerns raised by the triager This is typical LLM speak, it sounds like a robot butler. I don’t think I have encountered a single person who writes likes this. But it’s also got a weird 3rd person reference that indicates there is another party that is promoting for a response. I am okay with LLMs having a specific voice that makes them identifiable. My worry is that people will start talking like LLMs instead of LLMs sounding like people.

nicklecompte 3y ago
Daniel Stenberg[1] brought up a good point: the complexity here is that curl is used all over the world, and there's certainly nothing wrong with a non-English speaker using an LLM to help write their bug report. So the superficial giveaways that the English text is LLM-generated doesn't necessarily mean the report's content was LLM-generated. [1] https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
- ericbarrett 3y ago
  He makes a great point, but I think writing in an authentic, un-augmented voice will very quickly become the only way to be noticed. Which is a shame for otherwise benign uses of the tool.
- fnordpiglet 3y ago
  That’s absolutely fair IMO, but they aren’t use it right if that’s what’s happening. LLMs can do a decent job translating, and they should have written their bug report in their native language and asked for a translation. Then translated responses, wrote their response, and translated back This appears to be “write a bug report about X” then “write a response to triager for their reply Y” without intermediating, let alone factually checking the output. That use doesn’t fall prey to LLM voice because the translation is of your text and phrasing. God bless people who use LLMs to improve their life, translate, etc. But using them to think isn’t acceptable.
- skepticATX 3y ago

ranting-moth 3y ago

Welcome to the early days of the internet post AI.

OhMeadhbh 3y ago

I think this is a person imitating an LLM. For instance, it seems to "understand" that 15 is greater than 10. I've yet to find an LLM that can internalize that.

pests 3y ago
What? I have had no issues ever comparing numbers in an LLM context. > Me: you have the numbers 3, 15, 10, and 72. can you put them in order, from least to greatest? > GPT3.5Turbo: Sure! The numbers in order from least to greatest are: 3, 10, 15, 72. > Me: now add 15, 13, 0, 293, and 1023 to the list > GPT3.5Turbo: If you add 15, 13, 0, 293, and 1023 to the list of numbers we had before (3, 15, 10, and 72), the resulting list would be: 0, 3, 10, 13, 15, 15, 72, 293, 1023 Seems to handle duplicates, larger factors having a lesser leading digit, and making changes to it. > Me: now add -5 and 5 > GPT3.5Turbo: Adding -5 and 5 to the previous list of numbers in order, we have: -5, 0, 3, 5, 10, 13, 15, 15, 72, 293, 1023 Negatives are fine too.
- OhMeadhbh 3y ago
  Hmm... that's not been my experience. But I wasn't using GPT3.5Turbo.
  - dexwiz 3y ago
    ChatGPT does seem to have gotten better at math. It was pretty garbage at it about a year ago, and really easy to trick it so you could get some sweet "haha, LLM dumb content." Choose a random number between 100 and 200, divide it by 5, multiple it by the original number, add 10, and then multiple by pi. What is the final number? ChatGPT3.5: Certainly, let's choose a random number between 100 and 200. Let's say the randomly chosen number is 150: Chosen number: 150 150 / 5 = 30 30 \* 150 = 4500 4500 + 10 = 4510 4510 \* π ≈ 14171.03 (rounded to 2 decimal places) So, the final number is approximately 14171.03 when using the randomly chosen number of 150. If you were to choose a different random number within the specified range, you would get a different result. ChatGPT4: analyzing import random import math # Choose a random number between 100 and 200 random_number = random.randint(100, 200) # Perform the calculations result = (((random_number / 5) \* random_number) + 10) \* math.pi result, random_number Result (10487.264596213448, 129) The final number, after choosing 129 as the random number between 100 and 200, is approximately 10487.26. If you ask ChatGPT 4 for a multiple step equation it appears to first translate the equation into python, run the script, and then give you the output.

scopeh 3y ago

LLMs being LLMs. Reading this makes me a feel a little more secure in my job. They have a long way to go.

samjanis 3y ago
Certainly!

airstrike 3y ago

Why are we calling the subject "LLM" as if it had agency? Clickbaity title

richbell 3y ago
I wouldn't consider it clickbait any more than "pedestrian struck by vehcile." It's implied that LLMs are tools.
- airstrike 3y ago
  More like "gun kills person"... "Passenger struck by vehicle" is in the passive voice, which makes it very different. I imagine you agree "vehicle strikes passenger" would sound very strange, would it not?
  - richbell 3y ago
    I think "gun kills person" sounds strange, but "vehicle hits person" does not. I've seen countless articles with a headline of that nature. http://www.nbcconnecticut.com/news/local/metro-north-service-suspended-after-train-hits-person-near-ansonia/3156837/%3famp=1 http://www.nbcconnecticut.com/news/local/metro-north-service... Chalk it up to English being weird, I guess.
    - airstrike 3y ago
      Fair enough and thanks for the example! Not to move goalposts too much, but I still think the LLM scenario is a bit different precisely because it is not necessarily implied that LLM are tools given they are closely related to the concept of "intelligence"
deleted 3y ago
[deleted]

layer8 3y ago

I believe they have a bit of a point, because it isn’t locally obvious that randlen will match the zero-terminated length of randstr, at https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f4c066db6a9ea88/lib/ws.c#L579 https://github.com/curl/curl/blob/1d8e8c9ad1ff3351386422535f.... It depends on the exact behavior and interface contract of the external function Curl_base64_encode(). A static analyzer might rightfully flag this. The LLM does a bad job of explaining this though.

deleted 3y ago

[deleted]

esozbek 3y ago

I wonder if or when someone will also open a PR to "fix" this, at least for the lulzs

skeptrune 3y ago
That would be a fantastic meme PR lmao

lsy 3y ago

It's kind of fascinating that we've taken the lowest-bandwidth way of proving agency and effort - writing - and made it tremendously more labor-intensive to deduce whether any agency or effort were involved at all. I think the knock-on effects are probably huge - not only are the reporter and the maintainer both wasting time here that could be better spent on something useful, but the whole bug-bounty-crowdsourced-CVE process is now compromised by a lower signal-to-noise ratio and the probable outcome of raising barriers to submission as a means of dealing with the spam, meaning fewer bugs discovered and fixed, more security vulnerabilities, and everything that results from that. The same dynamic is at play in other sectors, meaning we find ourselves increasingly unable to assign any trust to product reviews, court filings, recipes, how-to guides, medical advice, and so forth. One of the promises of the internet was the rapid expansion of content through democratization of publishing, but I think we're witnessing the gutting of any benefit that had left.

leejo 3y ago

Spurious CVEs have been around for a long time, it seems that people feeding open source codebases into LLMs and asking it to point out potential issues is just the next step in that approach. LLMs don't understand the code they're being fed, and it seems the people feeding the LLMs don't understand it either...

Arch-TK 3y ago

Should just be using memcpy and not strcpy, or strncpy (which is definitely the worst option of the three). The code already knows the size of the source buffer and has checked if it will fit in the destination buffer, there is no reason to call strcpy and have it strlen the string unnecessarily. Genuinely the LLM recommendation is something I would actively discourage. If you don't know the size, and don't care about silent truncation, and don't care about performance (strncpy will unnecessarily zero the rest of the buffer), just use snprintf. Unless you're dealing with some UI or something, you probably SHOULD care about truncation and in that case strncpy won't save you.

smsm42 3y ago

Oh my. So now in addition to bounty beggars we'll have bogus generated security report that can waste hours of valuable dev time chasing after LLM hallucinations.

bananapub 3y ago

it really is fascinating to find out that there's a bunch of people eager to make the internet worse but have been held back by being unwilling to write a bunch of text / draw a lot of pictures / etc, but now can be lazy and harmful using LLM/diffusion etc.

pavel_lishin 3y ago

My favorite part is that the submitter of the Bard-based faux report got miffed that people weren't communicating with them in the way that they preferred.

feverzsj 3y ago

Today's AI is still bs generator, but lots of jobs are bs, so it has a value for generating bs.

gumballindie 3y ago

I am genuinely convinced that people who think llms are intelligent and rely on them for everything are idiots.

tamimio 3y ago

Fantastic.. up until recently cybersecurity was somehow immune from that junk and it was primarily in code development.. I believe the real problem in here is the ethics rather than AI itself, although AI is making it easier for those “scammers”, used to be in crafting an email for phishing, to a site UI, all the way to bug security report, as long as it looks “legit”, then it should pass!!