10 ms·

Hallucineted CVE against Curl: someone asked Bard to find a vulnerability

200 points by martijnarts 3y ago

abnercoimbre 3y ago
> you did not find anything worthy of reporting. You were fooled by an AI into believing that. The author's right. Reading the report I was stunned; the person disclosing the so-called vulnerability said: > To replicate the issue, I have searched in the Bard about this vulnerability. Does Bard clearly warn to never rely on it for facts? I know OpenAI says "ChatGPT may give you inaccurate information" at the start of each session.
- xd1936 3y ago
  Oh yeah. Google has warnings like "Bard may display inaccurate or offensive information that doesn’t represent Google’s views" all over it; Permanently in the footer, on splash pages, etc.
  - abnercoimbre 3y ago
    Well Google has a branding problem with Bard... because everyone knows Google for search. "Surely Bard must be a reliable engine too."
- filterfiber 3y ago
  > Does Bard clearly warn to never rely on it for facts? I know OpenAI says "ChatGPT may give you inaccurate information" at the start of each session. I know I shouldn't be, but I'm surprised the disclosure is even needed. People clearly don't understand how LLMs work - LLM's predict text. That's it, they're glorified autocomplete (that's really good). When their prediction is wrong we call it a "hallucination" for some reason. Humans do the same thing all the time. Of course it's not always correct!
  - masklinn 3y ago
    > People clearly don't understand how LLMs work - Of course not. Most developers don't understand how LLM work, even roughly. > Humans do the same thing all the time. Of course it's not always correct! The difference is that LLMs can not acknowledge incompetence, are always confidently incorrect, and will never reach a stopping point, at best they'll start going circular.
    - nerpderp82 3y ago
      Everything out of an LLM is a confabulation, but you can constrain the output space of that confab by restraining it with proper prompting. You could ask it to put confidence intervals for each one of its sentences (ask it in the prompt), but those will be confabulated as well, but will give it some self doubt, as for now, it hasn't been programmed with any. Probably costs a lot more in power to run it with doubt. :) Edit, I played around with this. It looks like GPT4 has a guard against asking for this. It flat out refused with the two prompts I gave it to include confidence intervals. Maybe that is a good thing.
    - lazide 3y ago
      At least they don’t control any militaries yet, since the natural outgrowth of that is making reality meet their answers. Or at least it would be if they were human!
    - fomine3 3y ago
      > The difference is that LLMs can not acknowledge incompetence, are always confidently incorrect, and will never reach a stopping point, at best they'll start going circular. Like a smartass?
orf 3y ago
> I responsibly disclosed the information as soon as I found it. I believe there is a better way to communicate to the researchers, and I hope that the curl staff can implement it for future submissions to maintain a better relationship with the researcher community. Thank you! … yeah…
- HtmlProgrammer 3y ago
  Poor fella was embarrassed and looking to throw anything back at them
- hypeatei 3y ago
  It looks like AI generated that response.
evrimoztamur 3y ago
If not malicious, then this shows that there are people out there who don't quite know how much to rely on LLMs or understand the limits of their capabilities. It's distressing.
- filterfiber 3y ago
  The average person sadly just hears the marketed "artificial intelligence" and doesn't grasp that it simply predicts text. It's really good at predicting text we like, but that's all it does. It shouldn't be surprising that sometimes it's prediction is either wrong or unwanted. Interestingly even intelligent, problem solving, educated humans "incorrectly predict" all the time.
  - kordlessagain 3y ago
    > It's really good at predicting text we like, but that's all it does. It's important to recognize that predicting text is not merely about guessing the next letter or word, but rather a complex set of probabilities grounded in language and context. When we look at language, we might see intricate relationships between letters, words, and ideas. Starting with individual letters, like 't,' we can assign probabilities to their occurrence based on the language and alphabet we've studied. These probabilities enable us to anticipate the next character in a sequence, given the context and our familiarity with the language. As we move to words, they naturally follow each other in a logical manner, contingent on the context. For instance, in a discussion about electronics, the likelihood of "effect" following "hall" is much higher than in a discourse about school buildings. These linguistic probabilities become even more pronounced when we construct sentences. One type of sentence tends to follow another, and the arrangement of words within them becomes predictable to some extent, again based on the context and training data. Nevertheless, it's not only about probabilities and prediction. Language models, such as Large Language Models (LLMs), possess a capacity that transcends mere prediction. They can grapple with 'thoughts'—an abstract concept that may not always be apparent but is undeniably a part of their functionality. These 'thoughts' can manifest as encoded 'ideas' or concepts associated with the language they've learned. It may be true that LLMs predict the next "thought" based on the corpus they were trained on, but it's not to say they can generalize this behavior, past what "ideas" they were trained on. I'm not claiming generalized intelligence exists, yet. Much like how individual letters and words combine to create variables and method names in coding, the 'ideas' encoded within LLMs become the building blocks for complex language behavior. These ideas have varying weights and connections, and as a result, they can generate intricate responses. So, while the outcome may sometimes seem random, it's rooted in the very real complex interplay of ideas and their relationships, much like the way methods and variables in code are structured by the 'idea' they represent when laid out in a logical manner. Language is a means to communicate thought, so it's not a huge surprise that words, used correctly, might convey an idea someone else can "process", and that likely includes LLMs. That we get so much useful content from LLMs is a good indication that they are dealing with "ideas" now, not just letters and words. I realize that people are currently struggling with whether or not LLMs can "reason". For as many times as I've thought it was reasoning, I'm sure there are many times it wasn't reasoning well. But, did it ever "reason" at all, or was that simply an illusion, or happy coincidence based on probability? The rub with the word "reasoning" is that it directly involves "being logical" and how we humans arrive at being logical is a bit of a mystery. It's logical to think a cat can't jump higher than a tree, but what if it was a very small tree? The ability to reason about cats jumping abilities doesn't require understanding trees come in different heights, rather that when we refer to "tree" we mean "something tall". So, reasoning has "shortcuts" to arrive at an answer about a thing, without looking at all the things probabilities. For whatever reason, most humans won't argue with you about tree height at that point and just reply "No, cats can't jump higher than a tree, but they can climb it." By adding the latter part, they are not arguing the point, but rather ensuring that someone can't pigeonhole their idea of truth of the matter. Maybe when LLMs get as squirrely as humans in their thinking we'll finally admit they really do "reason".
chrsig 3y ago
Perhaps not useful to the conversation, but I really wish that whomever coined the behavior as a 'hallucination' had consulted a dictionary first. It's delusional, not hallucinated. Delusions are the irrational holdings of false belief, especially after contrary evidence has been provided. Hallucinations are false sensations or perceptions of things that do not exist. May some influential ML person read this and start to correct the vocabulary in the field :)
- giantrobot 3y ago
  I prefer the term confabulation. To the AI the made up thing isn't necessarily irrational. It's in fact very rational, simply incorrect.
  - asadotzler 3y ago
    aka bullshit. it is a bullshitter or a bullshit artist, virtually synonymous with confabulator.
- 65a 3y ago
  Confabulation seems better aligned to neuropsychology, as far as I can tell: https://en.wikipedia.org/wiki/Confabulation https://en.wikipedia.org/wiki/Confabulation
  - garba_dlm 3y ago
    cool, the scientific name for studying gaslighting!
    - groby_b 3y ago
      Gaslighting and confabulation are very different things. Gaslighting are deliberate lies with the intent of creating self-doubt in the targeted person. Confabulation is creating falsehoods without an intent to deceive. When we're discussing naming, it might be a good idea not to throw more misleading names onto the bonfire.
pengaru 3y ago
ChatGPT is the epitome of a useful idiot.
ImAnAmateur 3y ago
There's a second wind to this story in the Mastodon replies. It sounds like the LLM appeared to be basing this output on a CVE that hadn't yet been made public, implying that it had access to text that wasn't public. I can't quite tell if that's an accurate interpretation of what I'm reading. >> @bagder it’s all the weirder because they aren’t even trying to report a new vulnerability. Their complaint seems to be that detailed information about a “vulnerability” is public. But that’s how public disclosure works? And open source? Like are they going to start submitting blog posts of vulnerability analysis and ask curl maintainers to somehow get the posts taken down??? >> @derekheld they reported this before that vulnerability was made public though >> @bagder oh as in saying the embargo was broken but with LLM hallucinations as the evidence? >> @derekheld something like that yes
- jameshart 3y ago
  Took me a while to figure out from the toot thread and comment history, but it appears that the curl 8.4.0 release notes (https://daniel.haxx.se/blog/2023/10/11/curl-8-4-0/ https://daniel.haxx.se/blog/2023/10/11/curl-8-4-0/) referred to the fact that it included a fix for an undisclosed CVE (CVE-2023-38545); the reporter ‘searched in Bard’ for information about that CVE and was given hallucinated details utterly unrelated to the actual curl issue. The reporter is complaining that they thought this constituted a premature leak of a predisclosure CVE, and was reporting this as a security issue to curl via HackerOne.
- Arnavion 3y ago
  No, it's not that Bard was trained on information that wasn't public. It's that the author of the report thought that the information about the upcoming CVE was public somewhere because Bard was reproducing it, because the author thinks Bard is a search engine. So they filed a report that the curl devs should take that information offline until the embargo is lifted.
  - sfink 3y ago
    Which is a fair request. Perhaps Bard should be taken offline. The curl devs might even be the right ones to do it, if they slipped a DDOS into the code...
tklinglol 3y ago
This is confusing - the reporter claims to have "crafted the exploit" using the info they got from Bard. So the hallucinated info was actionable enough to actually perform the/an exploit, even though the report was closed as bogus?
- sunbum 3y ago
  Or lied about crafting an exploit for a potential bug bounty payout
- Khoth 3y ago
  The thing they're they're reporting is that a CVE leaked and Bard found out about it before public disclosure. Except that it's false because Bard made it up. There's no real curl exploit involved.
- openasocket 3y ago
  No, they weren't able to "craft the exploit". The text claims an integer overflow bug in curl_easy_setopt, and provides a code snippet that fixes it. Except the code snippet has a completely different function signature than the real curl_easy_setopt, and doesn't even compile. I doubt this person did any follow through at all, just copy/pasted the output from Bard directly into this bug report.
_pdp_ 3y ago
While somewhat off-topic, I had an interesting experience highlighting the utility of GitHub's Copilot the other day. I decided to run Copilot on a piece of code functioning correctly to see if it would identify any non-existent issues. Surprisingly, it managed to pinpoint an actual bug. Following this discovery, I asked Copilot to generate a unit test to better understand the identified issue. Upon running the test, the program crashed just as Copilot had predicted. I then refactored the problematic lines as per Copilot's suggestions. This was my first time witnessing the effectiveness of Copilot in such a scenario, which provided small yet significant proof to me that Language Models can be invaluable tools for coding, capable of identifying and helping to resolve real bugs. Although they may have limitations, I believe any imperfections are merely temporary hurdles toward more robust coding assistants.
- WanderPanda 3y ago
  Copilot at present capabilities is already so valuable that not having it in some environment gives me the „disabledness feeling“ that I otherwise only get when vim bindings are not enabled. Absolute miracle technology! I‘m sure in the not too distant future we‘ll have have privacy preserving, open source versions that are good enough to not shovel everything over to openai
  - jgalt212 3y ago
    > shovel everything over to openai Seriously, if you're a niche market with specific know-how, the easiest way to broadly propagate this know-now is to use copilot.
- ushakov 3y ago
  That’s rather an exception in my experience. For unit-tests it starts hallucinating hard once you have functions imported from other files. This is probably the reason most unit tests in their marketing materials are things like fibonacci…
- chrisco255 3y ago
  Try it on a million line code base where it's not so cut and dry to even determine if the code is running correctly or what correctly means when it changes day to day.
openasocket 3y ago
I was curious how many bogus security reports big open source projects have. If you go to https://hackerone.com/curl/hacktivity https://hackerone.com/curl/hacktivity and scroll down to ones marked as "Not-applicable" you can find some additional examples. No other LLM hallucinations, but some pretty poorly-thought out "bugs".
bawolff 3y ago
I'm not sure why this is interesting. AI was asked to make a fake vulnerability and it did. That's the sort of thing these AIs are good at, not exactly new at this point.
- geraldcombs 3y ago
  You're leaving out the "...and then they reported it to the project" part, which meant that the project maintainers had to put in time and effort responding to a reported vulnerability.
  - bawolff 3y ago
    As someone who has been on the maintainer side of a bug bounty program - they are a mountain of BS with 1% being diamonds. This report probably didn't make much of a difference.
- mcmoor 3y ago
  For one thing for the last week I've seen several articles about "curl is vulnerable and will be exposed soon!!". For it to turn out this way is certainly a plot twist.
  - bawolff 3y ago
    This is not the way that turned it out. The curl vuln everyone was fretting about was https://curl.se/docs/CVE-2023-38545.html https://curl.se/docs/CVE-2023-38545.html still very much a serious and real vulnerability.
seniorsassycat 3y ago
So the reporter thinks that they were able to get accurate info about private details of a embargo'ed cve from Bard. If correct they would have found a cve in bard, not in curl. In this case the curl maintainers can tell the details are made up and don't correspond to any cve.
- nikanj 3y ago
  Mitre would probably still file a 10.0 CVE based on this report
19h 3y ago
I’m doing reverse engineering work every now and then and a year ago I’d have called myself a fool but I have found multiple exploitable vulnerabilities simply by asking an LLM (Claude refuses less often than GPT4, GPT4 generally got better results when properly phrasing the request). One interesting find is that I wrote an integration with GPT4 for binaryninja and funnily enough when asking the LLM to rewrite a function into “its idiomatic equivalent, refactored and simplified without detail removal” and then asking it to find vulnerabilities, it cracked most of our joke-hack-me’s in a matter of minutes. Interesting learning: nearly all LLMs can’t really properly work with disassembled Rust binaries, I guess that’s because the output doesn’t exactly resemble the rust code like it’d do in C and C++.
- zamalek 3y ago
  The difference is that you'd at least try to compile the alleged exploit before disclosing it. The usefulness of AI is inversely proportional to the laziness of its operator, and such a golden hammer is surefire fly's shit for lazy people. But totally, actual pure gold in responsible hands.

200 points by martijnarts 3y ago

> you did not find anything worthy of reporting. You were fooled by an AI into believing that. The author's right. Reading the report I was stunned; the person disclosing the so-called vulnerability said: > To replicate the issue, I have searched in the Bard about this vulnerability. Does Bard clearly warn to never rely on it for facts? I know OpenAI says "ChatGPT may give you inaccurate information" at the start of each session.

xd1936 3y ago
Oh yeah. Google has warnings like "Bard may display inaccurate or offensive information that doesn’t represent Google’s views" all over it; Permanently in the footer, on splash pages, etc.
- abnercoimbre 3y ago
  Well Google has a branding problem with Bard... because everyone knows Google for search. "Surely Bard must be a reliable engine too."
filterfiber 3y ago
> Does Bard clearly warn to never rely on it for facts? I know OpenAI says "ChatGPT may give you inaccurate information" at the start of each session. I know I shouldn't be, but I'm surprised the disclosure is even needed. People clearly don't understand how LLMs work - LLM's predict text. That's it, they're glorified autocomplete (that's really good). When their prediction is wrong we call it a "hallucination" for some reason. Humans do the same thing all the time. Of course it's not always correct!
- masklinn 3y ago
  > People clearly don't understand how LLMs work - Of course not. Most developers don't understand how LLM work, even roughly. > Humans do the same thing all the time. Of course it's not always correct! The difference is that LLMs can not acknowledge incompetence, are always confidently incorrect, and will never reach a stopping point, at best they'll start going circular.
  - nerpderp82 3y ago
    Everything out of an LLM is a confabulation, but you can constrain the output space of that confab by restraining it with proper prompting. You could ask it to put confidence intervals for each one of its sentences (ask it in the prompt), but those will be confabulated as well, but will give it some self doubt, as for now, it hasn't been programmed with any. Probably costs a lot more in power to run it with doubt. :) Edit, I played around with this. It looks like GPT4 has a guard against asking for this. It flat out refused with the two prompts I gave it to include confidence intervals. Maybe that is a good thing.
  - lazide 3y ago
    At least they don’t control any militaries yet, since the natural outgrowth of that is making reality meet their answers. Or at least it would be if they were human!
  - fomine3 3y ago
    > The difference is that LLMs can not acknowledge incompetence, are always confidently incorrect, and will never reach a stopping point, at best they'll start going circular. Like a smartass?

orf 3y ago

> I responsibly disclosed the information as soon as I found it. I believe there is a better way to communicate to the researchers, and I hope that the curl staff can implement it for future submissions to maintain a better relationship with the researcher community. Thank you! … yeah…

HtmlProgrammer 3y ago
Poor fella was embarrassed and looking to throw anything back at them
hypeatei 3y ago
It looks like AI generated that response.

evrimoztamur 3y ago

If not malicious, then this shows that there are people out there who don't quite know how much to rely on LLMs or understand the limits of their capabilities. It's distressing.

filterfiber 3y ago
The average person sadly just hears the marketed "artificial intelligence" and doesn't grasp that it simply predicts text. It's really good at predicting text we like, but that's all it does. It shouldn't be surprising that sometimes it's prediction is either wrong or unwanted. Interestingly even intelligent, problem solving, educated humans "incorrectly predict" all the time.
- kordlessagain 3y ago
  > It's really good at predicting text we like, but that's all it does. It's important to recognize that predicting text is not merely about guessing the next letter or word, but rather a complex set of probabilities grounded in language and context. When we look at language, we might see intricate relationships between letters, words, and ideas. Starting with individual letters, like 't,' we can assign probabilities to their occurrence based on the language and alphabet we've studied. These probabilities enable us to anticipate the next character in a sequence, given the context and our familiarity with the language. As we move to words, they naturally follow each other in a logical manner, contingent on the context. For instance, in a discussion about electronics, the likelihood of "effect" following "hall" is much higher than in a discourse about school buildings. These linguistic probabilities become even more pronounced when we construct sentences. One type of sentence tends to follow another, and the arrangement of words within them becomes predictable to some extent, again based on the context and training data. Nevertheless, it's not only about probabilities and prediction. Language models, such as Large Language Models (LLMs), possess a capacity that transcends mere prediction. They can grapple with 'thoughts'—an abstract concept that may not always be apparent but is undeniably a part of their functionality. These 'thoughts' can manifest as encoded 'ideas' or concepts associated with the language they've learned. It may be true that LLMs predict the next "thought" based on the corpus they were trained on, but it's not to say they can generalize this behavior, past what "ideas" they were trained on. I'm not claiming generalized intelligence exists, yet. Much like how individual letters and words combine to create variables and method names in coding, the 'ideas' encoded within LLMs become the building blocks for complex language behavior. These ideas have varying weights and connections, and as a result, they can generate intricate responses. So, while the outcome may sometimes seem random, it's rooted in the very real complex interplay of ideas and their relationships, much like the way methods and variables in code are structured by the 'idea' they represent when laid out in a logical manner. Language is a means to communicate thought, so it's not a huge surprise that words, used correctly, might convey an idea someone else can "process", and that likely includes LLMs. That we get so much useful content from LLMs is a good indication that they are dealing with "ideas" now, not just letters and words. I realize that people are currently struggling with whether or not LLMs can "reason". For as many times as I've thought it was reasoning, I'm sure there are many times it wasn't reasoning well. But, did it ever "reason" at all, or was that simply an illusion, or happy coincidence based on probability? The rub with the word "reasoning" is that it directly involves "being logical" and how we humans arrive at being logical is a bit of a mystery. It's logical to think a cat can't jump higher than a tree, but what if it was a very small tree? The ability to reason about cats jumping abilities doesn't require understanding trees come in different heights, rather that when we refer to "tree" we mean "something tall". So, reasoning has "shortcuts" to arrive at an answer about a thing, without looking at all the things probabilities. For whatever reason, most humans won't argue with you about tree height at that point and just reply "No, cats can't jump higher than a tree, but they can climb it." By adding the latter part, they are not arguing the point, but rather ensuring that someone can't pigeonhole their idea of truth of the matter. Maybe when LLMs get as squirrely as humans in their thinking we'll finally admit they really do "reason".

chrsig 3y ago

Perhaps not useful to the conversation, but I really wish that whomever coined the behavior as a 'hallucination' had consulted a dictionary first. It's delusional, not hallucinated. Delusions are the irrational holdings of false belief, especially after contrary evidence has been provided. Hallucinations are false sensations or perceptions of things that do not exist. May some influential ML person read this and start to correct the vocabulary in the field :)

giantrobot 3y ago
I prefer the term confabulation. To the AI the made up thing isn't necessarily irrational. It's in fact very rational, simply incorrect.
- asadotzler 3y ago
  aka bullshit. it is a bullshitter or a bullshit artist, virtually synonymous with confabulator.
65a 3y ago
Confabulation seems better aligned to neuropsychology, as far as I can tell: https://en.wikipedia.org/wiki/Confabulation https://en.wikipedia.org/wiki/Confabulation
- garba_dlm 3y ago
  cool, the scientific name for studying gaslighting!
  - groby_b 3y ago
    Gaslighting and confabulation are very different things. Gaslighting are deliberate lies with the intent of creating self-doubt in the targeted person. Confabulation is creating falsehoods without an intent to deceive. When we're discussing naming, it might be a good idea not to throw more misleading names onto the bonfire.

pengaru 3y ago

ChatGPT is the epitome of a useful idiot.

ImAnAmateur 3y ago

There's a second wind to this story in the Mastodon replies. It sounds like the LLM appeared to be basing this output on a CVE that hadn't yet been made public, implying that it had access to text that wasn't public. I can't quite tell if that's an accurate interpretation of what I'm reading. >> @bagder it’s all the weirder because they aren’t even trying to report a new vulnerability. Their complaint seems to be that detailed information about a “vulnerability” is public. But that’s how public disclosure works? And open source? Like are they going to start submitting blog posts of vulnerability analysis and ask curl maintainers to somehow get the posts taken down??? >> @derekheld they reported this before that vulnerability was made public though >> @bagder oh as in saying the embargo was broken but with LLM hallucinations as the evidence? >> @derekheld something like that yes

jameshart 3y ago
Took me a while to figure out from the toot thread and comment history, but it appears that the curl 8.4.0 release notes (https://daniel.haxx.se/blog/2023/10/11/curl-8-4-0/ https://daniel.haxx.se/blog/2023/10/11/curl-8-4-0/) referred to the fact that it included a fix for an undisclosed CVE (CVE-2023-38545); the reporter ‘searched in Bard’ for information about that CVE and was given hallucinated details utterly unrelated to the actual curl issue. The reporter is complaining that they thought this constituted a premature leak of a predisclosure CVE, and was reporting this as a security issue to curl via HackerOne.
Arnavion 3y ago
No, it's not that Bard was trained on information that wasn't public. It's that the author of the report thought that the information about the upcoming CVE was public somewhere because Bard was reproducing it, because the author thinks Bard is a search engine. So they filed a report that the curl devs should take that information offline until the embargo is lifted.
- sfink 3y ago
  Which is a fair request. Perhaps Bard should be taken offline. The curl devs might even be the right ones to do it, if they slipped a DDOS into the code...

tklinglol 3y ago

This is confusing - the reporter claims to have "crafted the exploit" using the info they got from Bard. So the hallucinated info was actionable enough to actually perform the/an exploit, even though the report was closed as bogus?

sunbum 3y ago
Or lied about crafting an exploit for a potential bug bounty payout
Khoth 3y ago
The thing they're they're reporting is that a CVE leaked and Bard found out about it before public disclosure. Except that it's false because Bard made it up. There's no real curl exploit involved.
openasocket 3y ago
No, they weren't able to "craft the exploit". The text claims an integer overflow bug in curl_easy_setopt, and provides a code snippet that fixes it. Except the code snippet has a completely different function signature than the real curl_easy_setopt, and doesn't even compile. I doubt this person did any follow through at all, just copy/pasted the output from Bard directly into this bug report.

_pdp_ 3y ago

While somewhat off-topic, I had an interesting experience highlighting the utility of GitHub's Copilot the other day. I decided to run Copilot on a piece of code functioning correctly to see if it would identify any non-existent issues. Surprisingly, it managed to pinpoint an actual bug. Following this discovery, I asked Copilot to generate a unit test to better understand the identified issue. Upon running the test, the program crashed just as Copilot had predicted. I then refactored the problematic lines as per Copilot's suggestions. This was my first time witnessing the effectiveness of Copilot in such a scenario, which provided small yet significant proof to me that Language Models can be invaluable tools for coding, capable of identifying and helping to resolve real bugs. Although they may have limitations, I believe any imperfections are merely temporary hurdles toward more robust coding assistants.

WanderPanda 3y ago
Copilot at present capabilities is already so valuable that not having it in some environment gives me the „disabledness feeling“ that I otherwise only get when vim bindings are not enabled. Absolute miracle technology! I‘m sure in the not too distant future we‘ll have have privacy preserving, open source versions that are good enough to not shovel everything over to openai
- jgalt212 3y ago
  > shovel everything over to openai Seriously, if you're a niche market with specific know-how, the easiest way to broadly propagate this know-now is to use copilot.
ushakov 3y ago
That’s rather an exception in my experience. For unit-tests it starts hallucinating hard once you have functions imported from other files. This is probably the reason most unit tests in their marketing materials are things like fibonacci…
chrisco255 3y ago
Try it on a million line code base where it's not so cut and dry to even determine if the code is running correctly or what correctly means when it changes day to day.

openasocket 3y ago

I was curious how many bogus security reports big open source projects have. If you go to https://hackerone.com/curl/hacktivity https://hackerone.com/curl/hacktivity and scroll down to ones marked as "Not-applicable" you can find some additional examples. No other LLM hallucinations, but some pretty poorly-thought out "bugs".

bawolff 3y ago

I'm not sure why this is interesting. AI was asked to make a fake vulnerability and it did. That's the sort of thing these AIs are good at, not exactly new at this point.

geraldcombs 3y ago
You're leaving out the "...and then they reported it to the project" part, which meant that the project maintainers had to put in time and effort responding to a reported vulnerability.
- bawolff 3y ago
  As someone who has been on the maintainer side of a bug bounty program - they are a mountain of BS with 1% being diamonds. This report probably didn't make much of a difference.
mcmoor 3y ago
For one thing for the last week I've seen several articles about "curl is vulnerable and will be exposed soon!!". For it to turn out this way is certainly a plot twist.
- bawolff 3y ago
  This is not the way that turned it out. The curl vuln everyone was fretting about was https://curl.se/docs/CVE-2023-38545.html https://curl.se/docs/CVE-2023-38545.html still very much a serious and real vulnerability.

seniorsassycat 3y ago

So the reporter thinks that they were able to get accurate info about private details of a embargo'ed cve from Bard. If correct they would have found a cve in bard, not in curl. In this case the curl maintainers can tell the details are made up and don't correspond to any cve.

nikanj 3y ago
Mitre would probably still file a 10.0 CVE based on this report

19h 3y ago

I’m doing reverse engineering work every now and then and a year ago I’d have called myself a fool but I have found multiple exploitable vulnerabilities simply by asking an LLM (Claude refuses less often than GPT4, GPT4 generally got better results when properly phrasing the request). One interesting find is that I wrote an integration with GPT4 for binaryninja and funnily enough when asking the LLM to rewrite a function into “its idiomatic equivalent, refactored and simplified without detail removal” and then asking it to find vulnerabilities, it cracked most of our joke-hack-me’s in a matter of minutes. Interesting learning: nearly all LLMs can’t really properly work with disassembled Rust binaries, I guess that’s because the output doesn’t exactly resemble the rust code like it’d do in C and C++.

zamalek 3y ago
The difference is that you'd at least try to compile the alleged exploit before disclosing it. The usefulness of AI is inversely proportional to the laziness of its operator, and such a golden hammer is surefire fly's shit for lazy people. But totally, actual pure gold in responsible hands.