7 ms·

AI hallucinates software packages and devs download them

62 points by dragonbonheur 3y ago

Shrezzing 3y ago
In the original Llama paper, the process of preparing the corpus was described. For the sourcecode aspect, it was fed into their model's corpus after being cleaned of boilerplate code. I think it'd be a fair assumption that most of the other vendors followed this practice when creating their datasets. I'm not a python user, but in most languages, libraries are referenced in (what most devs would consider) boilerplate code. Purely conjecture, but perhaps without boilerplate code, the LLMs are left guessing the names of popular libraries and just merges together two common naming conventions "huggingface" and "-cli".
- 1oooqooq 3y ago
  my guess is that some layer optimized for the package name being the repo. so the researchers just published a package that matched the repo name. ai is a bad non-losless search engine.
  - sa-code 3y ago
    I think the word you're looking for is lossy
    - 1oooqooq 3y ago
      shhh. don't make it easier for the bots.
- londons_explore 3y ago
  I think the issue here is that most commandline tools of name X are in an apt/pip package named X. For huggingface, the tool is called huggingface-cli, but the package is called huggingface_hub[cli]. IMO, thats bad naming. If you make a tool called X, just publish it in a package called X.
  - maxcoder4 3y ago
    In many cases yes, but: What if you publish multiple tools in a single package? What if you publish a package for a web service called X, but also include a tool for it called xcli? What if your package is primarily a library, but you also include an optional cli (like in this case, probably)? I think what's worse is publishing a package under a different name than its root namespace.
    - londons_explore 3y ago
      > What if you publish multiple tools in a single package? I question if one should ever do that... Packages are free... Just make one package per binary, and then a metapackage for those who want to install the whole suite.
- MrSkyNet 3y ago
  Boilerplate code is code you don't need to write or is cumbersome to write. Like getter/setter in java for all your attributes. I would never consider imports boilerplate code.
  - namaria 3y ago
    If some bits of meaningful (containing essential complexity) code imply other bits with mostly accidental complexity, writing them by hand is a waste of time. But isn't it also a waste to use data centers full of GPUs to process terabytes of text to accomplish the same thing better programming language design could?
    - MrSkyNet 3y ago
      We spend that much GPU compute not to just generate random boiler plate code but to create the real code. If it wouldn't be beneficial for writing code, which it is, we wouldn't use it. But yes if we could create better languages or systems, it would be a waste. But we tried multiply new programming languages, we have no code platforms etc. It does look like though that LLM is still better than all of those approaches.
      - namaria 3y ago
        I have to write very little boilerplate code as it is with the tooling I choose. And a lot of it is generated by scripts using some input from me. I don't need cloud GPUs to write code at all.
        MrSkyNet 3y ago
        I don't need it either. But thats not the point of it anyway? Its about writing code faster and potentially better. Cloud GPUs can also generate unit tests etc. I primarily use it for languages i don't use often enough, nonetheless its only a question of time until it doesn't make sense anymore to write code yourself.
valenterry 3y ago
I'm just waiting for AI to suggest to download package X for usage in language A, which exists in language B but not A, and some malicious actor creating X in A to spread their malware.
- Moru 3y ago
  I keep having this happen to a very small language I used to play around with. Now when I have forgotten how it works I thought I would try to learn it again with ChatGPT and the results were pretty disappointing. It keeps using a mix of different languages, mostly C++ and Java it seems. And I hunt thrugh the documentation to find the real way to do it. It corrects itself but it's still wrong and it keeps going in circles. I guess I won't be learning that language again with help from the AI. It's just too small of a subject for the AI.
  - dingnuts 3y ago
    This is a demonstrative example of why an LLM is NOT intelligent at all, and we need to stop calling these models AI. A real intelligence would be able to either put together the information needed and fill in gaps with research or it would be able to tell you that it does not know. "Hallucination" is yet another anthropomorphism of the model. A mind hallucinates; a model just produces incorrect results. LLMs are very cool but I'm really sick of the "intelligence" branding. If a person who didn't know about something just confidently made shit up, I wouldn't call them intelligent.
    - pfortuny 3y ago
      I’ve just tries GPT (again, after months) on high level pseudomathematics and it blabbers on and on about nonsense…
      - cool_dude85 3y ago
        I dont know about the newer models but you used to be able to get authoritative sounding answers asking about "the fallacy of affirming the antecedent". So you don't need to be so high level.
botanical 3y ago
Hallucinations as a term for incorrect answers made by token generators is a genius marketing term. In fact, labelling LLMs as AI is what propelled high valuations of these companies. This is what leads to people even in the industry to take at face value a human-like response in helping answer a query; and in this case, willingly downloading malware.
- usrusr 3y ago
  Genius marketing for what? It's a very fitting term, because between the lines it tells all who don't refuse to listen that the correct answers are no different, they just happen to be not wrong. Dishonest marketing would be calling them bugs, because that's what they are not.
  - ben_w 3y ago
    I think "bug" can also be a legitimate description in some cases. If I imagine a very naive translation system that just does a dictionary substitution, and the dictionary has some bad entries, the correct and incorrect results look the same, but it's still a bug. The system as a whole still messes up because "dictionary substitution" is not at all sufficient: Hydraulic ram -> aqua ovis -> water sheep … this probably counts as just "hallucination" while arguably not being a "bug", because the dictionary is correct (or close enough, I don't speak Latin), but the system as a whole can't ever be better because the model it relies on can't represent the right things.
    - usrusr 3y ago
      Dictionary substitution is so much closer to symbolic AI than to what LLM are that I don't think it can serve as an example for the "hallucination" wording. In an expert system, if a rule is wrong or inadequate it's clearly a bug. Just not a bug in the rule engine but in the rules. But in an LLM, there are no explicit rules (or only in some obscure background layers), it's all statistics. Statistics stacked on top of more statistics in a fascinating self-stabilizing way, where each guess provides some support to its peers, like how the weak paper slabs support each other in a house of cards. But it's statistical guesswork all the way down. Even the most correct answers are merely statistics playing out favorably in a case that may or may not have been very easy to get right.
vouaobrasil 3y ago
People will say this happened before with people purposefully doing this, but it will happen at a much faster rate with much less oversight with AI. Moreover, I can't help but think of the comments both Garry Kasparov and Go players made when they were playing against AI: they thought it had very counterintuitive and confusing moves compared to human play. The truth is, while both humans and AI can make errors, and both can be malicious as well, the actions of AI will be counterintuitive and confusing and we won't know how to counter them in the same way that we counter human follies. This is just one aspect that shows that AI is making society worse on average and it should be destroyed.
- ben_w 3y ago
  "Malice" is, at present, as much of an anthropomorphisation as calling it "good"; you could ask it to play a malicious role, and it will do so, but it is not so by itself. The output of AI can often be confusing, yes, and indeed this is part of the issue Yudkowsky has with AI in general: we can't predict it in detail, especially not in domains where it out-performs us (like Chess). However, this is tangential to any question about "making society worse" — even when it's making society better (because it's more capable than we are at something which we care about), Yudkowsky would caution that this may be a 5D chess move to gain more power.
jgalt212 3y ago
It used to be garbage in, garbage out (GIGO). But now, sometimes you put valid data in and get garbage out. I just can't go all-in on LLMs with the error / hallucinate rate where it currently is. And people say it's getting better. But I guess I'll just do things the slower, and more accurate way until such time arrives.
- yau8edq12i 3y ago
  "Garbage in, garbage out" means that if you feed garbage to your algorithm, then you're going to get garbage in return. It's always been a one-way implication. It doesn't mean that if you feed valid data, you're not going to get garbage.
  - andersa 3y ago
    The fun part is that with LLMs we can now sometimes put absolute garbage in and still get good results.
cageface 3y ago
Some of us here may be old enough to remember the days when Microsoft was pushing ActiveX controls as the future of the web. Basically they were binary components that would run in your browser and weren't really effectively sandboxed. Those of us that pointed out this was a major security hole were ignored. Inevitably carnage followed. People blindly accepting the output of LLMs seems similarly crazy to me and I think it's only a matter of time before we face a real reckoning over this. The lesson here I think is just because a lot of people are advocating something that seems reckless doesn't mean it isn't reckless. https://www.howtogeek.com/162282/what-activex-controls-are-and-why-theyre-dangerous/ https://www.howtogeek.com/162282/what-activex-controls-are-a...
- gruez 3y ago
  >Some of us here may be old enough to remember the days when Microsoft was pushing ActiveX controls as the future of the web. Basically they were binary components that would run in your browser and weren't really effectively sandboxed. That's basically what's happening every time a developer runs "npm install" or "pip install". Sure, it technically not be a "binary", but it makes no practical difference given that no code scrutiny is given the majority of the time. The introduction of LLMs doesn't change much. People are already predisposed to copy paste random commands from blogs/forums. LLM is just one more source on top of that.
  - scruple 3y ago
    > it makes no practical difference given that no code scrutiny is given the majority of the time. I was thinking about this the other day. I wonder what my leadership would say if I told them I spent the day scrutinizing some of our open source dependencies. I assume even a day would be treated as wasted time, especially on the product side. FWIW, I used to do this back in the early Rails day and was encouraged to do so. I ended up contributing heavily to the Rails ecosystem because of it, and it was all encouraged by my employer at the time, but they were a relatively small startup at the time and viewed things very differently than the FAANG I work for today.
- ben_w 3y ago
Y_Y 3y ago
A nice thing to do might be to put "Mountweazel" packages under names like this, i.e. a package that always fails to install and gives an error explaining you've been duped by LLMinati. There is already something like this for e.g. `nvidia-tlt` which exists on PyPI, but just as a placeholder telling you to go and add Nvidia's pip repository.
mrdevlar 3y ago
An LLM gets things wrong. It's no different than interacting with a person. You will get a confident answer. The difference is you can easily get a second opinion of an LLM without offending it. Hallucinations are unlikely to occur twice in a row. So just ask twice.
- maxcoder4 3y ago
  That's not true. For every non-trivial[1] question I've asked chatgpt hallucinated and then kept hallucinating no matter how many times I've asked, clarified, or gave it hints. [1] i.e. specific to my niche where i didn't know the answer immediately.
- dogleash 3y ago
  >Hallucinations are unlikely to occur twice in a row. So just ask twice. lol If it's not already easily googleable, the LLM will just spit out some boilerplate apology and then double down on the previous wrong answers, provide something correct about a tangentially related topic, or just reword my prompt as a course of investigation that - if written by a person - would just be a polite "fuck off and go figure it out yourself."
  - ben_w 3y ago
    I get that sometimes, but not all the time. ChatGPT seems to be able to fix, I'm going to say "about half" of it's own mistakes, but that's my gut feeling and not a detailed analysis even within the domain of questions I've been asking (which itself probably isn't representative).
nraynaud 3y ago
copilot hallucinates functions and packages versions all the time, it's a nightmare, I never know if I got a suggestion from a predictible IDE algorithm or the drunk overeager intern, and it's really annoying. Being on your toes means you have no mental flow, because the pothead intern is unpredictible in its suggestions.
skeledrew 3y ago
I get the malware-infection potential, but it's still blows my mind that the fake package wasn't caught earlier by the actual users. All those downloads and nobody wondered why X feature that depends on it wasn't working? Whatever happened to testing things and having decent coverage?

62 points by dragonbonheur 3y ago

In the original Llama paper, the process of preparing the corpus was described. For the sourcecode aspect, it was fed into their model's corpus after being cleaned of boilerplate code. I think it'd be a fair assumption that most of the other vendors followed this practice when creating their datasets. I'm not a python user, but in most languages, libraries are referenced in (what most devs would consider) boilerplate code. Purely conjecture, but perhaps without boilerplate code, the LLMs are left guessing the names of popular libraries and just merges together two common naming conventions "huggingface" and "-cli".

1oooqooq 3y ago
my guess is that some layer optimized for the package name being the repo. so the researchers just published a package that matched the repo name. ai is a bad non-losless search engine.
- sa-code 3y ago
  I think the word you're looking for is lossy
  - 1oooqooq 3y ago
    shhh. don't make it easier for the bots.
londons_explore 3y ago
I think the issue here is that most commandline tools of name X are in an apt/pip package named X. For huggingface, the tool is called huggingface-cli, but the package is called huggingface_hub[cli]. IMO, thats bad naming. If you make a tool called X, just publish it in a package called X.
- maxcoder4 3y ago
  In many cases yes, but: What if you publish multiple tools in a single package? What if you publish a package for a web service called X, but also include a tool for it called xcli? What if your package is primarily a library, but you also include an optional cli (like in this case, probably)? I think what's worse is publishing a package under a different name than its root namespace.
  - londons_explore 3y ago
    > What if you publish multiple tools in a single package? I question if one should ever do that... Packages are free... Just make one package per binary, and then a metapackage for those who want to install the whole suite.
MrSkyNet 3y ago
Boilerplate code is code you don't need to write or is cumbersome to write. Like getter/setter in java for all your attributes. I would never consider imports boilerplate code.
- namaria 3y ago
  If some bits of meaningful (containing essential complexity) code imply other bits with mostly accidental complexity, writing them by hand is a waste of time. But isn't it also a waste to use data centers full of GPUs to process terabytes of text to accomplish the same thing better programming language design could?
  - MrSkyNet 3y ago
    We spend that much GPU compute not to just generate random boiler plate code but to create the real code. If it wouldn't be beneficial for writing code, which it is, we wouldn't use it. But yes if we could create better languages or systems, it would be a waste. But we tried multiply new programming languages, we have no code platforms etc. It does look like though that LLM is still better than all of those approaches.
    - namaria 3y ago
      I have to write very little boilerplate code as it is with the tooling I choose. And a lot of it is generated by scripts using some input from me. I don't need cloud GPUs to write code at all.
      - MrSkyNet 3y ago
        I don't need it either. But thats not the point of it anyway? Its about writing code faster and potentially better. Cloud GPUs can also generate unit tests etc. I primarily use it for languages i don't use often enough, nonetheless its only a question of time until it doesn't make sense anymore to write code yourself.

valenterry 3y ago

I'm just waiting for AI to suggest to download package X for usage in language A, which exists in language B but not A, and some malicious actor creating X in A to spread their malware.

Moru 3y ago
I keep having this happen to a very small language I used to play around with. Now when I have forgotten how it works I thought I would try to learn it again with ChatGPT and the results were pretty disappointing. It keeps using a mix of different languages, mostly C++ and Java it seems. And I hunt thrugh the documentation to find the real way to do it. It corrects itself but it's still wrong and it keeps going in circles. I guess I won't be learning that language again with help from the AI. It's just too small of a subject for the AI.
- dingnuts 3y ago
  This is a demonstrative example of why an LLM is NOT intelligent at all, and we need to stop calling these models AI. A real intelligence would be able to either put together the information needed and fill in gaps with research or it would be able to tell you that it does not know. "Hallucination" is yet another anthropomorphism of the model. A mind hallucinates; a model just produces incorrect results. LLMs are very cool but I'm really sick of the "intelligence" branding. If a person who didn't know about something just confidently made shit up, I wouldn't call them intelligent.
  - pfortuny 3y ago
    I’ve just tries GPT (again, after months) on high level pseudomathematics and it blabbers on and on about nonsense…
    - cool_dude85 3y ago
      I dont know about the newer models but you used to be able to get authoritative sounding answers asking about "the fallacy of affirming the antecedent". So you don't need to be so high level.

botanical 3y ago

Hallucinations as a term for incorrect answers made by token generators is a genius marketing term. In fact, labelling LLMs as AI is what propelled high valuations of these companies. This is what leads to people even in the industry to take at face value a human-like response in helping answer a query; and in this case, willingly downloading malware.

usrusr 3y ago
Genius marketing for what? It's a very fitting term, because between the lines it tells all who don't refuse to listen that the correct answers are no different, they just happen to be not wrong. Dishonest marketing would be calling them bugs, because that's what they are not.
- ben_w 3y ago
  I think "bug" can also be a legitimate description in some cases. If I imagine a very naive translation system that just does a dictionary substitution, and the dictionary has some bad entries, the correct and incorrect results look the same, but it's still a bug. The system as a whole still messes up because "dictionary substitution" is not at all sufficient: Hydraulic ram -> aqua ovis -> water sheep … this probably counts as just "hallucination" while arguably not being a "bug", because the dictionary is correct (or close enough, I don't speak Latin), but the system as a whole can't ever be better because the model it relies on can't represent the right things.
  - usrusr 3y ago
    Dictionary substitution is so much closer to symbolic AI than to what LLM are that I don't think it can serve as an example for the "hallucination" wording. In an expert system, if a rule is wrong or inadequate it's clearly a bug. Just not a bug in the rule engine but in the rules. But in an LLM, there are no explicit rules (or only in some obscure background layers), it's all statistics. Statistics stacked on top of more statistics in a fascinating self-stabilizing way, where each guess provides some support to its peers, like how the weak paper slabs support each other in a house of cards. But it's statistical guesswork all the way down. Even the most correct answers are merely statistics playing out favorably in a case that may or may not have been very easy to get right.

vouaobrasil 3y ago

People will say this happened before with people purposefully doing this, but it will happen at a much faster rate with much less oversight with AI. Moreover, I can't help but think of the comments both Garry Kasparov and Go players made when they were playing against AI: they thought it had very counterintuitive and confusing moves compared to human play. The truth is, while both humans and AI can make errors, and both can be malicious as well, the actions of AI will be counterintuitive and confusing and we won't know how to counter them in the same way that we counter human follies. This is just one aspect that shows that AI is making society worse on average and it should be destroyed.

ben_w 3y ago
"Malice" is, at present, as much of an anthropomorphisation as calling it "good"; you could ask it to play a malicious role, and it will do so, but it is not so by itself. The output of AI can often be confusing, yes, and indeed this is part of the issue Yudkowsky has with AI in general: we can't predict it in detail, especially not in domains where it out-performs us (like Chess). However, this is tangential to any question about "making society worse" — even when it's making society better (because it's more capable than we are at something which we care about), Yudkowsky would caution that this may be a 5D chess move to gain more power.

jgalt212 3y ago

It used to be garbage in, garbage out (GIGO). But now, sometimes you put valid data in and get garbage out. I just can't go all-in on LLMs with the error / hallucinate rate where it currently is. And people say it's getting better. But I guess I'll just do things the slower, and more accurate way until such time arrives.

yau8edq12i 3y ago
"Garbage in, garbage out" means that if you feed garbage to your algorithm, then you're going to get garbage in return. It's always been a one-way implication. It doesn't mean that if you feed valid data, you're not going to get garbage.
- andersa 3y ago
  The fun part is that with LLMs we can now sometimes put absolute garbage in and still get good results.

cageface 3y ago

Some of us here may be old enough to remember the days when Microsoft was pushing ActiveX controls as the future of the web. Basically they were binary components that would run in your browser and weren't really effectively sandboxed. Those of us that pointed out this was a major security hole were ignored. Inevitably carnage followed. People blindly accepting the output of LLMs seems similarly crazy to me and I think it's only a matter of time before we face a real reckoning over this. The lesson here I think is just because a lot of people are advocating something that seems reckless doesn't mean it isn't reckless. https://www.howtogeek.com/162282/what-activex-controls-are-and-why-theyre-dangerous/ https://www.howtogeek.com/162282/what-activex-controls-are-a...

gruez 3y ago
>Some of us here may be old enough to remember the days when Microsoft was pushing ActiveX controls as the future of the web. Basically they were binary components that would run in your browser and weren't really effectively sandboxed. That's basically what's happening every time a developer runs "npm install" or "pip install". Sure, it technically not be a "binary", but it makes no practical difference given that no code scrutiny is given the majority of the time. The introduction of LLMs doesn't change much. People are already predisposed to copy paste random commands from blogs/forums. LLM is just one more source on top of that.
- scruple 3y ago
  > it makes no practical difference given that no code scrutiny is given the majority of the time. I was thinking about this the other day. I wonder what my leadership would say if I told them I spent the day scrutinizing some of our open source dependencies. I assume even a day would be treated as wasted time, especially on the product side. FWIW, I used to do this back in the early Rails day and was encouraged to do so. I ended up contributing heavily to the Rails ecosystem because of it, and it was all encouraged by my employer at the time, but they were a relatively small startup at the time and viewed things very differently than the FAANG I work for today.
ben_w 3y ago

Y_Y 3y ago

A nice thing to do might be to put "Mountweazel" packages under names like this, i.e. a package that always fails to install and gives an error explaining you've been duped by LLMinati. There is already something like this for e.g. `nvidia-tlt` which exists on PyPI, but just as a placeholder telling you to go and add Nvidia's pip repository.

mrdevlar 3y ago

An LLM gets things wrong. It's no different than interacting with a person. You will get a confident answer. The difference is you can easily get a second opinion of an LLM without offending it. Hallucinations are unlikely to occur twice in a row. So just ask twice.

maxcoder4 3y ago
That's not true. For every non-trivial[1] question I've asked chatgpt hallucinated and then kept hallucinating no matter how many times I've asked, clarified, or gave it hints. [1] i.e. specific to my niche where i didn't know the answer immediately.
dogleash 3y ago
>Hallucinations are unlikely to occur twice in a row. So just ask twice. lol If it's not already easily googleable, the LLM will just spit out some boilerplate apology and then double down on the previous wrong answers, provide something correct about a tangentially related topic, or just reword my prompt as a course of investigation that - if written by a person - would just be a polite "fuck off and go figure it out yourself."
- ben_w 3y ago
  I get that sometimes, but not all the time. ChatGPT seems to be able to fix, I'm going to say "about half" of it's own mistakes, but that's my gut feeling and not a detailed analysis even within the domain of questions I've been asking (which itself probably isn't representative).

nraynaud 3y ago

copilot hallucinates functions and packages versions all the time, it's a nightmare, I never know if I got a suggestion from a predictible IDE algorithm or the drunk overeager intern, and it's really annoying. Being on your toes means you have no mental flow, because the pothead intern is unpredictible in its suggestions.

skeledrew 3y ago

I get the malware-infection potential, but it's still blows my mind that the fake package wasn't caught earlier by the actual users. All those downloads and nobody wondered why X feature that depends on it wasn't working? Whatever happened to testing things and having decent coverage?