5 ms·

Can you trust ChatGPT’s package recommendations?

29 points by DantesKite 3y ago

mnd999 3y ago
No, you can’t trust ChatGPT for anything. Check, check and check again. Don’t want to be like that lawyer citing made up cases.
- masklinn 3y ago
  It’s ok you can then ask GPT to write up the cases^H^H^H^H^H packages.
  - aziaziazi 3y ago
    And the packages’s dependencies.
puttycat 3y ago
"No". [1] [1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
matkoniecz 3y ago
Pop ups are annoying and for me sufficient to close tab and treat website as spammers. In this case popup spammed me to read article that I was reading before popup appeared.
ziml77 3y ago
WTF is a recommendation in the context of a language model? It does not have preferences, just weights influenced by proximity and frequency of tokens.
- datenyan 3y ago
  To be fair, it's realistically not too far off how most humans make "reccomendations" for packages. Still doesn't mean that it should be trusted, though.
  - veidr 3y ago
    That is indeed the funniest (and slightly horrifying) thing. Even if you understand "glitch tokens"[1] (which is BTW the best way I've seen to explain to nontechnicals (hi, Dad!!) and also some subset of technicals (hi various friends of mine!! +_+...) that LLM applied statistics models aren't anything like sentient AGI)... ...regardless, ChatGPT and its ilk nevertheless do often already outperform "random human being coworker or uncle" at recommending just about anything... which is like Ted Chiang said[2], “That’s more a statement about how much bullshit we are required to generate and deal with in our daily lives.” Although he was not referring to inputs from your (dumbass) coworkers/uncles, it still holds true. [1]: https://youtu.be/WO2X3oZEJOA https://youtu.be/WO2X3oZEJOA [2]: https://archive.is/2023.06.04-050658/https://www.ft.com/content/c1f6d948-3dde-405f-924c-09cc0dcf8c84 https://archive.is/2023.06.04-050658/https://www.ft.com/cont...
- veidr 3y ago
  You are of course right, but ChatGPT (and most others I presume) do recommend things in conversational English. For instance I asked ChatGPT this just now: --- This guy on Hacker News just said, "WTF is a recommendation in the context of a language model? It does not have preferences, just weights influenced by proximity and frequency of tokens." I mean, that is correct you don't have preferences. But you do recommend things based on those models, in conversational English. Do you have any suggestions for how I can convince him that even though is is correct about how language models work, the utility of these models is largely derived from how they recommend solutions in conversational English? --- And ChatGPT gave me some recommendations[1]: --- When discussing language models and their recommendations, it's important to clarify the terminology and concepts involved. While it is true that language models don't have preferences ... they can still provide recommendations based on their training data and patterns they have learned. These recommendations are derived from the statistical analysis of... To convince the person on Hacker News, you can explain the following points: Language models learn from vast amounts of text data: Language models are trained on massive blah blah blah plausible-sounding sentences... Proximity and frequency influence recommendations: Language models assign weights e blah blah blah plausible-sounding sentences... weights influence the model's recommendations by making it more likely to generate familiar or commonly used phrases blah blah blah plausible-sounding sentences... --- [1]: https://chat.openai.com/share/a7d9e2ca-0104-469e-a6e4-6b08a5abe5aa https://chat.openai.com/share/a7d9e2ca-0104-469e-a6e4-6b08a5...
- rad_gruchalski 3y ago
  Recommendations are what I use ChatGPT for. “You are X doing Y, you know A, B, C. What 20 other things should you know. Recommend reading material.” Of course those prompts are usually bit longer.
veidr 3y ago
No, that's the only thing to understand about it! You can't trust anything it says about anything. What it does is emit plausible-sounding information. But we've seen -- I think across every single realm of anything -- that this information includes complete nonsense, whether due to it being trained on people just shitposting bullshit, or due to combining tokens (words, sentence fragments, whatever) in new ways without the benefit of any actual understanding of it. It can be useful, but it's at best as useful as a smart dog that's figured out how to operate a voice synthesizer, and is also on amphetamines or hallucinogens. I'm glad to see this term "LLM hallucination", as that's how it's felt to me. I find ChatGPT really useful for things that I can double-check instantaneously (or nearly so), such as doc comments for the code I just wrote, or small unit tests that match the comment I just wrote. But in my experience, it's worse than useless for writing real code, or any complex endeavor that isn't instantly verifiable/rejectable at a glance. Because vetting plausible looking code -- including dependency specification -- is almost always more taxing than just writing the code or package.json entries yourself.
- visarga 3y ago
  > No, that's the only thing to understand about it! You can't trust anything it says about anything. What it does is emit plausible-sounding information. But we've seen -- I think across every single realm of anything -- that this information includes complete nonsense And that is in my opinion the best argument why we won't be "replaced" by AI at work. This is consistent among all the tasks, NLP, vision, RL - they are all stumbling very quickly without assistance. How do you go from 0 to 1? AI will make for a nice assistant, really nice actually, but not a human replacement. Of course in the far future there will eventually come a day when AI might work without hand holding on some tasks, but we have no idea when. 15 years of SDC development attest to it, the closer you get to the goal, the most hidden hurdles you discover.
  - taneq 3y ago
    You’re missing the reality of how automation “replaces” people. It doesn’t do one person’s job 100%, it does the easiest 90% of everyone’s job. You won’t be replaced by an AI, you and 8 other people will be replaced by an AI plus one guy who has good prompt-fu and editing/fact checking skills.
NikkiA 3y ago
Every time I ask it (3.5) to produce C code it assumes clang's blocks are present, even if I ask for C99 compliant code... blocks, blocks everywhere.
ulrischa 3y ago
No
taneq 3y ago
Treat its output like you’d treat an anonymous forum post. It can be a good source of pointers or ideas but don’t use it as ground truth for anything ever.
ryukoposting 3y ago
Does ChatGPT have a hard-wired incentive structure that encourages good-faith answers? No. Can you trust its package recommendations? No.
JestUM 3y ago
Sooner or later, this can accidentally slip into important pieces of code. Attackers could upload malicious modules that don't exist yet due these hallucinations.
kasrkin623 3y ago
But how do you make sure that it provides consistent hallucinations? Between GPT being consistently patched and such output being prone to statistical error I'm not sure if this vector of attack is efficient in the first place
- johnisgood 3y ago
  I really hate the term "hallucinations". By which definition of hallucination is it a hallucination, really?
  - SirMaster 3y ago
    Well, often when I ask ChatGPT how to code something, it uses some external package that turns out doesn't exist. I don't know what else you would call that. Hallucination seems appropriate enough to me.
    - johnisgood 3y ago
      Okay, but if you ask me the same thing and I use a library or tell you the name of a library that does not exist (although the name comes close), would you say I am hallucinating? If not, then how could we apply this to AI if we cannot even apply the term used for humans to me?
      - SirMaster 3y ago
        OK, personally If a friend recommended me a library that didn't exist I would reply back to him "you made that up". So, ChatGPT is "making stuff up". To me hallucinating is a really similar term for "making something up". So the term works perfectly well for me. I mean the definition on Google at least is: "an experience involving the apparent perception of something not present." Isn't that what's happening? ChatGPT is experiencing (writing code using) the perception of something (the library) that is not present.
        deleted 3y ago
johnisgood 3y ago
Probably not. It is outdated, and sometimes it gets the name of the libraries wrong, too.

29 points by DantesKite 3y ago

No, you can’t trust ChatGPT for anything. Check, check and check again. Don’t want to be like that lawyer citing made up cases.

masklinn 3y ago
It’s ok you can then ask GPT to write up the cases^H^H^H^H^H packages.
- aziaziazi 3y ago
  And the packages’s dependencies.

puttycat 3y ago

"No". [1] [1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

matkoniecz 3y ago

Pop ups are annoying and for me sufficient to close tab and treat website as spammers. In this case popup spammed me to read article that I was reading before popup appeared.

ziml77 3y ago

WTF is a recommendation in the context of a language model? It does not have preferences, just weights influenced by proximity and frequency of tokens.

datenyan 3y ago
To be fair, it's realistically not too far off how most humans make "reccomendations" for packages. Still doesn't mean that it should be trusted, though.
- veidr 3y ago
  That is indeed the funniest (and slightly horrifying) thing. Even if you understand "glitch tokens"[1] (which is BTW the best way I've seen to explain to nontechnicals (hi, Dad!!) and also some subset of technicals (hi various friends of mine!! +_+...) that LLM applied statistics models aren't anything like sentient AGI)... ...regardless, ChatGPT and its ilk nevertheless do often already outperform "random human being coworker or uncle" at recommending just about anything... which is like Ted Chiang said[2], “That’s more a statement about how much bullshit we are required to generate and deal with in our daily lives.” Although he was not referring to inputs from your (dumbass) coworkers/uncles, it still holds true. [1]: https://youtu.be/WO2X3oZEJOA https://youtu.be/WO2X3oZEJOA [2]: https://archive.is/2023.06.04-050658/https://www.ft.com/content/c1f6d948-3dde-405f-924c-09cc0dcf8c84 https://archive.is/2023.06.04-050658/https://www.ft.com/cont...
veidr 3y ago
You are of course right, but ChatGPT (and most others I presume) do recommend things in conversational English. For instance I asked ChatGPT this just now: --- This guy on Hacker News just said, "WTF is a recommendation in the context of a language model? It does not have preferences, just weights influenced by proximity and frequency of tokens." I mean, that is correct you don't have preferences. But you do recommend things based on those models, in conversational English. Do you have any suggestions for how I can convince him that even though is is correct about how language models work, the utility of these models is largely derived from how they recommend solutions in conversational English? --- And ChatGPT gave me some recommendations[1]: --- When discussing language models and their recommendations, it's important to clarify the terminology and concepts involved. While it is true that language models don't have preferences ... they can still provide recommendations based on their training data and patterns they have learned. These recommendations are derived from the statistical analysis of... To convince the person on Hacker News, you can explain the following points: Language models learn from vast amounts of text data: Language models are trained on massive blah blah blah plausible-sounding sentences... Proximity and frequency influence recommendations: Language models assign weights e blah blah blah plausible-sounding sentences... weights influence the model's recommendations by making it more likely to generate familiar or commonly used phrases blah blah blah plausible-sounding sentences... --- [1]: https://chat.openai.com/share/a7d9e2ca-0104-469e-a6e4-6b08a5abe5aa https://chat.openai.com/share/a7d9e2ca-0104-469e-a6e4-6b08a5...
rad_gruchalski 3y ago
Recommendations are what I use ChatGPT for. “You are X doing Y, you know A, B, C. What 20 other things should you know. Recommend reading material.” Of course those prompts are usually bit longer.

veidr 3y ago

No, that's the only thing to understand about it! You can't trust anything it says about anything. What it does is emit plausible-sounding information. But we've seen -- I think across every single realm of anything -- that this information includes complete nonsense, whether due to it being trained on people just shitposting bullshit, or due to combining tokens (words, sentence fragments, whatever) in new ways without the benefit of any actual understanding of it. It can be useful, but it's at best as useful as a smart dog that's figured out how to operate a voice synthesizer, and is also on amphetamines or hallucinogens. I'm glad to see this term "LLM hallucination", as that's how it's felt to me. I find ChatGPT really useful for things that I can double-check instantaneously (or nearly so), such as doc comments for the code I just wrote, or small unit tests that match the comment I just wrote. But in my experience, it's worse than useless for writing real code, or any complex endeavor that isn't instantly verifiable/rejectable at a glance. Because vetting plausible looking code -- including dependency specification -- is almost always more taxing than just writing the code or package.json entries yourself.

visarga 3y ago
> No, that's the only thing to understand about it! You can't trust anything it says about anything. What it does is emit plausible-sounding information. But we've seen -- I think across every single realm of anything -- that this information includes complete nonsense And that is in my opinion the best argument why we won't be "replaced" by AI at work. This is consistent among all the tasks, NLP, vision, RL - they are all stumbling very quickly without assistance. How do you go from 0 to 1? AI will make for a nice assistant, really nice actually, but not a human replacement. Of course in the far future there will eventually come a day when AI might work without hand holding on some tasks, but we have no idea when. 15 years of SDC development attest to it, the closer you get to the goal, the most hidden hurdles you discover.
- taneq 3y ago
  You’re missing the reality of how automation “replaces” people. It doesn’t do one person’s job 100%, it does the easiest 90% of everyone’s job. You won’t be replaced by an AI, you and 8 other people will be replaced by an AI plus one guy who has good prompt-fu and editing/fact checking skills.

NikkiA 3y ago

Every time I ask it (3.5) to produce C code it assumes clang's blocks are present, even if I ask for C99 compliant code... blocks, blocks everywhere.

ulrischa 3y ago

taneq 3y ago

Treat its output like you’d treat an anonymous forum post. It can be a good source of pointers or ideas but don’t use it as ground truth for anything ever.

ryukoposting 3y ago

Does ChatGPT have a hard-wired incentive structure that encourages good-faith answers? No. Can you trust its package recommendations? No.

JestUM 3y ago

Sooner or later, this can accidentally slip into important pieces of code. Attackers could upload malicious modules that don't exist yet due these hallucinations.

kasrkin623 3y ago

But how do you make sure that it provides consistent hallucinations? Between GPT being consistently patched and such output being prone to statistical error I'm not sure if this vector of attack is efficient in the first place

johnisgood 3y ago
I really hate the term "hallucinations". By which definition of hallucination is it a hallucination, really?
- SirMaster 3y ago
  Well, often when I ask ChatGPT how to code something, it uses some external package that turns out doesn't exist. I don't know what else you would call that. Hallucination seems appropriate enough to me.
  - johnisgood 3y ago
    Okay, but if you ask me the same thing and I use a library or tell you the name of a library that does not exist (although the name comes close), would you say I am hallucinating? If not, then how could we apply this to AI if we cannot even apply the term used for humans to me?
    - SirMaster 3y ago
      OK, personally If a friend recommended me a library that didn't exist I would reply back to him "you made that up". So, ChatGPT is "making stuff up". To me hallucinating is a really similar term for "making something up". So the term works perfectly well for me. I mean the definition on Google at least is: "an experience involving the apparent perception of something not present." Isn't that what's happening? ChatGPT is experiencing (writing code using) the perception of something (the library) that is not present.
      - deleted 3y ago

johnisgood 3y ago

Probably not. It is outdated, and sometimes it gets the name of the libraries wrong, too.