5 ms·

This is all true for getting information from Google or other places. The same "critical reasoning" skills are required here, except the interaction is slightly

This is all true for getting information from Google or other places. The same "critical reasoning" skills are required here, except the interaction is slightly different.

pydry 3y ago
SERPs have more cues about trustworthiness but yes it is a little bit like Google's knowledge graph results: https://digitalreadymarketing.com/wp-content/uploads/2014/08/Google-Knowledge-Graph-Fail-greggs-company.png https://digitalreadymarketing.com/wp-content/uploads/2014/08...
rgrieselhuber 3y ago
Friendly reminder that critical thinking is dangerous. Just trust your betters. https://www.nytimes.com/2021/02/18/opinion/fake-news-media-attention.html https://www.nytimes.com/2021/02/18/opinion/fake-news-media-a...
wadd1e 3y ago
The problem is the average worker is more likely to blindly trust an "advanced chatbot"(from a layperson pov) than a random google result. (edit) how many people do you think are aware of LLM hallucinations as opposed to inaccurate google results
- mensetmanusman 3y ago
  It's trivial to build some type of never-perfect verification on top of LLM output. LLMs will be involved in complex tool chains like everything else.
- timcobb 3y ago
  > The problem is the average worker is more likely to blindly trust an "advanced chatbot"(from a layperson pov) than a random google result. Why? Any who understands about Google or Wikipedia can understand about LLM.
  - mrbungie 3y ago
    This. I don't think people lower their thresholds for applying critical thinking / fact checking just because it is an LLM's answer. If someone blindly trusts and uses an LLM without checking for more evidence, then they would've done the same with any other source, be it Google, the newspaper or whatever.
    - sottol 3y ago
      You know, like lawyers that present motions full of made-up case law in court? "In a cringe-inducing court hearing, a lawyer who relied on A.I. to craft a motion full of made-up case law said he “did not comprehend” that the chat bot could lead him astray." [1] [1] https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-sanctions.html https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-s...
      - timcobb 3y ago
        Yeah, lawyers can be negligent and lazy, too
  - wadd1e 3y ago
    I think you're giving too much credit to the average user, empirically, we as humans tend to question less and believe more about things that are beyond our understanding, and LLMs are certainly way more of a black box than a search engine. The idea of a search engine is intuitive enough, but a large language model which involves maths that people likely haven't even heard of? and is the next big thing(TM) and will revolutionise the world? Yeah I have a feeling it will take some time before the average user understands the ins and outs of this tech. Also the problem isn't necessarily about understanding the tech, but more about how it will be perceived at first. As long as every big player hypes up the tech as if it's perfect and revolutionary, I can't blame people for blindly trusting it at first(because it might as well work for most things they use it for, it will take some time before the pitfalls become apparent).
- krrrh 3y ago
  If you query Wikipedia or Google for an hour on a topic you know about there’s much less of a chance you’ll get inaccurate information than with ChatGPT where it’s almost a given. Assuming people get some basic training it won’t be hard to ask someone in the class about a baseball record and then let them see ChatGPT hallucinate the players and dates.
davb 3y ago
Except, LLMs don’t cite their sources accurately (if at all). A Google search may yield information with a source I can verify; my understanding is that LLMs are fundamentally incapable of doing so.
- 2devnull 3y ago
  If one gave you a 10,000 page bibliography would you read all the sources? The same problem exists (long bibliographies) in human knowledge transfer. I’m suggesting this is an architectural issue, not a LLM problem per se. LLMs are not supposed to be AGI.
  - creata 3y ago
    I never need to read all the sources, only the sources that are relevant to the bits that I'm referencing, no?
    - 2devnull 3y ago
      If you only need bits of info with citation, an LLM can do that. It’s synthesis that’s the problem. It’s true for humans as well.
      - NBJack 3y ago
        An actual citation and a string that happens to be formatted to look like a citation are very different. I wish we were at the point that most LLMs could do the former.
        hughesjj 3y ago
        I've seen the former happen for some GitHub docs and when enabling plugins. I agree there's cause for concern, and I hope it's a gradual rollout that's introspected in between each propagation of such, but I definitely think it's viable to be used no more less safely than other tooling in the government today (for better or worse).

This is all true for getting information from Google or other places. The same "critical reasoning" skills are required here, except the interaction is slightly different.

pydry 3y ago

SERPs have more cues about trustworthiness but yes it is a little bit like Google's knowledge graph results: https://digitalreadymarketing.com/wp-content/uploads/2014/08/Google-Knowledge-Graph-Fail-greggs-company.png https://digitalreadymarketing.com/wp-content/uploads/2014/08...

rgrieselhuber 3y ago

Friendly reminder that critical thinking is dangerous. Just trust your betters. https://www.nytimes.com/2021/02/18/opinion/fake-news-media-attention.html https://www.nytimes.com/2021/02/18/opinion/fake-news-media-a...

wadd1e 3y ago

The problem is the average worker is more likely to blindly trust an "advanced chatbot"(from a layperson pov) than a random google result. (edit) how many people do you think are aware of LLM hallucinations as opposed to inaccurate google results

mensetmanusman 3y ago
It's trivial to build some type of never-perfect verification on top of LLM output. LLMs will be involved in complex tool chains like everything else.
timcobb 3y ago
> The problem is the average worker is more likely to blindly trust an "advanced chatbot"(from a layperson pov) than a random google result. Why? Any who understands about Google or Wikipedia can understand about LLM.
- mrbungie 3y ago
  This. I don't think people lower their thresholds for applying critical thinking / fact checking just because it is an LLM's answer. If someone blindly trusts and uses an LLM without checking for more evidence, then they would've done the same with any other source, be it Google, the newspaper or whatever.
  - sottol 3y ago
    You know, like lawyers that present motions full of made-up case law in court? "In a cringe-inducing court hearing, a lawyer who relied on A.I. to craft a motion full of made-up case law said he “did not comprehend” that the chat bot could lead him astray." [1] [1] https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-sanctions.html https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-s...
    - timcobb 3y ago
      Yeah, lawyers can be negligent and lazy, too
- wadd1e 3y ago
  I think you're giving too much credit to the average user, empirically, we as humans tend to question less and believe more about things that are beyond our understanding, and LLMs are certainly way more of a black box than a search engine. The idea of a search engine is intuitive enough, but a large language model which involves maths that people likely haven't even heard of? and is the next big thing(TM) and will revolutionise the world? Yeah I have a feeling it will take some time before the average user understands the ins and outs of this tech. Also the problem isn't necessarily about understanding the tech, but more about how it will be perceived at first. As long as every big player hypes up the tech as if it's perfect and revolutionary, I can't blame people for blindly trusting it at first(because it might as well work for most things they use it for, it will take some time before the pitfalls become apparent).
krrrh 3y ago
If you query Wikipedia or Google for an hour on a topic you know about there’s much less of a chance you’ll get inaccurate information than with ChatGPT where it’s almost a given. Assuming people get some basic training it won’t be hard to ask someone in the class about a baseball record and then let them see ChatGPT hallucinate the players and dates.

davb 3y ago

Except, LLMs don’t cite their sources accurately (if at all). A Google search may yield information with a source I can verify; my understanding is that LLMs are fundamentally incapable of doing so.

2devnull 3y ago
If one gave you a 10,000 page bibliography would you read all the sources? The same problem exists (long bibliographies) in human knowledge transfer. I’m suggesting this is an architectural issue, not a LLM problem per se. LLMs are not supposed to be AGI.
- creata 3y ago
  I never need to read all the sources, only the sources that are relevant to the bits that I'm referencing, no?
  - 2devnull 3y ago
    If you only need bits of info with citation, an LLM can do that. It’s synthesis that’s the problem. It’s true for humans as well.
    - NBJack 3y ago
      An actual citation and a string that happens to be formatted to look like a citation are very different. I wish we were at the point that most LLMs could do the former.
      - hughesjj 3y ago
        I've seen the former happen for some GitHub docs and when enabling plugins. I agree there's cause for concern, and I hope it's a gradual rollout that's introspected in between each propagation of such, but I definitely think it's viable to be used no more less safely than other tooling in the government today (for better or worse).