3 ms·

This has been my experience as well, consistently. It's similar a phenomenon attributed to news: it's confident, and sounds plausible until the topic is on some

This has been my experience as well, consistently. It's similar a phenomenon attributed to news: it's confident, and sounds plausible until the topic is on something you know, then you realize it's full of errors. I don't trust ChatGpt to teach me something new; when I ask it about topics I do know about, it answers in a way that's a mix of correct, incorrect, and borderline/misleading information; all mixed with a confident tone. Based on this, I wouldn't use it to learn information or solve a problem where I don't already have a solid grasp on the material.

SoftTalker 4y ago
Yeah, 'confidently wrong "experts"' is nothing new. We just have to keep reminding ourselves that they exist. Gell-Mann amnesia is real. Also the models for these AI systems are built (in part) on "confidently wrong" inputs.
- gnunez 4y ago
  It could be that the corpus ChatGPT was trained on is full of ‘confidently wrong’ answers from these ‘experts’. One solution could be to train these LLM on a higher quality corpus from real experts instead of random text from the internet. But would that just bring us back to the days of expert systems?
  - DennisP 4y ago
    If we weren't completely hamstrung by copyright law we could legally train it on lots of actual books.
  - GuB-42 4y ago
    That will not solve the problem, because when GPT doesn't have the answer, it will make one up by copying the structure of correct answers but without any substance. For instance, let's say your LLM has never been told how many legs a snake has, it knows however that a snake is a reptile and that most reptiles have four legs. It will then confidently tell you "a snake has four legs", because it mirrors sentences like "a lizard has four legs" and "a crocodile has four legs" from its training set.
    - jjeaff 4y ago
      I don't think this is necessarily the case anymore. The bing implementation of chatgpt has a toggle for how cautious it should be with getting things wrong. I was working on a very niche issue today and asked it what a certain pin was designated for on a control board I am working on. I believe it is actually undocumented and wanted to see what chatgpt would say. And it actually said it didn't know and gave some tips on how I might figure it out. I suppose it is possible that it synthesized that answer from some previous q&a somewhere, but i couldn't find any mention of it online except for in the documentation.
- SigmundA 4y ago
  Con-man literally comes from confidence-man, if you have no morality or even ego and your only goal is to answer a question then confident answers will be the result regardless of their validity.
AceJohnny2 4y ago
> It's similar a phenomenon attributed to news: it's confident, and sounds plausible until the topic is on something you know, then you realize it's full of errors. Reminds me of the "Gell-Mann Amnesia effect" (as penned by Michael Crichton): https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnesiaEffect https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes... In short: trusting a source on topics you're unfamiliar with, but when the source talks about topics you are familiar with, you realize it's full of errors, but then turn the page and then trust it again on further topics you're unfamiliar with.
- azinman2 4y ago
  Sounds like journalism generally
- nagonago 4y ago
  The idea is stated succinctly as Knoll's Law of Media Accuracy: "Everything you read in the newspapers is absolutely true except for the rare story of which you happen to have firsthand knowledge." The earliest instance I found of Erwin Knoll's quote is from 1982[0]. I suspect either Michael Crichton or Gell-Mann had heard this quote, which in turn influenced their discussion. [0] https://www.nytimes.com/1982/02/27/us/required-reading-smith-on-lawyers.html https://www.nytimes.com/1982/02/27/us/required-reading-smith...
two_in_one 4y ago
From my experience it's a mixed bag. Sometimes it works perfectly. It wrote a simple graphics game in javascript in the first try, from just one prompt. Funny thing, I don't know much about JS or HTML. In another try it failed to create simple python program using opengl. Never managed to point camera at (0,0,0). Had problems with rotations. Did only 2 instead of 3. And so on in spite of me trying to correct. It was complete failure. Another case, python working with text files, did a great job. That was something useful.
- ptx 4y ago
  This sounds consistent with the parent comment's point. The case where it seemed to work perfectly for you is the one where you didn't know much about the subject.
  - Dylan16807 4y ago
    No, because it objectively worked. It made the game.

SoftTalker 4y ago

Yeah, 'confidently wrong "experts"' is nothing new. We just have to keep reminding ourselves that they exist. Gell-Mann amnesia is real. Also the models for these AI systems are built (in part) on "confidently wrong" inputs.

gnunez 4y ago
It could be that the corpus ChatGPT was trained on is full of ‘confidently wrong’ answers from these ‘experts’. One solution could be to train these LLM on a higher quality corpus from real experts instead of random text from the internet. But would that just bring us back to the days of expert systems?
- DennisP 4y ago
  If we weren't completely hamstrung by copyright law we could legally train it on lots of actual books.
- GuB-42 4y ago
  That will not solve the problem, because when GPT doesn't have the answer, it will make one up by copying the structure of correct answers but without any substance. For instance, let's say your LLM has never been told how many legs a snake has, it knows however that a snake is a reptile and that most reptiles have four legs. It will then confidently tell you "a snake has four legs", because it mirrors sentences like "a lizard has four legs" and "a crocodile has four legs" from its training set.
  - jjeaff 4y ago
    I don't think this is necessarily the case anymore. The bing implementation of chatgpt has a toggle for how cautious it should be with getting things wrong. I was working on a very niche issue today and asked it what a certain pin was designated for on a control board I am working on. I believe it is actually undocumented and wanted to see what chatgpt would say. And it actually said it didn't know and gave some tips on how I might figure it out. I suppose it is possible that it synthesized that answer from some previous q&a somewhere, but i couldn't find any mention of it online except for in the documentation.
SigmundA 4y ago
Con-man literally comes from confidence-man, if you have no morality or even ego and your only goal is to answer a question then confident answers will be the result regardless of their validity.

AceJohnny2 4y ago

> It's similar a phenomenon attributed to news: it's confident, and sounds plausible until the topic is on something you know, then you realize it's full of errors. Reminds me of the "Gell-Mann Amnesia effect" (as penned by Michael Crichton): https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnesiaEffect https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes... In short: trusting a source on topics you're unfamiliar with, but when the source talks about topics you are familiar with, you realize it's full of errors, but then turn the page and then trust it again on further topics you're unfamiliar with.

azinman2 4y ago
Sounds like journalism generally
nagonago 4y ago
The idea is stated succinctly as Knoll's Law of Media Accuracy: "Everything you read in the newspapers is absolutely true except for the rare story of which you happen to have firsthand knowledge." The earliest instance I found of Erwin Knoll's quote is from 1982[0]. I suspect either Michael Crichton or Gell-Mann had heard this quote, which in turn influenced their discussion. [0] https://www.nytimes.com/1982/02/27/us/required-reading-smith-on-lawyers.html https://www.nytimes.com/1982/02/27/us/required-reading-smith...

two_in_one 4y ago

From my experience it's a mixed bag. Sometimes it works perfectly. It wrote a simple graphics game in javascript in the first try, from just one prompt. Funny thing, I don't know much about JS or HTML. In another try it failed to create simple python program using opengl. Never managed to point camera at (0,0,0). Had problems with rotations. Did only 2 instead of 3. And so on in spite of me trying to correct. It was complete failure. Another case, python working with text files, did a great job. That was something useful.

ptx 4y ago
This sounds consistent with the parent comment's point. The case where it seemed to work perfectly for you is the one where you didn't know much about the subject.
- Dylan16807 4y ago
  No, because it objectively worked. It made the game.