4 ms·

For humans there's a social cost to wild lies and fabrications, even if one is otherwise generally reliable. I would probably consider a person who is wrong 50%

For humans there's a social cost to wild lies and fabrications, even if one is otherwise generally reliable. I would probably consider a person who is wrong 50% of the time but can reason about how they came to a conclusion and the limits of their knowledge/certainty to be more reliable than someone who is correct 90% of the time but lies/fabricates/hallucinates the other 10% of what they say. If a human acting in good faith is pressed for the evidence for something they said that is untrue, they will probably give a hazy recollection of how they got the information ("I think I read it in a NYT article", etc). They might be indignant, but they won't fabricate an equally erroneous trail of citations. ChatGPT produces some shockingly good text, but the rate of hallucinations and its inability to reliably reason about either correct or incorrect statements would be enough to mark a human as untrustworthy. The fact that LLMs can produce plausible, authoritative text that appears well evidenced, and can convincingly argue its validity regardless of any actual truth does however mean that we might be entering an era of ever more accessible and convincing fraud and misinformation.

ModernMech 4y ago
> ChatGPT produces some shockingly good text, but the rate of hallucinations and its inability to reliably reason about either correct or incorrect statements would be enough to mark a human as untrustworthy. It's not even the rate, which is troubling enough. It's the kinds of things it gets wrong too. For instance, you can say to ChatGPT, "Tell me about X" where X is something you made up. Then it will say "I don't know anything about X, why don't you tell me about it?" So you proceed to tell it about X, and eventually you ask "Tell me about X" and it will summarize what you've said. Here's where it gets strange. Now you start telling it more things about X, and it will start telling you that you're wrong. It didn't know anything about X before, now all of a sudden it's an authority on X, willing to correct actual an actual authority after knowing just a couple things. It will even assert its authority and expertise: as "As a language model, I must clarify that this statement is not entirely accurate". The "clarification" that followed was another lie and a non sequitur. Such clarity. What does ChatGPT mean by "As a language model, I must clarify". Why must it clarify? Why does its identity as "a language model" give it this imperative? Well, in actuality it doesn't, it's just saying things. But to the listener, it does. Language Models are currently being sold as passing the bar, passing medical exams, passing the SAT. They are being sold to us as experts before they've even established themselves. And now these so called experts are correcting humans about something it literally said it has no knowledge. If a 4-year old came up to you and said "As a four year old, I must clarify that this statement is not entirely accurate", you would dismiss them out of hand, because you know they just make shit up all the time. But not the language model that can pass the Bar, SAT, GRE, and MCATS?. Can you do that? No? Then why are you going to doubt the language model when it's trying to clear things up. Language models are going to be a boon for experts. I can spot the nonsense and correct in real time. For non experts, they when LLMs work they will work great, and when they don't you'll be left holding the bag when you act on its wrong information.
- withinboredom 4y ago
  My wife and I were just talking about this exact thing earlier today. I was using an AI to assist in some boring and repetitive “programming” with yaml. It was wrong a good chunk of the time, but I was mostly working as a “supervisor.” This would have been useless to the point of breaking things if a junior engineer had been using it. It even almost tripped me up a few times when it would write something correct, but with a punctuation in the wrong place. At least it made the repetitive task interesting.
- mattpallissard 4y ago
  > Language models are going to be a boon for experts. This is the key takeaway IMO.
- wizzwizz4 4y ago
  I'm concerned that they'll prevent non-experts from becoming experts. Most of my learning is done through observation: if I'm observing an endless stream of subtly-wrong bullshit, what am I learning?

ModernMech 4y ago

> ChatGPT produces some shockingly good text, but the rate of hallucinations and its inability to reliably reason about either correct or incorrect statements would be enough to mark a human as untrustworthy. It's not even the rate, which is troubling enough. It's the kinds of things it gets wrong too. For instance, you can say to ChatGPT, "Tell me about X" where X is something you made up. Then it will say "I don't know anything about X, why don't you tell me about it?" So you proceed to tell it about X, and eventually you ask "Tell me about X" and it will summarize what you've said. Here's where it gets strange. Now you start telling it more things about X, and it will start telling you that you're wrong. It didn't know anything about X before, now all of a sudden it's an authority on X, willing to correct actual an actual authority after knowing just a couple things. It will even assert its authority and expertise: as "As a language model, I must clarify that this statement is not entirely accurate". The "clarification" that followed was another lie and a non sequitur. Such clarity. What does ChatGPT mean by "As a language model, I must clarify". Why must it clarify? Why does its identity as "a language model" give it this imperative? Well, in actuality it doesn't, it's just saying things. But to the listener, it does. Language Models are currently being sold as passing the bar, passing medical exams, passing the SAT. They are being sold to us as experts before they've even established themselves. And now these so called experts are correcting humans about something it literally said it has no knowledge. If a 4-year old came up to you and said "As a four year old, I must clarify that this statement is not entirely accurate", you would dismiss them out of hand, because you know they just make shit up all the time. But not the language model that can pass the Bar, SAT, GRE, and MCATS?. Can you do that? No? Then why are you going to doubt the language model when it's trying to clear things up. Language models are going to be a boon for experts. I can spot the nonsense and correct in real time. For non experts, they when LLMs work they will work great, and when they don't you'll be left holding the bag when you act on its wrong information.

withinboredom 4y ago
My wife and I were just talking about this exact thing earlier today. I was using an AI to assist in some boring and repetitive “programming” with yaml. It was wrong a good chunk of the time, but I was mostly working as a “supervisor.” This would have been useless to the point of breaking things if a junior engineer had been using it. It even almost tripped me up a few times when it would write something correct, but with a punctuation in the wrong place. At least it made the repetitive task interesting.
mattpallissard 4y ago
> Language models are going to be a boon for experts. This is the key takeaway IMO.
wizzwizz4 4y ago
I'm concerned that they'll prevent non-experts from becoming experts. Most of my learning is done through observation: if I'm observing an endless stream of subtly-wrong bullshit, what am I learning?