5 ms·

LLMs, kind of like Bill Bryson's books, are great at presenting "information" that seems completely plausible, authoritative, and convincing to the reader. But

LLMs, kind of like Bill Bryson's books, are great at presenting "information" that seems completely plausible, authoritative, and convincing to the reader. But when you actually do know the truth about a subject, you realize how completely full of crap they too often are. And somehow after being given a patently counterfactual response to one query, we just blindly continue to take their responses to other queries as having value.

niccl 11mo ago
> But when you actually do know the truth about a subject, you realize how completely full of crap they too often are The Gell-Mann Amnesia Effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
serial_dev 11mo ago
Similar to (same as?) Gell-Mann amnesia effect.
obelos 11mo ago
I've most frequently heard this referred to as “Gell-Mann Amnesia,” and yes, LLMs are fertile ground to find it.
rockostrich 11mo ago
At the moment, I find them to be the perfect tool to get started with learning about something. I don't expect it to tell me everything I need to know or to even be right, but if I ask ChatGPT or another LLM a question about a subject I'm not familiar with then it will at least use a bunch of terminology that I didn't have in my vocabulary before starting. For example, I just bought a 1990 Miata and I want to install a couple of rocker switches in the dash to individually control the pop-up headlights. I have enough circuits knowledge to safely change outlets and light switches, but I didn't know about relays. I asked ChatGPT how to add these switches and it immediately mentioned buying DPDT switches and tying in the OEM relay into a SPDT relay. It may have gotten the actual circuit diagram completely wrong, but now I know exactly what to read up on.
- kaydub 11mo ago
  Yeah, it's definitely been terrific for figuring out terminology or "the right word" to use for things.
  - knownjorbist 11mo ago
    Or to put it another way, it's great at filling in the "don't know what you don't know" gap.
- brewtide 11mo ago
  Completely not related to any LLM usage, but welcome to the world of NA Miata ownership! I think you'll find that with just general maintenance it'll treat you very well -- My '91 is the most reliable car in the drive, and by far the most whimsical. (I just got back from a Miata errand trip in the pouring rain -- Why did I drive the Miata? Winter is very soon, and it gets put away for ~3 ish months -- so at this time of year, every possible trip is a miata trip!)
  - a_t48 11mo ago
    This weekend I stumbled upon a cars and coffee in Fremont. Was expecting a wide variety of cars, and was surprised to see instead all Miatas.
  - rockostrich 11mo ago
    Thanks! I've been looking for a while but couldn't find one that was in decent shape for less than $10k. Thankfully for some reason people shy away from RHD cars and I snagged a 1990 Eunos Roadster for $7700 on C&B. I'm in NJ and sadly it seems like that week was the last week that would've been a decent week to drive with the top down. I may still try and take it out but I'm definitely going to be bundled up.
- goalieca 11mo ago
  Now let me ask you the more fundamental question.. did this do you any better than if you had searched a youtube video or some other source? Would this be video from 2016 be relevant? This may not be the right video but my approach for DIY in the last 10-20 years was to hit youtube up. https://www.youtube.com/watch?v=77q9KtjnNTU https://www.youtube.com/watch?v=77q9KtjnNTU I'm trying to gauge whether LLMs are truly expanding our capabilities in a fundamental way or are really just another way to search for answers without going to google or a library.
j-bos 11mo ago
> like Bill Bryson's books, are great at presenting "information" that seems completely plausible, authoritative, and convincing to the reader. But when you actually do know the truth about a subject, you realize how completely full of crap Wow, I have a couple Bill Bryson books on my reading list, can you share some examples of that?
- sarmasamosarma 11mo ago
  [dead]
- cynicalkane 11mo ago
  I read this good breakdown on 'The Mother Tongue' on everything2 sometime ago: https://everything2.com/title/The+Mother+Tongue%253A+English+and+How+it+Got+That+Way#google_vignette https://everything2.com/title/The+Mother+Tongue%253A+English...
  - dcminter 11mo ago
    Hmm. Why should I take this critique as being any more accurate than Bryson, given that the writer says in so many words: "[...] I - someone who’s far from an expert at linguistics [...]" The rather sniffy observation about Wikipedia falls very flat as the book was written 10 years before Wikipedia existed! In fact Bryson wrote his book a good 20 years earlier than this critique so perhaps this huffy person has resources to draw upon that were not available in 1990. Not that I really expect Bryson's stuff to dot every i and cross every t - he's a humourist.
    - deleted 11mo ago
      [deleted]
    - onraglanroad 11mo ago
      > Why should I take this critique as being any more accurate than Bryson Because you have access to various dictionaries and can easily verify it for yourself? Assuming the quotes from the book are accurate, that's really poor.
jb1991 11mo ago
I think it is truly hilarious that you brought Bill Bryson into this discussion.
Panoramix 11mo ago
I don't quite disagree but this comparison is typically unfair, because when you really know about a subject you tend to ask way more difficult questions than about other subjects, so of course the LLMs are gonna struggle more. If you ask really basic questions they will regurgitate well known bachelor-level knowledge and look good. What do I know about biology anyway? about silos for grain storage? any passable answer is enough to wow me on those topics. But on the topics I really know about, I never ask the basics.
- deleted 11mo ago
  [deleted]
- pklausler 11mo ago
  It's a valuable but scary experiment to query an LLM on basic subject matter in a field that you know a lot about. Ask those basic questions first.
beng-nl 11mo ago
geLLMan amnesia
ivape 11mo ago
Can you give me examples of topics that you know about that LLMs don’t know about?
austin-cheney 11mo ago
I read your comment and just see the typical web developer. I can’t count how many times developers have tried to school me with their expert wisdom. It’s typically garbage, complete Dunning-Kruger. The causes are two-fold. On one hand most of these people’s capabilities are an inch wide. Maybe they are really good about JSX, but you take that away and wisdom becomes empty hostility. I don’t want anything to do with JSX or React, so it that’s all you got you are probably just blowing smoke. The other cause is no experience at all. For example somebody might think they are super knowledgeable on WebSockets because they used a package off NPM. They have no idea how it really works, can’t understand RFC6455 even with Cliff Notes, and can’t write original code. If you want to be an expert at least start with your own implementation of the thing you want to be an expert about, but most of the people doing that work can’t program. I can’t help but wonder if the DK coming out of LLMs is really any worse.
shagie 11mo ago
I was curious about exploring the motivations of a character (specifically Linter in The State of the Art) and asked a question to start off with (and bring existing understanding into the context) with another character (Diziet Sma)... and ChatGPT got things wrong... The chat is https://chatgpt.com/share/691266fa-c76c-8011-876c-027206abd2fd https://chatgpt.com/share/691266fa-c76c-8011-876c-027206abd2... if one is curious. I continued a bit to see what else it got right and wrong. The thing is if you don't know the story or the books mentioned... it's perfectly plausible that what was written is correct. And while a good bit of it is... maybe; that it got material facts wrong makes it "if it's working from that, then nothing it produces is based on the correct information." I've known that ChatGPT is full of crap (and experienced in other chats). It can be a good tool to augment some capacities - but the exploration of ideas based on facts and reality are often (at best) flawed and if one is to try to build upon those flaws and add in one's own misconceptions, then its output is even more questionable.
hyperbolablabla 11mo ago
Indeed, a couple years ago I tried to use chatgpt to clarify some confusing parts of geometric algebra to me, and it confidently told me completely useless information which I had to blindly trust. Fast forward a couple years, I've read more on the subject from trusted sources, and realised everything it was telling me was pure BS.
seattle_spring 11mo ago
> LLMs, kind of like Bill Bryson's books I wonder if maybe Malcolm Gladwell would be a more apt comparison?
jonahx 11mo ago
> And somehow after being given a patently counterfactual response to one query, we just blindly continue to take their responses to other queries as having value. They have value because they are very fast, concise, and right "often enough." People used to make your criticism about Wikipedia (and occasionally still do). All of the following are true: 1. They regularly make errors 2. They require more caution than most people have to use effectively. 3. They have tremendous value.

niccl 11mo ago

> But when you actually do know the truth about a subject, you realize how completely full of crap they too often are The Gell-Mann Amnesia Effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

serial_dev 11mo ago

Similar to (same as?) Gell-Mann amnesia effect.

obelos 11mo ago

I've most frequently heard this referred to as “Gell-Mann Amnesia,” and yes, LLMs are fertile ground to find it.

rockostrich 11mo ago

At the moment, I find them to be the perfect tool to get started with learning about something. I don't expect it to tell me everything I need to know or to even be right, but if I ask ChatGPT or another LLM a question about a subject I'm not familiar with then it will at least use a bunch of terminology that I didn't have in my vocabulary before starting. For example, I just bought a 1990 Miata and I want to install a couple of rocker switches in the dash to individually control the pop-up headlights. I have enough circuits knowledge to safely change outlets and light switches, but I didn't know about relays. I asked ChatGPT how to add these switches and it immediately mentioned buying DPDT switches and tying in the OEM relay into a SPDT relay. It may have gotten the actual circuit diagram completely wrong, but now I know exactly what to read up on.

kaydub 11mo ago
Yeah, it's definitely been terrific for figuring out terminology or "the right word" to use for things.
- knownjorbist 11mo ago
  Or to put it another way, it's great at filling in the "don't know what you don't know" gap.
brewtide 11mo ago
Completely not related to any LLM usage, but welcome to the world of NA Miata ownership! I think you'll find that with just general maintenance it'll treat you very well -- My '91 is the most reliable car in the drive, and by far the most whimsical. (I just got back from a Miata errand trip in the pouring rain -- Why did I drive the Miata? Winter is very soon, and it gets put away for ~3 ish months -- so at this time of year, every possible trip is a miata trip!)
- a_t48 11mo ago
  This weekend I stumbled upon a cars and coffee in Fremont. Was expecting a wide variety of cars, and was surprised to see instead all Miatas.
- rockostrich 11mo ago
  Thanks! I've been looking for a while but couldn't find one that was in decent shape for less than $10k. Thankfully for some reason people shy away from RHD cars and I snagged a 1990 Eunos Roadster for $7700 on C&B. I'm in NJ and sadly it seems like that week was the last week that would've been a decent week to drive with the top down. I may still try and take it out but I'm definitely going to be bundled up.
goalieca 11mo ago
Now let me ask you the more fundamental question.. did this do you any better than if you had searched a youtube video or some other source? Would this be video from 2016 be relevant? This may not be the right video but my approach for DIY in the last 10-20 years was to hit youtube up. https://www.youtube.com/watch?v=77q9KtjnNTU https://www.youtube.com/watch?v=77q9KtjnNTU I'm trying to gauge whether LLMs are truly expanding our capabilities in a fundamental way or are really just another way to search for answers without going to google or a library.

j-bos 11mo ago

> like Bill Bryson's books, are great at presenting "information" that seems completely plausible, authoritative, and convincing to the reader. But when you actually do know the truth about a subject, you realize how completely full of crap Wow, I have a couple Bill Bryson books on my reading list, can you share some examples of that?

sarmasamosarma 11mo ago
[dead]
cynicalkane 11mo ago
I read this good breakdown on 'The Mother Tongue' on everything2 sometime ago: https://everything2.com/title/The+Mother+Tongue%253A+English+and+How+it+Got+That+Way#google_vignette https://everything2.com/title/The+Mother+Tongue%253A+English...
- dcminter 11mo ago
  Hmm. Why should I take this critique as being any more accurate than Bryson, given that the writer says in so many words: "[...] I - someone who’s far from an expert at linguistics [...]" The rather sniffy observation about Wikipedia falls very flat as the book was written 10 years before Wikipedia existed! In fact Bryson wrote his book a good 20 years earlier than this critique so perhaps this huffy person has resources to draw upon that were not available in 1990. Not that I really expect Bryson's stuff to dot every i and cross every t - he's a humourist.
  - deleted 11mo ago
    [deleted]
  - onraglanroad 11mo ago
    > Why should I take this critique as being any more accurate than Bryson Because you have access to various dictionaries and can easily verify it for yourself? Assuming the quotes from the book are accurate, that's really poor.

jb1991 11mo ago

I think it is truly hilarious that you brought Bill Bryson into this discussion.

Panoramix 11mo ago

I don't quite disagree but this comparison is typically unfair, because when you really know about a subject you tend to ask way more difficult questions than about other subjects, so of course the LLMs are gonna struggle more. If you ask really basic questions they will regurgitate well known bachelor-level knowledge and look good. What do I know about biology anyway? about silos for grain storage? any passable answer is enough to wow me on those topics. But on the topics I really know about, I never ask the basics.

deleted 11mo ago
[deleted]
pklausler 11mo ago
It's a valuable but scary experiment to query an LLM on basic subject matter in a field that you know a lot about. Ask those basic questions first.

beng-nl 11mo ago

geLLMan amnesia

ivape 11mo ago

Can you give me examples of topics that you know about that LLMs don’t know about?

austin-cheney 11mo ago

I read your comment and just see the typical web developer. I can’t count how many times developers have tried to school me with their expert wisdom. It’s typically garbage, complete Dunning-Kruger. The causes are two-fold. On one hand most of these people’s capabilities are an inch wide. Maybe they are really good about JSX, but you take that away and wisdom becomes empty hostility. I don’t want anything to do with JSX or React, so it that’s all you got you are probably just blowing smoke. The other cause is no experience at all. For example somebody might think they are super knowledgeable on WebSockets because they used a package off NPM. They have no idea how it really works, can’t understand RFC6455 even with Cliff Notes, and can’t write original code. If you want to be an expert at least start with your own implementation of the thing you want to be an expert about, but most of the people doing that work can’t program. I can’t help but wonder if the DK coming out of LLMs is really any worse.

shagie 11mo ago

I was curious about exploring the motivations of a character (specifically Linter in The State of the Art) and asked a question to start off with (and bring existing understanding into the context) with another character (Diziet Sma)... and ChatGPT got things wrong... The chat is https://chatgpt.com/share/691266fa-c76c-8011-876c-027206abd2fd https://chatgpt.com/share/691266fa-c76c-8011-876c-027206abd2... if one is curious. I continued a bit to see what else it got right and wrong. The thing is if you don't know the story or the books mentioned... it's perfectly plausible that what was written is correct. And while a good bit of it is... maybe; that it got material facts wrong makes it "if it's working from that, then nothing it produces is based on the correct information." I've known that ChatGPT is full of crap (and experienced in other chats). It can be a good tool to augment some capacities - but the exploration of ideas based on facts and reality are often (at best) flawed and if one is to try to build upon those flaws and add in one's own misconceptions, then its output is even more questionable.

hyperbolablabla 11mo ago

Indeed, a couple years ago I tried to use chatgpt to clarify some confusing parts of geometric algebra to me, and it confidently told me completely useless information which I had to blindly trust. Fast forward a couple years, I've read more on the subject from trusted sources, and realised everything it was telling me was pure BS.

seattle_spring 11mo ago

> LLMs, kind of like Bill Bryson's books I wonder if maybe Malcolm Gladwell would be a more apt comparison?

jonahx 11mo ago

> And somehow after being given a patently counterfactual response to one query, we just blindly continue to take their responses to other queries as having value. They have value because they are very fast, concise, and right "often enough." People used to make your criticism about Wikipedia (and occasionally still do). All of the following are true: 1. They regularly make errors 2. They require more caution than most people have to use effectively. 3. They have tremendous value.