10 ms·

This reminds me of something from “The language instinct” by Steven pinker. One thing that stuck with me was a story of a girl named Denys who is or was severel

This reminds me of something from “The language instinct” by Steven pinker. One thing that stuck with me was a story of a girl named Denys who is or was severely mentally handicapped, but speaks perfectly. She can’t read, write, handle money, but confabulates engaging and convincing stories about receiving mail, haggling with her bank, her boyfriend. She speaks with proper grammar, intonation, etc, but doesn’t demonstrate any understanding of what she talks about beyond the words. She doesn’t have a bank account or a boyfriend. Trying to find more and I can’t find anything about her on the internet, I would have to look at the references. Maybe it’s questionable. Anyway, I like to think about what the hell we even mean when we say we “know something”, and where we should actually put the bar for AI. Where is the line between confabulation and knowledge? Is knowledge just confabulation that happens to be correct? I feel like it’s possible that Denys could feel that she “knows” what she is saying, but is simply trapped in, with a working knowledge of the world and is just unable to translate it into actions, and words are just her only degree of freedom. I don’t remember where I was going with this, but I agree that “confabulation” is a much better term to use, but we also shouldn’t discount what it means to be able to confabulate.

galaxyLogic 3y ago
I had the same thought that "hallucinate" is too strong a term for what LLMs are doing. Instead I would say they DREAM. Dreams follow some logic but basically they are disjoint from reality. They involve same characters as our real-life experiences but what those characters do in a dream is not based on reality. So one could think that ALL LLMs ever do is dream but much of the time their dreams are dreams which feel very real. Our dreams are basically an LLM based on our real experiences but recombined in arbitrary ways, still retaining some logical structure but now detached from reality because when you recombine different experiences including our thoughts from real life, the result cannot really reflect reality very well. LLMs are better in this respect than our dreams, but sometimes their "dreams" get really obviously detached from reality. Or if you want to use the term "hallucinate" with LLMs then fine but its more like they hallucinate all the time, it's just that sometimes, even often, their hallucinations seem to agree with the reality so well that we cannot tell the difference. LLMs do not describe reality because they cannot experience reality, all they can give us is an "average description" created from many existing descriptions. One way of looking at it is to say that all LLMs tell us is hearsay.
- orobinson 3y ago
  But do LLMs dream of electric sheep?
  - trashtester 3y ago
    They certainly do after having read that script.
- deleted 3y ago
  [deleted]
- Dylan16807 3y ago
  The way I conceptualize dream versus hallucinate, they're basically the same thing here and I see no reason to switch. It's not like the software has a sleep schedule. And hearsay is usually a lot better than LLM output. I wouldn't use that word.
- KaiserPro 3y ago
  Dreams involve the concept of self, consciousness and an understanding of reality. all of it is far too anthropomorphic. Hallucinations are much better described as noise. Its noise that lowers the quality of the signal. its just unlike white noise, this is in the form of coherent sentences/imagery.
  - throwaway4aday 3y ago
    The term the article uses is more accurate. A hallucination can be entirely disjoint from the context it appears in. Confabulation is a confident belief in an erroneous recall that one will attempt to defend and rationalize. That's exactly what's happening with an LLM. They choose the wrong token or series of tokens, essentially failing to remember correctly, and then present it confidently and make rationalizations about why it's correct.
    - KaiserPro 3y ago
      > make rationalizations about why it's correct. Its not rationalising, its statistically choosing the next token. Also, generally the LLM doesn't get defensive (although its not always the case.) Its not like a dementia patient not finding their keys and gradually convincing themselves that someone stole them instead. Because the alternative, that they are loosing cognitive faculties, is too horrific to acknowledge. This is why its so annoyingly problematic. too much humanising, and not enough witnessing someone with dementia "confabulating" It is noise, and nothing else.
bigDinosaur 3y ago
It'd be well worth verifying the story because if Steven Pinker made an error like that it's a fairly big deal.
- upwardbound 3y ago
  I agree so I did some digging. Please see my comment here https://news.ycombinator.com/item?id=37776558 https://news.ycombinator.com/item?id=37776558
accounter8or 3y ago
That story actually cracked me up, the way he ends it.
- upwardbound 3y ago
  We must have different senses of humor. The ending paragraph of the story includes this horrifying, terrifying description of an awful and truly sad condition that I wouldn't wish on anyone. "Denyse was born with spina bifida ("split spine"), a malformation of the vertebrae that leaves the spinal cord unprotected. Spina bifida often results in hydrocephalus, an increase in pressure in the cerebrospinal fluid filling the ventricles (large cavities) of the brain, distending the brain from within. "
  - wincy 3y ago
    If it makes you feel any better, my daughter is about to turn five and has spina bifida. She is quite the chatterbox, but due to advances in modern medicine, got a brain shunt put in at a very early age so she’s normal intelligence for her age. Her prognosis is quite good, she’ll like live into her 60s or 70s. The shunt isn’t something we have to think about very often, but it’s truly a remarkable piece of hardware that makes it so she can live a fairly normal life.
    - TeMPOraL 3y ago
      That's the most heartwarming thing I read today. I am very happy we have technology to mitigate the effect of this horrifying condition, and that your daughter benefits from it. Thank you for sharing.
    - YurgenJurgensen 3y ago
      So here's a scary thought: If these people, if not given shunts, are eloquent enough to appear normal to someone not interrogating all the details, I could easily see one raised on a diet of the right kind of rhetoric being able to go on a convincing-sounding rant about how spina bifida isn't really a disability and that it was morally wrong to want to surgically correct their condition. And that that content is exactly the sort of thing that would make a successful viral TikTok video. Much like how LLMs will gladly and emphatically inform you that they're sentient beings with real feelings if given the right leading questions.
hamstergene 3y ago
Some of the most annoying moments in my career were occasional coworkers who talked a lot without making a point. They just invoked topics loosely related to the problem at hand, like stories from previous jobs or what they've read on the internet about this problem, until the manager said something like "are you trying to say that..." and then formulated what they thought the point was, except that the speaker clearly never intended to say anything like that. All the other engineers in the room are giving each other side eye but the manager thinks they've just got "a valuable input". The tactic is simply to talk relevantly until the listener fools themselves into thinking they understand you, except that "understanding" will be an illusion and 100% manager's own analytical effort. And it's honestly embarrassing how often and well that works. Quite a few of those guys were promoted first, too. I'm pretty sure our upcoming future with AI will be like that quote from forest ranger at Yosemite National Park, on why it is hard to design the perfect garbage bin to keep bears from breaking into it: "There is a considerable overlap between the intelligence of the smartest bears and the dumbest tourists." The whole idea of a criteria to separate humans from AI is a misconception. I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future. There are humans who already do this successfully to other humans, what do you think will happen when they get to automate their efforts.
- Jensson 3y ago
  > I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future Ads already ruthlessly exploit these people, taking their money in exchange for garbage products. LLM might make this worse, but we have already automated displaying targeted manipulative messages via ad networks. People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots.
  - TeMPOraL 3y ago
    > People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots. Not necessarily. LLMs at the model of GPT-4, and eventually above, affect the economics of advertising. Currently, there's only so much sense in optimizing your ads before it's cheaper to just spam more of it (er, "increase exposure"). LLMs will make it cheaper to optimize the ads further (i.e. get more bang for the same buck), so I expect the baseline manipulative effect of most ads to increase.
famouswaffles 3y ago
There's Lots of evidence indicating language models roughly know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305 https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205 https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know - [https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207 https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207... It seems very much like hallucinations aren't a randomness or representation issue. The computation already knows. It just doesn't care about telling you this
- corobo 3y ago
  I've been using it as a joke but if they're predicting what we want to read, is it possible they just learned to make stuff up by having internet forums in their training data? The amount of BS we feed each other online, maybe it just thinks we'd prefer something made up than "I don't know"
  - corobo 3y ago
    Not sure if it came off as me saying the parent comment was BS or if what I said was just dumb. Downvotes don't explain anything, haha For the record I wasn't saying the parent comment was BS :)
- QuadmasterXLII 3y ago
  RLHF also drops GPT3.5’s chess elo from ~1800 to ~800 which i confabulate to be connected
  - naveen99 3y ago
    Is this gpt3.5 Elo a modern chess.com elo ? I feel like there has been some deflation in elo in the last 2 years.
weinzierl 3y ago
I knew someone who was like Denys after a stroke. They spoke well, in grammatically correct sentences that made sense even if the content was not rooted in reality. There was not much coherence between sentences or in longer statements but it was more like quick gradual shifts not hard jumps from topic to topic. So not unlike early LLM chat in areas where the model had not a lot of data about.
upwardbound 3y ago
Here's the full passage in The Language Instinct about Denyse: Here is another interview, this one between a fourteen-year-old girl called Denyse and the late psycholinguist Richard Cromer; the interview was transcribed and analyzed by Cromer's colleague Sigrid Lipka. > I like opening cards. I had a pile of post this morning and not one of them was a Christmas card. A bank statement I got this morning! < [A bank statement? I hope it was good news.] > No it wasn't good news. < [Sounds like mine.] > I hate . . . , My mum works over at the, over on the ward and she said "not another bank statement." I said "it's the second one in two days." And she said "Do you want me to go to the bank for you at lunchtime?" and I went "No, I'll go this time and explain it myself." I tell you what, my bank are awful. They've lost my bank book, you see, and I can't find it anywhere. I belong to the TSB Bank and I'm thinking of changing my bank 'cause they're so awful. They keep, they keep losing . . . [someone comes in to bring some tea] Oh, isn't that nice. < [Uhm. Very good.] > They've got the habit of doing that. They lose, they've lost my bank book twice, in a month, and I think I'll scream. My mum went yesterday to the bank for me. She said "They've lost your bank book again." I went "Can I scream?" and I went, she went "Yes, go on." So I hollered. But it is annoying when they do things like that. TSB, Trustees aren't. . . uh the best ones to be with actually. They're hopeless. I have seen Denyse on videotape, and she comes across as a loquacious, sophisticated conversationalist—all the more so, to American ears, because of her refined British accent. (My bank are awful, by the way, is grammatical in British, though not American, English.) It comes as a surprise to learn that the events she relates so earnestly are figments of her imagination. Denyse has no bank account, so she could not have received any statements in the mail, nor could her bank have lost her bankbook. Though she would talk about a joint bank account she shared with her boyfriend, she had no boyfriend, and obviously had only the most tenuous grasp of the concept "joint bank account" because she complained about the boyfriend taking money out of her side of the account. In other conversations Denyse would engage her listeners with lively tales about the wedding of her sister, her holiday in Scotland with a boy named Danny, and a happy airport reunion with a long-estranged father. But Denyse's sister is unmarried, Denyse has never been to Scotland, she does not know anyone named Danny, and her father has never been away for any length of time. In fact, Denyse is severely retarded. She never learned to read or write and cannot handle money or any of the other demands of everyday functioning. Denyse was born with spina bifida ("split spine"), a malformation of the vertebrae that leaves the spinal cord unprotected. Spina bifida often results in hydrocephalus, an increase in pressure in the cerebrospinal fluid filling the ventricles (large cavities) of the brain, distending the brain from within. For reasons no one understands, hydrocephalic children occasionally end up like Denyse, significantly retarded but with unimpaired—indeed, overdeveloped—language skills. (Perhaps the ballooning ventricles crush much of the brain tissue necessary for everyday intelligence but leave intact some other portions that can develop language circuitry.) The various technical terms for the condition include "cocktail party conversation," "chatterbox syndrome," and "blathering." Source: http://f.javier.io/rep/books/The-Language-Instinct-How-the-Mind-Creates-Language,-Steven-Pinker.pdf http://f.javier.io/rep/books/The-Language-Instinct-How-the-M...

galaxyLogic 3y ago

I had the same thought that "hallucinate" is too strong a term for what LLMs are doing. Instead I would say they DREAM. Dreams follow some logic but basically they are disjoint from reality. They involve same characters as our real-life experiences but what those characters do in a dream is not based on reality. So one could think that ALL LLMs ever do is dream but much of the time their dreams are dreams which feel very real. Our dreams are basically an LLM based on our real experiences but recombined in arbitrary ways, still retaining some logical structure but now detached from reality because when you recombine different experiences including our thoughts from real life, the result cannot really reflect reality very well. LLMs are better in this respect than our dreams, but sometimes their "dreams" get really obviously detached from reality. Or if you want to use the term "hallucinate" with LLMs then fine but its more like they hallucinate all the time, it's just that sometimes, even often, their hallucinations seem to agree with the reality so well that we cannot tell the difference. LLMs do not describe reality because they cannot experience reality, all they can give us is an "average description" created from many existing descriptions. One way of looking at it is to say that all LLMs tell us is hearsay.

orobinson 3y ago
But do LLMs dream of electric sheep?
- trashtester 3y ago
  They certainly do after having read that script.
deleted 3y ago
[deleted]
Dylan16807 3y ago
The way I conceptualize dream versus hallucinate, they're basically the same thing here and I see no reason to switch. It's not like the software has a sleep schedule. And hearsay is usually a lot better than LLM output. I wouldn't use that word.
KaiserPro 3y ago
Dreams involve the concept of self, consciousness and an understanding of reality. all of it is far too anthropomorphic. Hallucinations are much better described as noise. Its noise that lowers the quality of the signal. its just unlike white noise, this is in the form of coherent sentences/imagery.
- throwaway4aday 3y ago
  The term the article uses is more accurate. A hallucination can be entirely disjoint from the context it appears in. Confabulation is a confident belief in an erroneous recall that one will attempt to defend and rationalize. That's exactly what's happening with an LLM. They choose the wrong token or series of tokens, essentially failing to remember correctly, and then present it confidently and make rationalizations about why it's correct.
  - KaiserPro 3y ago
    > make rationalizations about why it's correct. Its not rationalising, its statistically choosing the next token. Also, generally the LLM doesn't get defensive (although its not always the case.) Its not like a dementia patient not finding their keys and gradually convincing themselves that someone stole them instead. Because the alternative, that they are loosing cognitive faculties, is too horrific to acknowledge. This is why its so annoyingly problematic. too much humanising, and not enough witnessing someone with dementia "confabulating" It is noise, and nothing else.

bigDinosaur 3y ago

It'd be well worth verifying the story because if Steven Pinker made an error like that it's a fairly big deal.

upwardbound 3y ago
I agree so I did some digging. Please see my comment here https://news.ycombinator.com/item?id=37776558 https://news.ycombinator.com/item?id=37776558

accounter8or 3y ago

That story actually cracked me up, the way he ends it.

upwardbound 3y ago
We must have different senses of humor. The ending paragraph of the story includes this horrifying, terrifying description of an awful and truly sad condition that I wouldn't wish on anyone. "Denyse was born with spina bifida ("split spine"), a malformation of the vertebrae that leaves the spinal cord unprotected. Spina bifida often results in hydrocephalus, an increase in pressure in the cerebrospinal fluid filling the ventricles (large cavities) of the brain, distending the brain from within. "
- wincy 3y ago
  If it makes you feel any better, my daughter is about to turn five and has spina bifida. She is quite the chatterbox, but due to advances in modern medicine, got a brain shunt put in at a very early age so she’s normal intelligence for her age. Her prognosis is quite good, she’ll like live into her 60s or 70s. The shunt isn’t something we have to think about very often, but it’s truly a remarkable piece of hardware that makes it so she can live a fairly normal life.
  - TeMPOraL 3y ago
    That's the most heartwarming thing I read today. I am very happy we have technology to mitigate the effect of this horrifying condition, and that your daughter benefits from it. Thank you for sharing.
  - YurgenJurgensen 3y ago
    So here's a scary thought: If these people, if not given shunts, are eloquent enough to appear normal to someone not interrogating all the details, I could easily see one raised on a diet of the right kind of rhetoric being able to go on a convincing-sounding rant about how spina bifida isn't really a disability and that it was morally wrong to want to surgically correct their condition. And that that content is exactly the sort of thing that would make a successful viral TikTok video. Much like how LLMs will gladly and emphatically inform you that they're sentient beings with real feelings if given the right leading questions.

hamstergene 3y ago

Some of the most annoying moments in my career were occasional coworkers who talked a lot without making a point. They just invoked topics loosely related to the problem at hand, like stories from previous jobs or what they've read on the internet about this problem, until the manager said something like "are you trying to say that..." and then formulated what they thought the point was, except that the speaker clearly never intended to say anything like that. All the other engineers in the room are giving each other side eye but the manager thinks they've just got "a valuable input". The tactic is simply to talk relevantly until the listener fools themselves into thinking they understand you, except that "understanding" will be an illusion and 100% manager's own analytical effort. And it's honestly embarrassing how often and well that works. Quite a few of those guys were promoted first, too. I'm pretty sure our upcoming future with AI will be like that quote from forest ranger at Yosemite National Park, on why it is hard to design the perfect garbage bin to keep bears from breaking into it: "There is a considerable overlap between the intelligence of the smartest bears and the dumbest tourists." The whole idea of a criteria to separate humans from AI is a misconception. I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future. There are humans who already do this successfully to other humans, what do you think will happen when they get to automate their efforts.

Jensson 3y ago
> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future Ads already ruthlessly exploit these people, taking their money in exchange for garbage products. LLM might make this worse, but we have already automated displaying targeted manipulative messages via ad networks. People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots.
- TeMPOraL 3y ago
  > People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots. Not necessarily. LLMs at the model of GPT-4, and eventually above, affect the economics of advertising. Currently, there's only so much sense in optimizing your ads before it's cheaper to just spam more of it (er, "increase exposure"). LLMs will make it cheaper to optimize the ads further (i.e. get more bang for the same buck), so I expect the baseline manipulative effect of most ads to increase.

famouswaffles 3y ago

There's Lots of evidence indicating language models roughly know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305 https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205 https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know - [https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207 https://arxiv.org/abs/2207.05221](https://arxiv.org/abs/2207... It seems very much like hallucinations aren't a randomness or representation issue. The computation already knows. It just doesn't care about telling you this

corobo 3y ago
I've been using it as a joke but if they're predicting what we want to read, is it possible they just learned to make stuff up by having internet forums in their training data? The amount of BS we feed each other online, maybe it just thinks we'd prefer something made up than "I don't know"
- corobo 3y ago
  Not sure if it came off as me saying the parent comment was BS or if what I said was just dumb. Downvotes don't explain anything, haha For the record I wasn't saying the parent comment was BS :)
QuadmasterXLII 3y ago
RLHF also drops GPT3.5’s chess elo from ~1800 to ~800 which i confabulate to be connected
- naveen99 3y ago
  Is this gpt3.5 Elo a modern chess.com elo ? I feel like there has been some deflation in elo in the last 2 years.

weinzierl 3y ago

I knew someone who was like Denys after a stroke. They spoke well, in grammatically correct sentences that made sense even if the content was not rooted in reality. There was not much coherence between sentences or in longer statements but it was more like quick gradual shifts not hard jumps from topic to topic. So not unlike early LLM chat in areas where the model had not a lot of data about.

upwardbound 3y ago

Here's the full passage in The Language Instinct about Denyse: Here is another interview, this one between a fourteen-year-old girl called Denyse and the late psycholinguist Richard Cromer; the interview was transcribed and analyzed by Cromer's colleague Sigrid Lipka. > I like opening cards. I had a pile of post this morning and not one of them was a Christmas card. A bank statement I got this morning! < [A bank statement? I hope it was good news.] > No it wasn't good news. < [Sounds like mine.] > I hate . . . , My mum works over at the, over on the ward and she said "not another bank statement." I said "it's the second one in two days." And she said "Do you want me to go to the bank for you at lunchtime?" and I went "No, I'll go this time and explain it myself." I tell you what, my bank are awful. They've lost my bank book, you see, and I can't find it anywhere. I belong to the TSB Bank and I'm thinking of changing my bank 'cause they're so awful. They keep, they keep losing . . . [someone comes in to bring some tea] Oh, isn't that nice. < [Uhm. Very good.] > They've got the habit of doing that. They lose, they've lost my bank book twice, in a month, and I think I'll scream. My mum went yesterday to the bank for me. She said "They've lost your bank book again." I went "Can I scream?" and I went, she went "Yes, go on." So I hollered. But it is annoying when they do things like that. TSB, Trustees aren't. . . uh the best ones to be with actually. They're hopeless. I have seen Denyse on videotape, and she comes across as a loquacious, sophisticated conversationalist—all the more so, to American ears, because of her refined British accent. (My bank are awful, by the way, is grammatical in British, though not American, English.) It comes as a surprise to learn that the events she relates so earnestly are figments of her imagination. Denyse has no bank account, so she could not have received any statements in the mail, nor could her bank have lost her bankbook. Though she would talk about a joint bank account she shared with her boyfriend, she had no boyfriend, and obviously had only the most tenuous grasp of the concept "joint bank account" because she complained about the boyfriend taking money out of her side of the account. In other conversations Denyse would engage her listeners with lively tales about the wedding of her sister, her holiday in Scotland with a boy named Danny, and a happy airport reunion with a long-estranged father. But Denyse's sister is unmarried, Denyse has never been to Scotland, she does not know anyone named Danny, and her father has never been away for any length of time. In fact, Denyse is severely retarded. She never learned to read or write and cannot handle money or any of the other demands of everyday functioning. Denyse was born with spina bifida ("split spine"), a malformation of the vertebrae that leaves the spinal cord unprotected. Spina bifida often results in hydrocephalus, an increase in pressure in the cerebrospinal fluid filling the ventricles (large cavities) of the brain, distending the brain from within. For reasons no one understands, hydrocephalic children occasionally end up like Denyse, significantly retarded but with unimpaired—indeed, overdeveloped—language skills. (Perhaps the ballooning ventricles crush much of the brain tissue necessary for everyday intelligence but leave intact some other portions that can develop language circuitry.) The various technical terms for the condition include "cocktail party conversation," "chatterbox syndrome," and "blathering." Source: http://f.javier.io/rep/books/The-Language-Instinct-How-the-Mind-Creates-Language,-Steven-Pinker.pdf http://f.javier.io/rep/books/The-Language-Instinct-How-the-M...