6 ms·

That's not the point of the Gilligan's Island test. If the LLMs had said something to the effect of "I don't know, I wasn't trained to answer trivia about telev

That's not the point of the Gilligan's Island test. If the LLMs had said something to the effect of "I don't know, I wasn't trained to answer trivia about television" then I would agree with your comment. Instead they all confabulated! Either they made up an episode title or insisted the episode doesn't exist. That's a serious problem: if the LLM doesn't properly recognize that it doesn't know something, how will it properly recognize it needs to use a tool to obtain that knowledge? It seems like the only answer we have is for a user to flag that GPT-4+Web flubbed the Gilligan test, OpenAI dispatches a data contractor for RLHF whack-a-mole + synthetic data generation, GPT learns to Google answers to Gilligan's Island prompts, then we cross our fingers and hope transformers are smart enough to transfer that knowledge to the Sanford and Son benchmark.

sorokod 2y ago
...and +1 on "confabulate" to invent experiences or events that did not really happen
- nicbou 2y ago
  If I'm not mistaken, it also involves not knowing that you are doing so. > fabricate imaginary experiences as compensation for loss of memory
krapp 2y ago
>It seems like the only answer we have is for a user to flag that GPT-4+Web flubbed the Gilligan test OpenAI dispatches a data contractor for RLHF whack-a-mole + synthetic data generation, GPT learns to Google answers to Gilligan's Island prompts, then we cross our fingers and hope transformers are smart enough to transfer that knowledge to the Sanford and Son benchmark. Or maybe... let me run this up the flagpole and see if anyone salutes it... maybe we accept that LLMs have fundamental architectural limitations and can't do certain things like "math" or "anything requiring an awareness of context or factual accuracy" and don't use them for everything? So like instead of that, which wouldn't even work because search engines have already been polluted by AI generated garbage so they would just be eating their own shit, we have search engines that actually work again, and people just look that stuff up? And LLMs get relegated to whatever niche they're actually useful in, rather than the current plan of rebuilding our entire technological society on top of them as soon as possible because money?
- fragmede 2y ago
  I know when I'm lying. It doesn't seem totally insane to think that there's a node/tensor (not calling it a neuron) inside the model that is activated when it's confabulating that we could find and highlight and program our way into not happening.
  - krapp 2y ago
    From what I understand of the way LLMs work, they don't "know" when they're confabulating or not. All of the text they generate is arbitrary, albeit not entirely random. Whether or not any particular response is useful is a matter of human interpretation. The problem is the tendency to assume LLMs behave the same way humans do. You know you're lying when you're lying. LLMs don't even have a concept of a "lie." Even though you can ask one what a lie is, and it responds with an accurate answer, that's still just an arbitrary statistically based response. It doesn't actually know.
    - sfn42 2y ago
      It's crazy to me how even in here there's such a huge amount of people who just don't understand. Even the article were discussing points this out > The implications are that LLMs do not perform reasoning over data in the way that most people conceive or desire. > There is no self-reflection of its information; it does not know what it knows and what it does not. The line between hallucination and truth is simply a probability factored by the prevalence of training data and post-training processes like fine-tuning It's not thinking, it's not conscious, it's just a mathematical function that's too complex for us to understand.
      - anon373839 2y ago
        I think the best way to think of it is that it’s an estimation of the outputs of reasoning and knowledge. Of course this means that the models do need to model the reasoning process in some way, but it in no way implies that they model it in a way that’s rigorous or reliable.
blooalien 2y ago
> If the LLMs had said something to the effect of "I don't know, I wasn't trained to answer <question type> about <question topic>" ... This is so very much the answer I want from an LLM every single time it's not reasonably certain about an answer to a query. Not "hallucinate" a plausible sounding answer and then confidently spew lies at me as if it's gospel truth. I can accept "I don't know" from a human just fine, so I can damn sure accept it from a machine made by humans; but I'm far less tolerant of lies from a machine than I am from humans. Humans will lie for a great many reasons, a fair few of which are easily enough forgivable / understandable. A machine will generally "lie" for only a comparatively very few possible reasons (a physical flaw or defect in the machine; faulty data, be it purposeful or accidental; a human designed it to be untruthful, etc.), most of which are just plain largely unacceptable on multiple levels. Even better than "I don't know" would be my fifth grade teacher's favorite answer ("way back in ye golden olden days") in situations where he didn't know the answer: "I don't know, but let's find out together." and then the research would proceed apace. An LLM should be capable of such quite easily one would think. They make great "research assistants" when trained on a relevant data set and guided properly with a well crafted system prompt, and they're centered around / trained upon human language so should be able to guide a human to available resources with pretty near zero hassle. :)
- lasereyes136 2y ago
  I think part of the point of the article is that LLMs don't lie because they are designed to just give you the next work based on making a credible sounding sentence or sequence of sentences. Expecting it to do more is an expectations problem based on the hype around GenAI. I don't think we have the correct word for what LLMs do but lie and hallucinations are not really correct.
  - codewench 2y ago
    > I don't think we have the correct word for what LLMs do but lie and hallucinations are not really correct. I believe 'bullshit' is accurate, as in "The chatbot didn't know the answer, so it started bullshitting".
  - 2y ago
tiltowait 2y ago
I just got an interesting[0] response from Claude 3 Opus: > I don't believe there was an episode of Gilligan's Island that centered around mind reading. The show ran for 3 seasons from 1964-1967 and featured the comic adventures of 7 castaways on an uncharted island. Typical plotlines involved their efforts to get rescued and zany schemes by Gilligan that would go awry. But to my knowledge, none of the 98 episodes had a storyline focused on mind reading or telepathic abilities. It's probably the closest I can remember seeing an LLM get to saying "I don't know". "I don't believe there was" at least acknowledges the possibility of being incorrect and should prompt a careful user to do further research. [0] Also interesting is that one of the article's comments shows Opus giving the correct episode title but incorrect details. So ... mixed bag.
apantel 2y ago
It should be possible to come up with generalizable algorithms to determine a confidence score for any output, something akin to strong or weak connections in the brain. Is the response to the prompt supported by robust connections and strong weights? Or is it flitting around in weird, weak pathways. If the confidence score is below a certain level, some sort of ‘fact check’ feedback loop kicks in. Isn’t it roughly that simple?
- vrighter 2y ago
  that is already exactly how it works. The problem is that how confident it is has no relation to whether it actually knows. I can confidently spout bullshit all day long, if need be.
rgavuliak 2y ago
Maybe it's training data bias -> very few documents claim not to know something.
- yunwal 2y ago
  If you trained an LLM on questions where the answer “I don’t know” is possible, it would likely learn to answer any non yes/no question with “I don’t know”, since it’s probably the most common answer outside of yes/no
HarHarVeryFunny 2y ago
> if the LLM doesn't properly recognize that it doesn't know something, how will it properly recognize it needs to use a tool to obtain that knowledge? Maybe by doing it all/most of the time, the way that LLM/search hybrids like Perplexity and Bing/Copilot already do? Ideally an LLM would either be trained (or better learn for itself) when it's appropriate to use different types of tool. Web search (or offline WikiPedia lookup) could be the default.

sorokod 2y ago

...and +1 on "confabulate" to invent experiences or events that did not really happen

nicbou 2y ago
If I'm not mistaken, it also involves not knowing that you are doing so. > fabricate imaginary experiences as compensation for loss of memory

krapp 2y ago

>It seems like the only answer we have is for a user to flag that GPT-4+Web flubbed the Gilligan test OpenAI dispatches a data contractor for RLHF whack-a-mole + synthetic data generation, GPT learns to Google answers to Gilligan's Island prompts, then we cross our fingers and hope transformers are smart enough to transfer that knowledge to the Sanford and Son benchmark. Or maybe... let me run this up the flagpole and see if anyone salutes it... maybe we accept that LLMs have fundamental architectural limitations and can't do certain things like "math" or "anything requiring an awareness of context or factual accuracy" and don't use them for everything? So like instead of that, which wouldn't even work because search engines have already been polluted by AI generated garbage so they would just be eating their own shit, we have search engines that actually work again, and people just look that stuff up? And LLMs get relegated to whatever niche they're actually useful in, rather than the current plan of rebuilding our entire technological society on top of them as soon as possible because money?

fragmede 2y ago
I know when I'm lying. It doesn't seem totally insane to think that there's a node/tensor (not calling it a neuron) inside the model that is activated when it's confabulating that we could find and highlight and program our way into not happening.
- krapp 2y ago
  From what I understand of the way LLMs work, they don't "know" when they're confabulating or not. All of the text they generate is arbitrary, albeit not entirely random. Whether or not any particular response is useful is a matter of human interpretation. The problem is the tendency to assume LLMs behave the same way humans do. You know you're lying when you're lying. LLMs don't even have a concept of a "lie." Even though you can ask one what a lie is, and it responds with an accurate answer, that's still just an arbitrary statistically based response. It doesn't actually know.
  - sfn42 2y ago
    It's crazy to me how even in here there's such a huge amount of people who just don't understand. Even the article were discussing points this out > The implications are that LLMs do not perform reasoning over data in the way that most people conceive or desire. > There is no self-reflection of its information; it does not know what it knows and what it does not. The line between hallucination and truth is simply a probability factored by the prevalence of training data and post-training processes like fine-tuning It's not thinking, it's not conscious, it's just a mathematical function that's too complex for us to understand.
    - anon373839 2y ago
      I think the best way to think of it is that it’s an estimation of the outputs of reasoning and knowledge. Of course this means that the models do need to model the reasoning process in some way, but it in no way implies that they model it in a way that’s rigorous or reliable.

blooalien 2y ago

> If the LLMs had said something to the effect of "I don't know, I wasn't trained to answer <question type> about <question topic>" ... This is so very much the answer I want from an LLM every single time it's not reasonably certain about an answer to a query. Not "hallucinate" a plausible sounding answer and then confidently spew lies at me as if it's gospel truth. I can accept "I don't know" from a human just fine, so I can damn sure accept it from a machine made by humans; but I'm far less tolerant of lies from a machine than I am from humans. Humans will lie for a great many reasons, a fair few of which are easily enough forgivable / understandable. A machine will generally "lie" for only a comparatively very few possible reasons (a physical flaw or defect in the machine; faulty data, be it purposeful or accidental; a human designed it to be untruthful, etc.), most of which are just plain largely unacceptable on multiple levels. Even better than "I don't know" would be my fifth grade teacher's favorite answer ("way back in ye golden olden days") in situations where he didn't know the answer: "I don't know, but let's find out together." and then the research would proceed apace. An LLM should be capable of such quite easily one would think. They make great "research assistants" when trained on a relevant data set and guided properly with a well crafted system prompt, and they're centered around / trained upon human language so should be able to guide a human to available resources with pretty near zero hassle. :)

lasereyes136 2y ago
I think part of the point of the article is that LLMs don't lie because they are designed to just give you the next work based on making a credible sounding sentence or sequence of sentences. Expecting it to do more is an expectations problem based on the hype around GenAI. I don't think we have the correct word for what LLMs do but lie and hallucinations are not really correct.
- codewench 2y ago
  > I don't think we have the correct word for what LLMs do but lie and hallucinations are not really correct. I believe 'bullshit' is accurate, as in "The chatbot didn't know the answer, so it started bullshitting".
- 2y ago

tiltowait 2y ago

I just got an interesting[0] response from Claude 3 Opus: > I don't believe there was an episode of Gilligan's Island that centered around mind reading. The show ran for 3 seasons from 1964-1967 and featured the comic adventures of 7 castaways on an uncharted island. Typical plotlines involved their efforts to get rescued and zany schemes by Gilligan that would go awry. But to my knowledge, none of the 98 episodes had a storyline focused on mind reading or telepathic abilities. It's probably the closest I can remember seeing an LLM get to saying "I don't know". "I don't believe there was" at least acknowledges the possibility of being incorrect and should prompt a careful user to do further research. [0] Also interesting is that one of the article's comments shows Opus giving the correct episode title but incorrect details. So ... mixed bag.

apantel 2y ago

It should be possible to come up with generalizable algorithms to determine a confidence score for any output, something akin to strong or weak connections in the brain. Is the response to the prompt supported by robust connections and strong weights? Or is it flitting around in weird, weak pathways. If the confidence score is below a certain level, some sort of ‘fact check’ feedback loop kicks in. Isn’t it roughly that simple?

vrighter 2y ago
that is already exactly how it works. The problem is that how confident it is has no relation to whether it actually knows. I can confidently spout bullshit all day long, if need be.

rgavuliak 2y ago

Maybe it's training data bias -> very few documents claim not to know something.

yunwal 2y ago
If you trained an LLM on questions where the answer “I don’t know” is possible, it would likely learn to answer any non yes/no question with “I don’t know”, since it’s probably the most common answer outside of yes/no

HarHarVeryFunny 2y ago

> if the LLM doesn't properly recognize that it doesn't know something, how will it properly recognize it needs to use a tool to obtain that knowledge? Maybe by doing it all/most of the time, the way that LLM/search hybrids like Perplexity and Bing/Copilot already do? Ideally an LLM would either be trained (or better learn for itself) when it's appropriate to use different types of tool. Web search (or offline WikiPedia lookup) could be the default.