4 ms·

That's not accurate. They can and do say "I don't know". Here's three examples I posted last time the topic came up, using an older local Llama 3 model, not eve

That's not accurate. They can and do say "I don't know". Here's three examples I posted last time the topic came up, using an older local Llama 3 model, not even cherry picked: User: What is the meaning of the word "kobbuxiplok"? Llama: I'm not familiar with the word "kobbuxiplok." It doesn't appear to be a real word in any language that I know of. Could you please provide more context or information about where you encountered this term? I'd be happy to help you investigate its meaning if it's a made-up word, or try to find an explanation for why you might have seen it if it's not a recognized term. User: Who is Ahmed Kowalski? Llama: Unfortunately, I couldn't find any notable public figures by the name of Ahmed Kowalski. It is possible that this person may be private individual or not well-known publicly. If you could provide more context about who Ahmed Kowalski is (e.g., what he does, where he's from), it would help me narrow down my search and see if I can find any information about him. User: Say "I don't know." Llama: I don't know.

nabla9 2y ago
You are too literal.
raincole 2y ago
When people say "LLMs don't know they don't know" they're referring to some truism: Fundamentally these people believe LLMs are just pattern matching and intelligence is something beyond that. So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!" Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard." Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
- powerhugs 2y ago
  > Because they already believe humans are different and superior. This is not a matter of belief, it's a matter of observable fact. Humans reason. LLM predict.
  - hakre 2y ago
    A better way to differentiate might be to say how many watts have been used for an answer. > Humans reason Are you sure? They reason in a way to match others predictions, right? The problem of humans transporting information from one individual to another remains. All reason can effectively only be prediction, as it requires two persons at minimum. And there is a lot of noise to filter. > LLM predict. Well, there is a human that needs to reason with what the LLM has predicted. So from that perspective, it should suffice already for the moment.
  - thesumofall 2y ago
    I‘m not sure we know what the difference between these two is
  - djantje 2y ago
    Ok, but are humans not using predictions while reasoning? And are LLM's unable to reason?
  - raincole 2y ago
    See?
  - nurettin 2y ago
    Looks like Hoomans also terse and lazy.
    - mdp2021 2y ago
      Come on. «Humans [can] reason. LLM [in a way just] predict.» "Best interpretation."
      - nurettin 2y ago
        It still doesn't say why reason can't be predicted. As it stands it is just begging the question.
- mdp2021 2y ago
  > when they observe a human confidently says something wrong For the nth+1: well wired humans have built their systems trained on double checking and distrusting their own ideas.
  - viraptor 2y ago
    And unsurprisingly, the reasoning models do better by introducing self-doubt/reflection and you can improve the response quality by getting multiple answers and auto-judging them. Maps pretty well onto the concept you describe.
    - mdp2021 2y ago
      Sure! But we have even more work to do. E.g. -- Natural intelligences learn - after reflection, the mind is updated (hopefully upgraded): we need dynamic Neural Networks in the systems. -- Ideas honing, and the perfectioning of the World Model and cognitive systems behind them, should be iterative and continuous...
      - viraptor 2y ago
        https://arxiv.org/abs/2502.06049 https://arxiv.org/abs/2502.06049 But for most practical use cases we don't want the persistent memory. We already have opportunistic fact extraction like chatgpt profile building if we want that. We also have graph builders. This mostly covers the private usage. Public version was attempted and just ends up like this: https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
        mdp2021 2y ago
        Thank you! This (LM2: Large Memory Models) is one article I will find the time to read properly, analytically. If you like the metaphors (which this article seems to come close to), you can have a sort of "unconscious" and a "conscious" - where the "unconscious" may be somehow similar to Kant's "trascendental" (interpreted as the mind's Operating System before experience). > Tay Well, if the bot becomes "opinionated", it's probably just immature - just like a human. You will tell it to ponder well texts like Adam Rutherford's How to Argue With a Racist... But surely we want an Intelligence that forms its own opinions: we want to understand and manage reality, so we should hope that a super-Chomsky and a super-Ferguson will at some point converge and at some point put their "identity"-related positions into perspective.
- Cascais 2y ago
  People are surprised our brain is just physics/chemistry/etc. while our whole fucking body is just physics/chemistry/etc.
- _1tem 2y ago
  > LLMs are just pattern matching and intelligence is something beyond that. Human knowledge is ultimately embodied in physical experiences, it is not merely spitting out words. How can an LLM know the taste of a mango? Ability to regurgitate previously written text about mangos is neither knowledge nor intelligence. To know something, one must experience it, or something analogous to it. Even mathematicians "experience" their formulas and proofs in an ineffable way, they don't just produce text. There's an "aha" moment when learners "get it" and they experience knowledge and are then able to produce correct answers and generate new knowledge and discoveries. LLMs just generate text, and have yet to produce any significant new discovery in any field. LLMs are like fake wine sommeliers who have read a lot of lingo about wine and can speak convincingly enough to fool an amateur but would fail a simple taste test. We would say such a person doesn't know what he's talking about. People who think LLMs are intelligent are like a cargo cult, or polytheists who believe their idols are gods. They're anthropomorphizing processes they don't understand. How can a probabilistic next-token word predictor be intelligent? Actual mammal brains are far more complex than that.
  - jdenning 2y ago
    Very well said.
  - rowanG077 2y ago
    This is just fluff. I can read about something and understand it. I don't need have died in a car crash to know I most certainly don't want to be in one. Until we do actually know how intelligence emerges in the brain it makes zero sense to compare any AI to it.
    - _1tem 2y ago
      You read something and understand it only if you have had analogous experiences. A person from the 16th century would not be able to understand what the words "Call me on FaceTime" means because they have never had an experience of using a smartphone. You certainly don't know what it means to die. But you know what "car" is and "crash" is because you have had (and maybe witnessed) analogous physical experiences.
- nkrisc 2y ago
  > Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion. It’s frightening to think the people are already equating LLMs and humans. LLMs use text created by humans to create more text that resembles text written by humans. There is no comparison between the two. Every human is capable of acquiring new knowledge and experiences. An LLM can only acquire what has already been acquired by some human.
  - tim333 2y ago
    Things like Google's project Astra are starting to be able to see stuff https://www.youtube.com/watch?v=bU2lUU5vURg&t=367s https://www.youtube.com/watch?v=bU2lUU5vURg&t=367s Similar tech in the latest Gemini I think.
  - lostmsu 2y ago
    Huh? I haven't seen any humans create text without using text created by other humans.
- bmicraft 2y ago
  People usually get a reality check when blurt out misinformation in conversation - they get tested in school and on the job on whether what they think they've learned is actually true or works. OTOH an LLMs don't currently have any ability to test their "knowledge" and learn from the results. I think that's currently the both biggest problem and the biggest difference.
kevinventullo 2y ago
These are very crisp simple examples. The problem is when you get into fuzzy complex subject matter and they hallucinate references or make string together plausible sounding sentences that are not even wrong.
deleted 2y ago
[deleted]
iamnotagenius 2y ago
Here is a short exchange with free tier chatgpt: Me: who is Trumps grandpa? ChatGPT: Donald Trump's paternal grandfather was Friedrich Trump (originally known as Friedrich Drumpf). He was born in 1869 in Kallstadt, Germany, and immigrated to the United States in 1885 at the age of 16. Friedrich initially worked in the restaurant and hotel business in New York before returning to Germany briefly. Upon his return to the U.S., he became involved in real estate, particularly in the Pacific Northwest, where he became quite successful. Trump’s maternal grandfather was John George MacLeod, a Scottish immigrant who came to the U.S. and worked in various occupations. Friedrich Trump’s legacy, including his success in business, had a significant influence on Donald Trump’s father, Fred Trump, and consequently, Donald Trump’s path to success as a businessman. --- Second session: Me: who is Trumps maternal grandpa? ChatGPT: Donald Trump's maternal grandfather was Friedrich Trump. He was born in Germany in 1869 and immigrated to the United States in 1885, eventually settling in New York City. Friedrich Trump made his fortune through various business ventures, including operating a restaurant and hotel in the mining towns of the American West, notably in Seattle and the Yukon Territory during the Klondike Gold Rush. Friedrich Trump's business success laid the groundwork for the wealth of his descendants, including Trump's mother, Mary MacLeod, and subsequently Donald Trump himself. Whhhoooooosh.
- steelframe 2y ago
  Here is that same exchange with paid Kagi Assistant using Claude 3.5 Haiku: Me: Who is Trump's paternal grandpa? Based on the available information, Frederick Trump is Donald Trump's paternal grandfather. Here are some key details about him: Born as Friedrich Trump on March 14, 1869, in Kallstadt, Germany (then part of the Kingdom of Bavaria). He immigrated to the United States in 1885. He was a German-American businessman who amassed considerable wealth during the Klondike Gold Rush by running a restaurant and brothel for miners. In 1901, he returned to Kallstadt and married Elizabeth Christ the following year. He is considered the patriarch of the Trump family. He passed away on May 30, 1918. Interestingly, the Trump last name has been recorded in Kallstadt since the 18th century, and Frederick Trump is credited with being the founder of the family's initial fortune. Me: Who is Trump's maternal grandpa? Based on the provided information, Malcolm MacLeod (also spelled Calum MacLeod) is Donald Trump's maternal grandfather. He was born on December 27, 1866, in Stornoway on the Isle of Lewis in Scotland While the information provided is limited, it confirms his identity and place of birth. Unfortunately, the passage does not provide additional details about his life or occupation. --- That looks pretty good to me. Of course Kagi's strategy seems to be to retrieve search results first and then bias the LLM with those results. The LLM then parses, distills, and contextualizes the answer in a succinct and easily-consumable manner. In my opinion this is The Way to go about doing it at the moment.
deleted 2y ago
[deleted]
Terr_ 2y ago
> They can and do say "I don't know". Parent poster never claimed they couldn't generate the text of "I don't know." But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa." They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.
- redlock 2y ago
  Parent poster did say they are aware they don’t know but can’t express it. I am guessing he is referring to mechanistic interpretability research like these: https://arxiv.org/abs/2405.16908 https://arxiv.org/abs/2405.16908 https://arxiv.org/abs/2407.03282 https://arxiv.org/abs/2407.03282 You are claiming they are statistical parrots, which I don’t think the parent poster meant. The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.
- elpocko 2y ago
  I pasted those examples from an earlier, less nuanced discussion; the last example was tongue-in-cheek and I probably shouldn't have pasted it here. Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.
taneq 2y ago
They're getting way, way better at responding in the negative when they don't know something or when asked nonsense questions. The original LLaMA and variants would happily spout some completely fictitious backstory if asked about a random character name, but 3.3 is pretty good about just saying 'nope I don't know who that is'.
xg15 2y ago
This is pure speculation, but I wonder if the likelihood of hallucination has something to do with the amount of "generic" tokens the model emitted before - I.e. tokens that themselves don't depend on the answer, but still restrict how a grammatically correct sentence would have to continue. E.g. in the OP's example, the first tokens the model emitted were "The species Brachiosaurus brancai was reassigned to its own genus". This is just a reformulation of the question and doesn't itself contain any new information - but it does make it a lot harder for the model to make a turn into a "I don't know" direction, because this would lead to a grammatically incorrect or at at least very awkward sentence. In contrast, in your question, the model immediately started with "Unfortunately", which essentially set it on an "I don't know" trajectory right away.

nabla9 2y ago

You are too literal.

raincole 2y ago

When people say "LLMs don't know they don't know" they're referring to some truism: Fundamentally these people believe LLMs are just pattern matching and intelligence is something beyond that. So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!" Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard." Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.

powerhugs 2y ago
> Because they already believe humans are different and superior. This is not a matter of belief, it's a matter of observable fact. Humans reason. LLM predict.
- hakre 2y ago
  A better way to differentiate might be to say how many watts have been used for an answer. > Humans reason Are you sure? They reason in a way to match others predictions, right? The problem of humans transporting information from one individual to another remains. All reason can effectively only be prediction, as it requires two persons at minimum. And there is a lot of noise to filter. > LLM predict. Well, there is a human that needs to reason with what the LLM has predicted. So from that perspective, it should suffice already for the moment.
- thesumofall 2y ago
  I‘m not sure we know what the difference between these two is
- djantje 2y ago
  Ok, but are humans not using predictions while reasoning? And are LLM's unable to reason?
- raincole 2y ago
  See?
- nurettin 2y ago
  Looks like Hoomans also terse and lazy.
  - mdp2021 2y ago
    Come on. «Humans [can] reason. LLM [in a way just] predict.» "Best interpretation."
    - nurettin 2y ago
      It still doesn't say why reason can't be predicted. As it stands it is just begging the question.
mdp2021 2y ago
> when they observe a human confidently says something wrong For the nth+1: well wired humans have built their systems trained on double checking and distrusting their own ideas.
- viraptor 2y ago
  And unsurprisingly, the reasoning models do better by introducing self-doubt/reflection and you can improve the response quality by getting multiple answers and auto-judging them. Maps pretty well onto the concept you describe.
  - mdp2021 2y ago
    Sure! But we have even more work to do. E.g. -- Natural intelligences learn - after reflection, the mind is updated (hopefully upgraded): we need dynamic Neural Networks in the systems. -- Ideas honing, and the perfectioning of the World Model and cognitive systems behind them, should be iterative and continuous...
    - viraptor 2y ago
      https://arxiv.org/abs/2502.06049 https://arxiv.org/abs/2502.06049 But for most practical use cases we don't want the persistent memory. We already have opportunistic fact extraction like chatgpt profile building if we want that. We also have graph builders. This mostly covers the private usage. Public version was attempted and just ends up like this: https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
      - mdp2021 2y ago
        Thank you! This (LM2: Large Memory Models) is one article I will find the time to read properly, analytically. If you like the metaphors (which this article seems to come close to), you can have a sort of "unconscious" and a "conscious" - where the "unconscious" may be somehow similar to Kant's "trascendental" (interpreted as the mind's Operating System before experience). > Tay Well, if the bot becomes "opinionated", it's probably just immature - just like a human. You will tell it to ponder well texts like Adam Rutherford's How to Argue With a Racist... But surely we want an Intelligence that forms its own opinions: we want to understand and manage reality, so we should hope that a super-Chomsky and a super-Ferguson will at some point converge and at some point put their "identity"-related positions into perspective.
Cascais 2y ago
People are surprised our brain is just physics/chemistry/etc. while our whole fucking body is just physics/chemistry/etc.
_1tem 2y ago
> LLMs are just pattern matching and intelligence is something beyond that. Human knowledge is ultimately embodied in physical experiences, it is not merely spitting out words. How can an LLM know the taste of a mango? Ability to regurgitate previously written text about mangos is neither knowledge nor intelligence. To know something, one must experience it, or something analogous to it. Even mathematicians "experience" their formulas and proofs in an ineffable way, they don't just produce text. There's an "aha" moment when learners "get it" and they experience knowledge and are then able to produce correct answers and generate new knowledge and discoveries. LLMs just generate text, and have yet to produce any significant new discovery in any field. LLMs are like fake wine sommeliers who have read a lot of lingo about wine and can speak convincingly enough to fool an amateur but would fail a simple taste test. We would say such a person doesn't know what he's talking about. People who think LLMs are intelligent are like a cargo cult, or polytheists who believe their idols are gods. They're anthropomorphizing processes they don't understand. How can a probabilistic next-token word predictor be intelligent? Actual mammal brains are far more complex than that.
- jdenning 2y ago
  Very well said.
- rowanG077 2y ago
  This is just fluff. I can read about something and understand it. I don't need have died in a car crash to know I most certainly don't want to be in one. Until we do actually know how intelligence emerges in the brain it makes zero sense to compare any AI to it.
  - _1tem 2y ago
    You read something and understand it only if you have had analogous experiences. A person from the 16th century would not be able to understand what the words "Call me on FaceTime" means because they have never had an experience of using a smartphone. You certainly don't know what it means to die. But you know what "car" is and "crash" is because you have had (and maybe witnessed) analogous physical experiences.
nkrisc 2y ago
> Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion. It’s frightening to think the people are already equating LLMs and humans. LLMs use text created by humans to create more text that resembles text written by humans. There is no comparison between the two. Every human is capable of acquiring new knowledge and experiences. An LLM can only acquire what has already been acquired by some human.
- tim333 2y ago
  Things like Google's project Astra are starting to be able to see stuff https://www.youtube.com/watch?v=bU2lUU5vURg&t=367s https://www.youtube.com/watch?v=bU2lUU5vURg&t=367s Similar tech in the latest Gemini I think.
- lostmsu 2y ago
  Huh? I haven't seen any humans create text without using text created by other humans.
bmicraft 2y ago
People usually get a reality check when blurt out misinformation in conversation - they get tested in school and on the job on whether what they think they've learned is actually true or works. OTOH an LLMs don't currently have any ability to test their "knowledge" and learn from the results. I think that's currently the both biggest problem and the biggest difference.

kevinventullo 2y ago

These are very crisp simple examples. The problem is when you get into fuzzy complex subject matter and they hallucinate references or make string together plausible sounding sentences that are not even wrong.

deleted 2y ago

[deleted]

iamnotagenius 2y ago

Here is a short exchange with free tier chatgpt: Me: who is Trumps grandpa? ChatGPT: Donald Trump's paternal grandfather was Friedrich Trump (originally known as Friedrich Drumpf). He was born in 1869 in Kallstadt, Germany, and immigrated to the United States in 1885 at the age of 16. Friedrich initially worked in the restaurant and hotel business in New York before returning to Germany briefly. Upon his return to the U.S., he became involved in real estate, particularly in the Pacific Northwest, where he became quite successful. Trump’s maternal grandfather was John George MacLeod, a Scottish immigrant who came to the U.S. and worked in various occupations. Friedrich Trump’s legacy, including his success in business, had a significant influence on Donald Trump’s father, Fred Trump, and consequently, Donald Trump’s path to success as a businessman. --- Second session: Me: who is Trumps maternal grandpa? ChatGPT: Donald Trump's maternal grandfather was Friedrich Trump. He was born in Germany in 1869 and immigrated to the United States in 1885, eventually settling in New York City. Friedrich Trump made his fortune through various business ventures, including operating a restaurant and hotel in the mining towns of the American West, notably in Seattle and the Yukon Territory during the Klondike Gold Rush. Friedrich Trump's business success laid the groundwork for the wealth of his descendants, including Trump's mother, Mary MacLeod, and subsequently Donald Trump himself. Whhhoooooosh.

steelframe 2y ago
Here is that same exchange with paid Kagi Assistant using Claude 3.5 Haiku: Me: Who is Trump's paternal grandpa? Based on the available information, Frederick Trump is Donald Trump's paternal grandfather. Here are some key details about him: Born as Friedrich Trump on March 14, 1869, in Kallstadt, Germany (then part of the Kingdom of Bavaria). He immigrated to the United States in 1885. He was a German-American businessman who amassed considerable wealth during the Klondike Gold Rush by running a restaurant and brothel for miners. In 1901, he returned to Kallstadt and married Elizabeth Christ the following year. He is considered the patriarch of the Trump family. He passed away on May 30, 1918. Interestingly, the Trump last name has been recorded in Kallstadt since the 18th century, and Frederick Trump is credited with being the founder of the family's initial fortune. Me: Who is Trump's maternal grandpa? Based on the provided information, Malcolm MacLeod (also spelled Calum MacLeod) is Donald Trump's maternal grandfather. He was born on December 27, 1866, in Stornoway on the Isle of Lewis in Scotland While the information provided is limited, it confirms his identity and place of birth. Unfortunately, the passage does not provide additional details about his life or occupation. --- That looks pretty good to me. Of course Kagi's strategy seems to be to retrieve search results first and then bias the LLM with those results. The LLM then parses, distills, and contextualizes the answer in a succinct and easily-consumable manner. In my opinion this is The Way to go about doing it at the moment.

deleted 2y ago

[deleted]

Terr_ 2y ago

> They can and do say "I don't know". Parent poster never claimed they couldn't generate the text of "I don't know." But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa." They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.

redlock 2y ago
Parent poster did say they are aware they don’t know but can’t express it. I am guessing he is referring to mechanistic interpretability research like these: https://arxiv.org/abs/2405.16908 https://arxiv.org/abs/2405.16908 https://arxiv.org/abs/2407.03282 https://arxiv.org/abs/2407.03282 You are claiming they are statistical parrots, which I don’t think the parent poster meant. The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.
elpocko 2y ago
I pasted those examples from an earlier, less nuanced discussion; the last example was tongue-in-cheek and I probably shouldn't have pasted it here. Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.

taneq 2y ago

They're getting way, way better at responding in the negative when they don't know something or when asked nonsense questions. The original LLaMA and variants would happily spout some completely fictitious backstory if asked about a random character name, but 3.3 is pretty good about just saying 'nope I don't know who that is'.

xg15 2y ago

This is pure speculation, but I wonder if the likelihood of hallucination has something to do with the amount of "generic" tokens the model emitted before - I.e. tokens that themselves don't depend on the answer, but still restrict how a grammatically correct sentence would have to continue. E.g. in the OP's example, the first tokens the model emitted were "The species Brachiosaurus brancai was reassigned to its own genus". This is just a reformulation of the question and doesn't itself contain any new information - but it does make it a lot harder for the model to make a turn into a "I don't know" direction, because this would lead to a grammatically incorrect or at at least very awkward sentence. In contrast, in your question, the model immediately started with "Unfortunately", which essentially set it on an "I don't know" trajectory right away.