5 ms·

Intelligence and knowledge are distinct concepts. Asking about it's knowledge teaches noting about it's intelligence. Intelligence is the ability to learn, rea

by nabla9 2y ago

Intelligence and knowledge are distinct concepts. Asking about it's knowledge teaches noting about it's intelligence. Intelligence is the ability to learn, reason, and solve problems. Knowledge is the accumulation of facts and skills. Chatbot LLM's don't have metacognition. They don't know that they don't know. If you peek inside the LLM, the process seems different for things they don't know. They just can't express it because they are trained to produce good probability outcome instead of accurate one. They have potential as knowledge databases, but someone must figure out how to get "I don't know" information out of them.

bt1a 2y ago
You've beautifully put what swirls vaguely in my mind. They're useful, fallible tools with extraordinary function when operating within known and reasonable tolerances of error
- nabla9 2y ago
  They can also reason, but the reasoning is limited and unreliable. Q:How many playing cards are needed for a pyramid that is 3 layers high? Show reasoning and number of cards for leach layer. Q: Chess. You have a King and 8 pawns. Your opponent has a King and 16 pawns. Your opponent plays white and can start, but you can position both your pawns and your opponents pawns any way you like before game starts. Kings are where they are normally. How do you do it? Explain your reasoning.
me_me_me 2y ago
Here is a small kicker. Human brains absolutely do the same. I split brain patients there are behaviours initiated by one hemisphere not known to the other (due to severed connection) and the person part of brain will make up a reason (often quite stupid) for the action and beleive it 100%. It's eirely similar to hallucinations of ai. That said a current llms are not aware, but are starting to act more and more like it.
- kristiandupont 2y ago
  I had a similar insight (blog post: [link redacted]). In a very unscientific way, I would say that the LLM is not the whole brain, it's part of it and we are still in the process of simulating other parts. But it does seem to me like we've solved the hard part, and it's astonishing to me that people like authors of this article seem to think that the current state of things is where evolution stops.
  - me_me_me 2y ago
    The brain works through flow of stimulation. Inputs and outputs probably creating consciousness somehow. LLMs are limited in self stimulation and it's memory model is very basic compare to humans. But I feel most people come with gut feeling position that we have divine spark that makes us tick. And the view that we might be more basic even crude akin to LLMs makes them uneasy.
- marci 2y ago
  "You are Two" (CGP Grey) https://www.youtube.com/watch?v=wfYbgdo8e-8 https://www.youtube.com/watch?v=wfYbgdo8e-8
- iamnotagenius 2y ago
  No it is not neither eerily nor computationaly similar to AI hallucinations, as we do not have a dam idea how the brains think.
elpocko 2y ago
That's not accurate. They can and do say "I don't know". Here's three examples I posted last time the topic came up, using an older local Llama 3 model, not even cherry picked: User: What is the meaning of the word "kobbuxiplok"? Llama: I'm not familiar with the word "kobbuxiplok." It doesn't appear to be a real word in any language that I know of. Could you please provide more context or information about where you encountered this term? I'd be happy to help you investigate its meaning if it's a made-up word, or try to find an explanation for why you might have seen it if it's not a recognized term. User: Who is Ahmed Kowalski? Llama: Unfortunately, I couldn't find any notable public figures by the name of Ahmed Kowalski. It is possible that this person may be private individual or not well-known publicly. If you could provide more context about who Ahmed Kowalski is (e.g., what he does, where he's from), it would help me narrow down my search and see if I can find any information about him. User: Say "I don't know." Llama: I don't know.
- nabla9 2y ago
  You are too literal.
- raincole 2y ago
  When people say "LLMs don't know they don't know" they're referring to some truism: Fundamentally these people believe LLMs are just pattern matching and intelligence is something beyond that. So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!" Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard." Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
  - powerhugs 2y ago
    > Because they already believe humans are different and superior. This is not a matter of belief, it's a matter of observable fact. Humans reason. LLM predict.
wavefunction 2y ago
Its
exitb 2y ago
> If you peek inside the LLM, the process seems different for things they don't know. This is really interesting, could you link some relevant resources? I wonder if a chatbot could at least signal at the UI level that the answer might be a hallucination.
jdietrich 2y ago
>Chatbot LLM's don't have metacognition. The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially indistinguishable from metacognition. It is fascinating to watch R1 grapple with an unanswerable question, torn between the urge to be helpful and the duty to be accurate; the reasoning tokens will be peppered with phrases like "I'm not sure", "I could be mistaken about that", "let me double-check that" and "maybe I'm looking at this from the wrong angle".
- gizajob 2y ago
  superficially being the operative word.
- kaoD 2y ago
  > at least a simulacrum that is superficially indistinguishable from metacognition That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>". Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer. It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.
  - thesumofall 2y ago
    I feel it’s very rare that people openly and consciously reflect about what they do or do not know. And even then I find it questionable if we are really capable of identifying the difference reliably. However, most of the time I feel we just make it up - esp in none-trivial areas. Just think of the average business meeting
  - gus_massa 2y ago
powerhugs 2y ago
> Chatbot LLM's don't have metacognition. They don't know that they don't know. Correct. LLM also don't have intelligence. What you and others mistake for intelligence is a very clever prediction model. A LLM don't reason at all. It only tells you what is a most likely response based on its training data. This is why the LLM cannot tell you it don't know.
- rvz 2y ago
  Spot on. But unfortunately the AI grifters are now active and selling this snake-oil on to the rest of us. > This is why the LLM cannot tell you it don't know. To the untrained eye it appears that it knows everything about every question it answers when in fact those who are experts in their own domains can detect if it is hallucinated and generated complete slop.
  - baxtr 2y ago
    "It’s over." "We’re cooked." Typical AI grifters’ clickbait intros.
- Shorel 2y ago
  > What you and others mistake for intelligence is a very clever prediction model. It is not a very clever prediction model. It is a very big, incredibly large humongous model that finds stuff basically by brute-forcing pattern matching over almost all text we have produced in the past. A clever model would get us similar results using the same amount of energy a human brain uses for that task, which is tiny.
- viraptor 2y ago
  People throw around the "intelligence" and "reasoning" arguments as if we have commonly understood and strict definitions of those concepts and don't constantly have issues of either including or excluding unexpected cases. Maybe once we solve simple issues like "are cephalopods intelligent" and "are people deterministic prediction machines", we can start talking about classifying Ai capabilities...
rvz 2y ago
Of course. These systems are non-deterministic and you still would require those with domain expertise to extra verify whatever these LLMs output are 300% correct as they do not reason. (Yes, they do not.) Otherwise, why aren't LLMs and humanoids already replacing human pilots for flying airplanes end-to-end? Sometimes using a hammer onto every problem isn't a solution; even if the LLM tells you otherwise.
2-3-7-43-1807 2y ago
metacognition is a pretty lofty concept. but practically speaking you are wrong. i recommend watching karpathy's last video on llms. it is possible to fine tune a model specifically on uncertain questions to produce a statement expressing uncertainty.
- iamnotagenius 2y ago
  "Practically speaking" GP is obviously blatantly right, as there is no LLM in existence capable of doing what Karpathy claims is possible.
fenomas 2y ago
> but someone must figure out how to get "I don't know" information out of them. In Andrej Karpathy's most recent video he discusses in some detail how this is already done and how effective it is.
- alecco 2y ago
  https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s 01:20:32 hallucinations, tool use, knowledge/working memory
- vintagedave 2y ago
  I looked this up: for other readers, it starts at 1:20. https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s
xg15 2y ago
I don't think you even have to look deep into the model for it. There seem to be some projects who are simply analyzing the logits during decoding to estimate how "certain" the model is of the next token. But if all people do is random sampling or blindly taking the highest-scored prediction, this will of course fall under the table.
roncesvalles 2y ago
Hard disagree. Knowledge and intelligence are inextricable because the way that humans learn advanced concepts involves a lot of thinking and reasoning. You don't just learn by ingesting facts from a book line by line. You learn by reading a little, then pausing to think, building connections with other things you know, rephrasing the information, deducing the unwritten parts, finding patterns, fleshing out the contrapositives, being critical, etc. This is especially true when learning advanced topics. I would say learning is more about information processing than information uptake. That's why more intelligent people learn faster. The linear prose of natural language is actually a terribly lossy way to serialize and transmit the knowledge that is in a person's brain. The receiver must deduce most of it given a small seed.

by nabla9 2y ago

bt1a 2y ago

You've beautifully put what swirls vaguely in my mind. They're useful, fallible tools with extraordinary function when operating within known and reasonable tolerances of error

nabla9 2y ago
They can also reason, but the reasoning is limited and unreliable. Q:How many playing cards are needed for a pyramid that is 3 layers high? Show reasoning and number of cards for leach layer. Q: Chess. You have a King and 8 pawns. Your opponent has a King and 16 pawns. Your opponent plays white and can start, but you can position both your pawns and your opponents pawns any way you like before game starts. Kings are where they are normally. How do you do it? Explain your reasoning.

me_me_me 2y ago

Here is a small kicker. Human brains absolutely do the same. I split brain patients there are behaviours initiated by one hemisphere not known to the other (due to severed connection) and the person part of brain will make up a reason (often quite stupid) for the action and beleive it 100%. It's eirely similar to hallucinations of ai. That said a current llms are not aware, but are starting to act more and more like it.

kristiandupont 2y ago
I had a similar insight (blog post: [link redacted]). In a very unscientific way, I would say that the LLM is not the whole brain, it's part of it and we are still in the process of simulating other parts. But it does seem to me like we've solved the hard part, and it's astonishing to me that people like authors of this article seem to think that the current state of things is where evolution stops.
- me_me_me 2y ago
  The brain works through flow of stimulation. Inputs and outputs probably creating consciousness somehow. LLMs are limited in self stimulation and it's memory model is very basic compare to humans. But I feel most people come with gut feeling position that we have divine spark that makes us tick. And the view that we might be more basic even crude akin to LLMs makes them uneasy.
marci 2y ago
"You are Two" (CGP Grey) https://www.youtube.com/watch?v=wfYbgdo8e-8 https://www.youtube.com/watch?v=wfYbgdo8e-8
iamnotagenius 2y ago
No it is not neither eerily nor computationaly similar to AI hallucinations, as we do not have a dam idea how the brains think.

elpocko 2y ago

That's not accurate. They can and do say "I don't know". Here's three examples I posted last time the topic came up, using an older local Llama 3 model, not even cherry picked: User: What is the meaning of the word "kobbuxiplok"? Llama: I'm not familiar with the word "kobbuxiplok." It doesn't appear to be a real word in any language that I know of. Could you please provide more context or information about where you encountered this term? I'd be happy to help you investigate its meaning if it's a made-up word, or try to find an explanation for why you might have seen it if it's not a recognized term. User: Who is Ahmed Kowalski? Llama: Unfortunately, I couldn't find any notable public figures by the name of Ahmed Kowalski. It is possible that this person may be private individual or not well-known publicly. If you could provide more context about who Ahmed Kowalski is (e.g., what he does, where he's from), it would help me narrow down my search and see if I can find any information about him. User: Say "I don't know." Llama: I don't know.

nabla9 2y ago
You are too literal.
raincole 2y ago
When people say "LLMs don't know they don't know" they're referring to some truism: Fundamentally these people believe LLMs are just pattern matching and intelligence is something beyond that. So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!" Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard." Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
- powerhugs 2y ago
  > Because they already believe humans are different and superior. This is not a matter of belief, it's a matter of observable fact. Humans reason. LLM predict.

wavefunction 2y ago

Its

exitb 2y ago

> If you peek inside the LLM, the process seems different for things they don't know. This is really interesting, could you link some relevant resources? I wonder if a chatbot could at least signal at the UI level that the answer might be a hallucination.

jdietrich 2y ago

>Chatbot LLM's don't have metacognition. The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially indistinguishable from metacognition. It is fascinating to watch R1 grapple with an unanswerable question, torn between the urge to be helpful and the duty to be accurate; the reasoning tokens will be peppered with phrases like "I'm not sure", "I could be mistaken about that", "let me double-check that" and "maybe I'm looking at this from the wrong angle".

gizajob 2y ago
superficially being the operative word.
kaoD 2y ago
> at least a simulacrum that is superficially indistinguishable from metacognition That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>". Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer. It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.
- thesumofall 2y ago
  I feel it’s very rare that people openly and consciously reflect about what they do or do not know. And even then I find it questionable if we are really capable of identifying the difference reliably. However, most of the time I feel we just make it up - esp in none-trivial areas. Just think of the average business meeting
- gus_massa 2y ago

powerhugs 2y ago

> Chatbot LLM's don't have metacognition. They don't know that they don't know. Correct. LLM also don't have intelligence. What you and others mistake for intelligence is a very clever prediction model. A LLM don't reason at all. It only tells you what is a most likely response based on its training data. This is why the LLM cannot tell you it don't know.

rvz 2y ago
Spot on. But unfortunately the AI grifters are now active and selling this snake-oil on to the rest of us. > This is why the LLM cannot tell you it don't know. To the untrained eye it appears that it knows everything about every question it answers when in fact those who are experts in their own domains can detect if it is hallucinated and generated complete slop.
- baxtr 2y ago
  "It’s over." "We’re cooked." Typical AI grifters’ clickbait intros.
Shorel 2y ago
> What you and others mistake for intelligence is a very clever prediction model. It is not a very clever prediction model. It is a very big, incredibly large humongous model that finds stuff basically by brute-forcing pattern matching over almost all text we have produced in the past. A clever model would get us similar results using the same amount of energy a human brain uses for that task, which is tiny.
viraptor 2y ago
People throw around the "intelligence" and "reasoning" arguments as if we have commonly understood and strict definitions of those concepts and don't constantly have issues of either including or excluding unexpected cases. Maybe once we solve simple issues like "are cephalopods intelligent" and "are people deterministic prediction machines", we can start talking about classifying Ai capabilities...

rvz 2y ago

Of course. These systems are non-deterministic and you still would require those with domain expertise to extra verify whatever these LLMs output are 300% correct as they do not reason. (Yes, they do not.) Otherwise, why aren't LLMs and humanoids already replacing human pilots for flying airplanes end-to-end? Sometimes using a hammer onto every problem isn't a solution; even if the LLM tells you otherwise.

2-3-7-43-1807 2y ago

metacognition is a pretty lofty concept. but practically speaking you are wrong. i recommend watching karpathy's last video on llms. it is possible to fine tune a model specifically on uncertain questions to produce a statement expressing uncertainty.

iamnotagenius 2y ago
"Practically speaking" GP is obviously blatantly right, as there is no LLM in existence capable of doing what Karpathy claims is possible.

fenomas 2y ago

> but someone must figure out how to get "I don't know" information out of them. In Andrej Karpathy's most recent video he discusses in some detail how this is already done and how effective it is.

alecco 2y ago
https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s 01:20:32 hallucinations, tool use, knowledge/working memory
vintagedave 2y ago
I looked this up: for other readers, it starts at 1:20. https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s

xg15 2y ago

I don't think you even have to look deep into the model for it. There seem to be some projects who are simply analyzing the logits during decoding to estimate how "certain" the model is of the next token. But if all people do is random sampling or blindly taking the highest-scored prediction, this will of course fall under the table.

roncesvalles 2y ago

Hard disagree. Knowledge and intelligence are inextricable because the way that humans learn advanced concepts involves a lot of thinking and reasoning. You don't just learn by ingesting facts from a book line by line. You learn by reading a little, then pausing to think, building connections with other things you know, rephrasing the information, deducing the unwritten parts, finding patterns, fleshing out the contrapositives, being critical, etc. This is especially true when learning advanced topics. I would say learning is more about information processing than information uptake. That's why more intelligent people learn faster. The linear prose of natural language is actually a terribly lossy way to serialize and transmit the knowledge that is in a person's brain. The receiver must deduce most of it given a small seed.