3 ms·

People use the term "stochastic parrot" in different ways ... some just as a put-down ("it's just autocomplete"), but others like Geoff Hinton acknowledging tha

People use the term "stochastic parrot" in different ways ... some just as a put-down ("it's just autocomplete"), but others like Geoff Hinton acknowledging that there is of course some truth to it (an LLM is, at the end of the day, a system who's (only) goal is to predict "what would a human say"), while pointing out the depth of "understanding" needed to be a really good at this. There are fundamental limitations to LLMs though - a limit to what can be learned by training a system to predict next word form a fixed training corpus. It can get REALLY good at that task, as we've seen, to extent that it's not just predicting next word but rather predicting an entire continuation/response that is statistically consistent with the training set. However, what is fundamentally missing is any grounding in anything other than the training set, which is the what causes hallucinations/bullshitting. In a biological intelligent system predicting reality is the goal, not just predicting what "sounds good". LLMs are a good start in as much as they prove the power of prediction as a form of feedback, but to match biological systems we need a closed-loop cognitive architecture that can predict then self-correct based on mismatch between reality and prediction (which is what our cortex does). For all of the glib prose that an LLM can generate, even if it seems to understand what you are asking (after all, it was trained with the goal of sounding good), it doesn't have the intelligence of even a simple animal like a rat that doesn't use language at all, but is grounded in reality.

deleted 3y ago
[deleted]
xcv123 3y ago
> even if it seems to understand what you are asking (after all, it was trained with the goal of sounding good It was trained not only to "sound good" aesthetically but also to solve a wide range of NLP tasks accurately. It not only "seems to" understand the prompt but it actually does have a mechanical understanding of it. With ~100 layers in the network it mechanically builds a model of very abstract concepts at the higher layers. > it doesn't have the intelligence of even a simple animal It has higher intelligence than humans by some metrics, but no consciousness.
- HarHarVeryFunny 3y ago
  > It was trained not only to "sound good" aesthetically but also to solve a wide range of NLP tasks accurately. Was it? I've only heard of pre-training (predict next word) and subsequent RLHF + SFT "alignment" (incl. aligning to goal of being conversational). AFAIK the NLP skills that these LLMs achieve are all emergent rather than explicitly trained. I'm not sure we can really say the net fully understands even if it answers as if it does - it was only trained to "predict next word", which in effect means being trained to generate a human-like response. It will have learnt enough to accomplish that goal, and no more (training loss tends to zero as goal is met). Contrast this to an animal with a much richer type of feedback - reality, and with continual (aka online) learning. The animal truly understands it's actions - i.e. has learnt to accurately predict what will happen as a result of them. The LLM does not understand it's own output in this sense - it exists only in a world of words, and has no idea if the ideas it is expressing are true or not (hence all the hallucinating/bullshitting). It only knew enough to generate something that sounded like what a person might say.
  - xcv123 3y ago
    > Was it? I've only heard of pre-training (predict next word) and subsequent RLHF + SFT "alignment" (incl. aligning to goal of being conversational). AFAIK the NLP skills that these LLMs achieve are all emergent rather than explicitly trained. I believe you are right about that. I did some research after reading your comment. Transformers were certainly designed for NLP, but with large enough models the abilities can emerge without necessarily being explicitly trained for it. > I'm not sure we can really say the net fully understands even if it answers as if it does - it was only trained to "predict next word", which in effect means being trained to generate a human-like response. It depends on your definition of "understand". If that requires consciousness then there is no universally agreed formal definition. Natural Language Understanding (NLU) is a subset of Natural Language Processing (NLP). If we take the word "understanding" as used in an academic and technical context then yes they do understand quite well. In order to simply "predict the next word" they learn an abstract model of syntax, semantics, meaning, relationships, etc, from the text. > and has no idea if the ideas it is expressing are true or not (hence all the hallucinating/bullshitting). That is not really an issue when solving tasks that are within it's context window. It is an issue for factual recall. The model is not a type of database that stores its training set verbatim. Humans have analogous problems with long term memory recall. I can think straight within my working memory but my brain will "hallucinate" to some extent when recalling distant memories.

deleted 3y ago

[deleted]

xcv123 3y ago

> even if it seems to understand what you are asking (after all, it was trained with the goal of sounding good It was trained not only to "sound good" aesthetically but also to solve a wide range of NLP tasks accurately. It not only "seems to" understand the prompt but it actually does have a mechanical understanding of it. With ~100 layers in the network it mechanically builds a model of very abstract concepts at the higher layers. > it doesn't have the intelligence of even a simple animal It has higher intelligence than humans by some metrics, but no consciousness.

HarHarVeryFunny 3y ago
> It was trained not only to "sound good" aesthetically but also to solve a wide range of NLP tasks accurately. Was it? I've only heard of pre-training (predict next word) and subsequent RLHF + SFT "alignment" (incl. aligning to goal of being conversational). AFAIK the NLP skills that these LLMs achieve are all emergent rather than explicitly trained. I'm not sure we can really say the net fully understands even if it answers as if it does - it was only trained to "predict next word", which in effect means being trained to generate a human-like response. It will have learnt enough to accomplish that goal, and no more (training loss tends to zero as goal is met). Contrast this to an animal with a much richer type of feedback - reality, and with continual (aka online) learning. The animal truly understands it's actions - i.e. has learnt to accurately predict what will happen as a result of them. The LLM does not understand it's own output in this sense - it exists only in a world of words, and has no idea if the ideas it is expressing are true or not (hence all the hallucinating/bullshitting). It only knew enough to generate something that sounded like what a person might say.
- xcv123 3y ago
  > Was it? I've only heard of pre-training (predict next word) and subsequent RLHF + SFT "alignment" (incl. aligning to goal of being conversational). AFAIK the NLP skills that these LLMs achieve are all emergent rather than explicitly trained. I believe you are right about that. I did some research after reading your comment. Transformers were certainly designed for NLP, but with large enough models the abilities can emerge without necessarily being explicitly trained for it. > I'm not sure we can really say the net fully understands even if it answers as if it does - it was only trained to "predict next word", which in effect means being trained to generate a human-like response. It depends on your definition of "understand". If that requires consciousness then there is no universally agreed formal definition. Natural Language Understanding (NLU) is a subset of Natural Language Processing (NLP). If we take the word "understanding" as used in an academic and technical context then yes they do understand quite well. In order to simply "predict the next word" they learn an abstract model of syntax, semantics, meaning, relationships, etc, from the text. > and has no idea if the ideas it is expressing are true or not (hence all the hallucinating/bullshitting). That is not really an issue when solving tasks that are within it's context window. It is an issue for factual recall. The model is not a type of database that stores its training set verbatim. Humans have analogous problems with long term memory recall. I can think straight within my working memory but my brain will "hallucinate" to some extent when recalling distant memories.