6 ms·

> That is, language models learn the implicit meaning in text, as opposed to the early belief some researchers held that sequence-to-sequence models (includin

> That is, language models learn the implicit meaning in text, as opposed to the early belief some researchers held that sequence-to-sequence models (including transformers) simply fit correlations between sequential words. Is this so, that the research community is agreed? Are there papers discussing this topic?

wavemode 2y ago
My sense has always been that, there actually is no difference between "the implicit meaning in text" and "correlations between sequential words". That is to say, the fact that LLMs are able to communicate effectively with humans is a discovery about the regularity of the semantics of human communication, rather than a discovery about the intelligence of neural networks.
- naasking 2y ago
  Agreed: semantics boils down to a relational network between words that designate concepts, therefore large language models build a relational network of concepts. Meta just published large concept models which builds on this: https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/ https://ai.meta.com/research/publications/large-concept-mode...
mjburgess 2y ago
This is certainly not agreed. Computer scientists here don't even have a theory of meaning, because it isn't part of the discipline, nor do almost any have any prior research background in it -- hence making these sort of outrageous claims all over the place. However you want to give natural language semantics, ML models certainly to not use this semantics. The very best that might be said is that the correlational structure of words under transformer-like supervision (ie., where "predict the next word" is the goal) produces a distribution which is an extremely approximate model of natural language semantics. Though this has never been disputed. The question comes down to what kind of extreme approximation is involved. Eg., the truth conditions for "I have a pen in my hand" are that I have a pen in my hand -- direct access to these truth conditions is very plausibly necessary to mean "I have a pen in my hand" in the relevant context. Since a machine has no access to the truth conditions of such utterances it cannot possibly mean them. Thus if a machine manages to say, "I have a pen in my hand" at an appropriate occasion -- the "extreme approximation to natural language semantics" has to do with this occasion and what "appropriateness" means. Critics of LLMs and "computer-science-addled thinking" about such matters (such as myself) would say that there are a very narrow range of "occasions" (ie., situations in which you're prompting) that allow such responses to seem appropraite. That a response seems appropriate to a user is a good engineering condition on a tool working -- it has nothing to do with whether a model understands natural language semantics. What we might say is that it approximates conversations between agents who understand such semantics on a narrow range of occasions, and succeeds in modelling appropriate language use. And so you might call LLMs models of 'average appropriateness of replies'. It obviously does not, nor cannot mean, "I have a pen in my hand"
- idiotsecant 2y ago
  In what way do you have access to the truth conditions for 'I have a pen in my hand' that an LLM can not? This smells circular to me.
  - deleted 2y ago
    [deleted]
  - mjburgess 2y ago
    Well, by having a hand, and having a pen in it
    - idiotsecant 2y ago
      I see. Your sense of sight is in some sense 'true' in a way that a webcam feed is not?
      - mjburgess 2y ago
        Well if you can show me an LLM responding to it's having a pen in its hand via a robotic hand, webcam and the like -- then we are at the bare minimum for it possibly meaning, "i have a pen in my hand". No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) -- since, at least, no such corpus exists
        magicalhippo 2y ago
        > No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) Seems we might not be that far away given the work on action tokens[1] and such. NVIDIA divoted a lot of the CES presentation to this kind of stuff[2]. [1]: https://arxiv.org/abs/2403.19578 https://arxiv.org/abs/2403.19578 (semi-random example) [2]: https://www.nvidia.com/en-us/ai/cosmos/ https://www.nvidia.com/en-us/ai/cosmos/
        idiotsecant 2y ago
jbarrow 2y ago
Research community is definitely not agreed on this, and there are a number of different camps. Broadly, 2 perspectives from the NLP community: The 2020 Bender and Koller paper[1] that argues that meaning is not learnable from form, and LLMs are trained on form. They propose a thought experiment ("The Octopus Test" section of the paper) featuring an octopus that can intercept the conversation two humans are having, but "having only form available as training data, [the Octopus] did not learn meaning." And a contradicting response from Yoav Goldberg (another NLP researcher)[2] with a much more informal discussion of "groundedness" and what LLMs learn. His argument is broadly that instruction tuning + post-training can meaningfully grounds terms like "summarize" etc. [1] https://aclanthology.org/2020.acl-main.463/ https://aclanthology.org/2020.acl-main.463/ [2] https://gist.github.com/yoavg/59d174608e92e845c8994ac2e234c8a9 https://gist.github.com/yoavg/59d174608e92e845c8994ac2e234c8...
- naasking 2y ago
  > The 2020 Bender and Koller paper[1] that argues that meaning is not learnable from form, and LLMs are trained on form. They propose a thought experiment ("The Octopus Test" section of the paper) featuring an octopus that can intercept the conversation two humans are having, but "having only form available as training data, [the Octopus] did not learn meaning." This is just Searle's Chinese Room, and it's obviously false. How can we know it's false? Because there is no "meaning" in the standard model of particle physics (all interactions are by "form"/syntax), and therefore all humans must learn meaning from "form" as well.

wavemode 2y ago

My sense has always been that, there actually is no difference between "the implicit meaning in text" and "correlations between sequential words". That is to say, the fact that LLMs are able to communicate effectively with humans is a discovery about the regularity of the semantics of human communication, rather than a discovery about the intelligence of neural networks.

naasking 2y ago
Agreed: semantics boils down to a relational network between words that designate concepts, therefore large language models build a relational network of concepts. Meta just published large concept models which builds on this: https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/ https://ai.meta.com/research/publications/large-concept-mode...

mjburgess 2y ago

This is certainly not agreed. Computer scientists here don't even have a theory of meaning, because it isn't part of the discipline, nor do almost any have any prior research background in it -- hence making these sort of outrageous claims all over the place. However you want to give natural language semantics, ML models certainly to not use this semantics. The very best that might be said is that the correlational structure of words under transformer-like supervision (ie., where "predict the next word" is the goal) produces a distribution which is an extremely approximate model of natural language semantics. Though this has never been disputed. The question comes down to what kind of extreme approximation is involved. Eg., the truth conditions for "I have a pen in my hand" are that I have a pen in my hand -- direct access to these truth conditions is very plausibly necessary to mean "I have a pen in my hand" in the relevant context. Since a machine has no access to the truth conditions of such utterances it cannot possibly mean them. Thus if a machine manages to say, "I have a pen in my hand" at an appropriate occasion -- the "extreme approximation to natural language semantics" has to do with this occasion and what "appropriateness" means. Critics of LLMs and "computer-science-addled thinking" about such matters (such as myself) would say that there are a very narrow range of "occasions" (ie., situations in which you're prompting) that allow such responses to seem appropraite. That a response seems appropriate to a user is a good engineering condition on a tool working -- it has nothing to do with whether a model understands natural language semantics. What we might say is that it approximates conversations between agents who understand such semantics on a narrow range of occasions, and succeeds in modelling appropriate language use. And so you might call LLMs models of 'average appropriateness of replies'. It obviously does not, nor cannot mean, "I have a pen in my hand"

idiotsecant 2y ago
In what way do you have access to the truth conditions for 'I have a pen in my hand' that an LLM can not? This smells circular to me.
- deleted 2y ago
  [deleted]
- mjburgess 2y ago
  Well, by having a hand, and having a pen in it
  - idiotsecant 2y ago
    I see. Your sense of sight is in some sense 'true' in a way that a webcam feed is not?
    - mjburgess 2y ago
      Well if you can show me an LLM responding to it's having a pen in its hand via a robotic hand, webcam and the like -- then we are at the bare minimum for it possibly meaning, "i have a pen in my hand". No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) -- since, at least, no such corpus exists
      - magicalhippo 2y ago
        > No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) Seems we might not be that far away given the work on action tokens[1] and such. NVIDIA divoted a lot of the CES presentation to this kind of stuff[2]. [1]: https://arxiv.org/abs/2403.19578 https://arxiv.org/abs/2403.19578 (semi-random example) [2]: https://www.nvidia.com/en-us/ai/cosmos/ https://www.nvidia.com/en-us/ai/cosmos/
      - idiotsecant 2y ago

jbarrow 2y ago

Research community is definitely not agreed on this, and there are a number of different camps. Broadly, 2 perspectives from the NLP community: The 2020 Bender and Koller paper[1] that argues that meaning is not learnable from form, and LLMs are trained on form. They propose a thought experiment ("The Octopus Test" section of the paper) featuring an octopus that can intercept the conversation two humans are having, but "having only form available as training data, [the Octopus] did not learn meaning." And a contradicting response from Yoav Goldberg (another NLP researcher)[2] with a much more informal discussion of "groundedness" and what LLMs learn. His argument is broadly that instruction tuning + post-training can meaningfully grounds terms like "summarize" etc. [1] https://aclanthology.org/2020.acl-main.463/ https://aclanthology.org/2020.acl-main.463/ [2] https://gist.github.com/yoavg/59d174608e92e845c8994ac2e234c8a9 https://gist.github.com/yoavg/59d174608e92e845c8994ac2e234c8...

naasking 2y ago
> The 2020 Bender and Koller paper[1] that argues that meaning is not learnable from form, and LLMs are trained on form. They propose a thought experiment ("The Octopus Test" section of the paper) featuring an octopus that can intercept the conversation two humans are having, but "having only form available as training data, [the Octopus] did not learn meaning." This is just Searle's Chinese Room, and it's obviously false. How can we know it's false? Because there is no "meaning" in the standard model of particle physics (all interactions are by "form"/syntax), and therefore all humans must learn meaning from "form" as well.