9 ms·

It's language and speech patterns that seem designed to trick readers into believing that claims are correct, even when the claims aren't based on anything and

It's language and speech patterns that seem designed to trick readers into believing that claims are correct, even when the claims aren't based on anything and are possibly wrong. It was rewarded for this during training for some reason. Alternative theory: The LLMs only way to "think" about abstract concepts is through language, and this leaks into into conversation it has with humans. But humans generally prefer to communicate on low levels of abstraction, through a back and forth, until the hard-to-express higher abstraction exists in the head of everyone involved - without ever being directly communicated. This is because we don't think using language. Language is merely a lossy translation of our thought into something expressible, happening after the fact or alongside it. So when the LLM starts speaking to us using patterns and terms it created for itself during training to encode abstract thought in language, communicating with it becomes painful.

pixl97 2mo ago
The term 'language model' throws some people off thinking you can only put english or french, or both into a model. Technically an LLM can learn about anything that can be digitized. If you wanted to spend a billion dollars training one on wireless signals it wouldn't be impossible for it to connect to your router with the right antenna attached. So only limiting it to the idea of language leaves off a lot of other types of abstractions and concepts they encode.
- vrighter 2mo ago
  shhh don't give anyone any ideas, because they'll try it and sell it to management, as idiotic as the idea itself is
  - forshaper 2mo ago
    I have a fantasy that we will do this to turn trees into antennas again, with usable bandwidth.
chrisweekly 2mo ago
"There is a depth of thought untouched by words, and deeper still a depth of formless feeling untouched by thought." - Rilke Your assertion that we don't think in language is questionable. It runs counter to the lived experience of developing thoughts through writing ("writing isn't capturing thinking -- it is thinking"). I believe there is more to thought than language alone, but I also feel quite sure that language forms an essential part of thinking beyond a base layer of instinctive animalistic associations. Sophisticated thoughts are impossible to construct or maintain in the absence of language to represent concepts. Edt to add: I cited Rilke because I find the notion [some deep thoughts are beyond language] interesting. But I disagree with the idea that language is only ever epiphenomenal (co-occurring with thought), or akin to a hard-of-hearing scribe attempting to convey thoughts which always have independent existence.
- deleted 2mo ago
  [deleted]
- JohnBooty 2mo ago
  This is true and it's barely even debatable. Whatever exact role language plays in our thought processes, it is most definitely nonzero. It's why I think "LLMs are only fancy autocorrect" style takes are really underselling how wild it is that we've, in a roundabout way, sort of crystallized a bit of the human thought process in a way that is genuinely useful for a lot of tasks. Linguistic Relativity — John Lucy https://www.annualreviews.org/doi/10.1146/annurev.anthro.26.1.291 https://www.annualreviews.org/doi/10.1146/annurev.anthro.26.... Russian Blues Reveal Effects of Language on Color Discrimination https://www.pnas.org/doi/10.1073/pnas.0701644104 https://www.pnas.org/doi/10.1073/pnas.0701644104 Unconscious Effects of Language-Specific Terminology on Pre-Attentive Color Perception https://www.pnas.org/doi/10.1073/pnas.0811155106 https://www.pnas.org/doi/10.1073/pnas.0811155106 Newly Trained Lexical Categories Produce Lateralized Categorical Perception of Color https://www.pnas.org/doi/10.1073/pnas.1005669107 https://www.pnas.org/doi/10.1073/pnas.1005669107
  - otabdeveloper4 2mo ago
    > sort of crystallized a bit of the human thought process a) LLMs don't think. They predict a most probable sequence of language tokens. Huge difference there. b) Whatever LLMs do doesn't model human behavior whatsoever. LLMs are basically very fancy logistic regressors. I.e., it's a mathematical abstraction first and foremost.
cookiengineer 2mo ago
It's actually an effect that happens in the (re-)alignment process due to harmonic properties of the positional encoding in the attention matrix. (I recommend reading and implementing the Attention is all you need paper. By hand. Otherwise you won't learn anything from it.)
worldthruword 2mo ago
> even when the claims aren't based on anything and are possibly wrong. Are you saying Claude is engaging in Rhetorics because the RL data generated by humans were influenced more by it and persuasion rather than actual logic or reasoning?
- noduerme 2mo ago
  I think that's precisely what they're saying. It shouldn't be a surprise that it's successful. Eliza proved the same thing 35 years ago.
oezi 2mo ago
Maybe we need LLMs which have an internal dialogue rather than the current monologue.
- locknitpicker 2mo ago
  > Maybe we need LLMs which have an internal dialogue rather than the current monologue. We already have them. They are called LLMs. The internal dialogue you speak of are the vectors in the so called latent space.
swiftcoder 2mo ago
> It's language and speech patterns that seem designed to trick readers into believing that claims are correct, even when the claims aren't based on anything and are possibly wrong. When your training set contains more or less the complete output of every capital-C Consulting firm...
deleted 2mo ago
[deleted]
kingkawn 2mo ago
All of these rhetorical devices to make the assertion seem authoritative and correct are derived from academia

pixl97 2mo ago

The term 'language model' throws some people off thinking you can only put english or french, or both into a model. Technically an LLM can learn about anything that can be digitized. If you wanted to spend a billion dollars training one on wireless signals it wouldn't be impossible for it to connect to your router with the right antenna attached. So only limiting it to the idea of language leaves off a lot of other types of abstractions and concepts they encode.

vrighter 2mo ago
shhh don't give anyone any ideas, because they'll try it and sell it to management, as idiotic as the idea itself is
- forshaper 2mo ago
  I have a fantasy that we will do this to turn trees into antennas again, with usable bandwidth.

chrisweekly 2mo ago

"There is a depth of thought untouched by words, and deeper still a depth of formless feeling untouched by thought." - Rilke Your assertion that we don't think in language is questionable. It runs counter to the lived experience of developing thoughts through writing ("writing isn't capturing thinking -- it is thinking"). I believe there is more to thought than language alone, but I also feel quite sure that language forms an essential part of thinking beyond a base layer of instinctive animalistic associations. Sophisticated thoughts are impossible to construct or maintain in the absence of language to represent concepts. Edt to add: I cited Rilke because I find the notion [some deep thoughts are beyond language] interesting. But I disagree with the idea that language is only ever epiphenomenal (co-occurring with thought), or akin to a hard-of-hearing scribe attempting to convey thoughts which always have independent existence.

deleted 2mo ago
[deleted]
JohnBooty 2mo ago
This is true and it's barely even debatable. Whatever exact role language plays in our thought processes, it is most definitely nonzero. It's why I think "LLMs are only fancy autocorrect" style takes are really underselling how wild it is that we've, in a roundabout way, sort of crystallized a bit of the human thought process in a way that is genuinely useful for a lot of tasks. Linguistic Relativity — John Lucy https://www.annualreviews.org/doi/10.1146/annurev.anthro.26.1.291 https://www.annualreviews.org/doi/10.1146/annurev.anthro.26.... Russian Blues Reveal Effects of Language on Color Discrimination https://www.pnas.org/doi/10.1073/pnas.0701644104 https://www.pnas.org/doi/10.1073/pnas.0701644104 Unconscious Effects of Language-Specific Terminology on Pre-Attentive Color Perception https://www.pnas.org/doi/10.1073/pnas.0811155106 https://www.pnas.org/doi/10.1073/pnas.0811155106 Newly Trained Lexical Categories Produce Lateralized Categorical Perception of Color https://www.pnas.org/doi/10.1073/pnas.1005669107 https://www.pnas.org/doi/10.1073/pnas.1005669107
- otabdeveloper4 2mo ago
  > sort of crystallized a bit of the human thought process a) LLMs don't think. They predict a most probable sequence of language tokens. Huge difference there. b) Whatever LLMs do doesn't model human behavior whatsoever. LLMs are basically very fancy logistic regressors. I.e., it's a mathematical abstraction first and foremost.

cookiengineer 2mo ago

It's actually an effect that happens in the (re-)alignment process due to harmonic properties of the positional encoding in the attention matrix. (I recommend reading and implementing the Attention is all you need paper. By hand. Otherwise you won't learn anything from it.)

worldthruword 2mo ago

> even when the claims aren't based on anything and are possibly wrong. Are you saying Claude is engaging in Rhetorics because the RL data generated by humans were influenced more by it and persuasion rather than actual logic or reasoning?

noduerme 2mo ago
I think that's precisely what they're saying. It shouldn't be a surprise that it's successful. Eliza proved the same thing 35 years ago.

oezi 2mo ago

Maybe we need LLMs which have an internal dialogue rather than the current monologue.

locknitpicker 2mo ago
> Maybe we need LLMs which have an internal dialogue rather than the current monologue. We already have them. They are called LLMs. The internal dialogue you speak of are the vectors in the so called latent space.

swiftcoder 2mo ago

> It's language and speech patterns that seem designed to trick readers into believing that claims are correct, even when the claims aren't based on anything and are possibly wrong. When your training set contains more or less the complete output of every capital-C Consulting firm...

deleted 2mo ago

[deleted]

kingkawn 2mo ago

All of these rhetorical devices to make the assertion seem authoritative and correct are derived from academia