4 ms·

I think the major lessons from the success of LLMs are two: 1) the astonishing power of a largely trivial association engine based only on the semantic categori

I think the major lessons from the success of LLMs are two: 1) the astonishing power of a largely trivial association engine based only on the semantic categories inferred by word2vec, and 2) that so much of the communication abilities of the human mind require so little rational thought (since LLMs demonstrate essentially none of the skills in Kahneman's and Tversky's System 2 thinking (logic, circumspection, self-correction, reflection, etc). I guess this also disproves Minsky's 'Society of Mind' conjecture - a large part of human cognition (System 1) does not require the complex interaction of heterogeneous mental components.

GeneralMayhem 2y ago
What makes this tough is that LLMs can show logical thinking and self-correction when specifically prompted (e.g. "think step by step", "double-check and then correct your work"). It seems unlikely that they can truthfully self-reflect, but I don't think it's strictly impossible.
- deleted 2y ago
  [deleted]
- Terr_ 2y ago
  > LLMs can show logical thinking and self-correction The same way they "show" sadness or contrition or excitement? We need to be careful with our phrasing here: LLMs can be prompted to provide you associated phrases that usually seem to fit with the rest of the word-soup, but whether the model is actually demonstrating "logical thinking" or "self-correction" is a Chinese Room problem [0]. (Or else a "No, it doesn't, I can tell because I checked the code.") [0] https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
autoexec 2y ago
> that so much of the communication abilities of the human mind require so little rational thought Beyond that fact that human minds created them, I doubt that LLMs can tell us anything about the abilities of the human mind or what language in humans requires. The most we can learn from LLMs about ourselves will be in how we react to them, or more broadly, what the datasets LLMs use show about who we are.

GeneralMayhem 2y ago

What makes this tough is that LLMs can show logical thinking and self-correction when specifically prompted (e.g. "think step by step", "double-check and then correct your work"). It seems unlikely that they can truthfully self-reflect, but I don't think it's strictly impossible.

deleted 2y ago
[deleted]
Terr_ 2y ago
> LLMs can show logical thinking and self-correction The same way they "show" sadness or contrition or excitement? We need to be careful with our phrasing here: LLMs can be prompted to provide you associated phrases that usually seem to fit with the rest of the word-soup, but whether the model is actually demonstrating "logical thinking" or "self-correction" is a Chinese Room problem [0]. (Or else a "No, it doesn't, I can tell because I checked the code.") [0] https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

autoexec 2y ago

> that so much of the communication abilities of the human mind require so little rational thought Beyond that fact that human minds created them, I doubt that LLMs can tell us anything about the abilities of the human mind or what language in humans requires. The most we can learn from LLMs about ourselves will be in how we react to them, or more broadly, what the datasets LLMs use show about who we are.