4 ms·

The correct conclusion to draw and also to reiterate: LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsew

The correct conclusion to draw and also to reiterate: LLM's do not think, understand, reason, reflect, comprehend and they never shall. I have commented elsewhere but this bears repeating If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model. Then once you have trained the model you could use even more pen and paper to step through the correct prompts to arrive at the answer. All of this would be a completely mechanical process. This really does bear thinking about. It's amazing the results that LLM's are able to acheive. But let's not kid ourselves and start throwing about terms like AGI or emergence just yet. It makes a mechanical process seem magical (as do computers in general). I should add it also makes sense as to why it would, just look at the volume of human knowledge (the training data). It's the training data with the mass quite literally of mankind's knowledge, genius, logic, inferences, language and intellect that does the heavy lifting.

senordevnyc 11mo ago
I’m curious what your mental model is for how human cognition works. Is it any less mechanical in your view?
- andsoitis 11mo ago
  > I’m curious what your mental model is for how human cognition works. Is it any less mechanical in your view? human cognition is not constrained to pattern recognition and prediction of text and symbols.
  - mflkgknr 11mo ago
    [flagged]
  - greenpizza13 11mo ago
    The thesis of "What is Intelligence" is based around intelligence being just that. > Intelligence is the ability to model, predict, and influence one’s future; it can evolve in relation to other intelligences to create a larger symbiotic intelligence. The book is worth a read. But I don't believe it limits the type of intelligence we have to humans, by definition. Then again, I'm only halfway through the book :). [https://mitpress.mit.edu/9780262049955/what-is-intelligence/ https://mitpress.mit.edu/9780262049955/what-is-intelligence/]
    - dap 11mo ago
      It seems obvious to me that "the ability to model, predict, and influence one’s future" is far more general and capable than "constrained to pattern recognition and prediction of text and symbols." How do you conclude that those are the same? I do like that definition because it seems to capture what's different between LLMs and people even when they come up with the same answers. If you give a person a high school physics question about projectile motion, they'll use a mental model that's a combination of explicit physical principles and algebraic equations. They might talk to themselves or use human language to work through it, but one can point to a clear underlying model (principles, laws, and formulas) that are agnostic to the human language they're using to work through them. I realize some people believe (and it could be) that ultimately it really is the same process. Either the LLM does have such a model encoded implicitly in all those numbers or human thought using those principles and formulas is the same kind of statistical walk that the LLM is doing. At the very least, that seems far from clear. This seems reflected in the results like the OP's.
    - andsoitis 11mo ago
      > Intelligence is the ability to model, predict, and influence one’s future LLM's do pattern match and predict on textual symbols. Humans brains pattern match and predict beyond mere text. LLMs also do not learn in the moment, which I would argue is a sign of lack of intelligence.
- encyclopedism 11mo ago
  That's a very difficult question to answer. It's an open problem in academia. To tease out something often it can be useful to approach problems from the opposite end. For example what is NOT the way in which human cognition works? We know how LLM's function, humans certainly do not function in a similar fashion. For one I can reason well enough that next year is 2026 without having most all human literary output fed to me. It's amazing how much the human mind does with so little information.
  - deleted 11mo ago
    [deleted]
- ReplicantMaker 11mo ago
  Human cognition comes bundled with subjective experience. There is no mechanism known, even in principle, that explains the taste of strawberry. We have no justifiable reasons to believe that our cognition is in any way similar to a bunch of matrix multiplications.
- 12_throw_away 11mo ago
  Animal cognition is comprised of many intricate, quasi-redundant, deeply coupled systems that, among other things, can learn, form memories, interact with its environment, and grow. It is not remotely comparable to a computational neural network in any sense except that they both include "neural" in their jargon, albeit to mean vastly different things.
zahlman 11mo ago
> LLM's do not think, understand, reason, reflect, comprehend and they never shall. ... It's amazing the results that LLM's are able to acheive. ... it also makes sense as to why it would, just look at the volume of human knowledge Not so much amazing as bewildering that certain results are possible in spite of a lack of thinking etc. I find it highly counterintuitive that simply referencing established knowledge would ever get the correct answer to novel problems, absent any understanding of that knowledge.
- andsoitis 11mo ago
  > simply referencing established knowledge would ever get the correct answer to novel problems, absent any understanding of that knowledge. What is a concrete example of this?
  - Dilettante_ 11mo ago
    Coding seems like the most prominent example.
    - andsoitis 11mo ago
      Can you tell us more?
      - keeda 11mo ago
        Unless everybody is writing the same code to solve the same exact problems over and over again, by definition LLMs are solving novel problems every time somebody prompts them for code. Sure, the fundamental algorithms and data structures and dependencies would be the same, but they would be composed in novel ways to address unique use-cases, which describes approximately all of software engineering. If you want to define "novel problems" as those requiring novel algorithms and data structures etc, well, how often do humans solve those in their day-to-day coding?
        TingPing 11mo ago
Zambyte 11mo ago
Related research: https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection

senordevnyc 11mo ago

I’m curious what your mental model is for how human cognition works. Is it any less mechanical in your view?

andsoitis 11mo ago
> I’m curious what your mental model is for how human cognition works. Is it any less mechanical in your view? human cognition is not constrained to pattern recognition and prediction of text and symbols.
- mflkgknr 11mo ago
  [flagged]
- greenpizza13 11mo ago
  The thesis of "What is Intelligence" is based around intelligence being just that. > Intelligence is the ability to model, predict, and influence one’s future; it can evolve in relation to other intelligences to create a larger symbiotic intelligence. The book is worth a read. But I don't believe it limits the type of intelligence we have to humans, by definition. Then again, I'm only halfway through the book :). [https://mitpress.mit.edu/9780262049955/what-is-intelligence/ https://mitpress.mit.edu/9780262049955/what-is-intelligence/]
  - dap 11mo ago
    It seems obvious to me that "the ability to model, predict, and influence one’s future" is far more general and capable than "constrained to pattern recognition and prediction of text and symbols." How do you conclude that those are the same? I do like that definition because it seems to capture what's different between LLMs and people even when they come up with the same answers. If you give a person a high school physics question about projectile motion, they'll use a mental model that's a combination of explicit physical principles and algebraic equations. They might talk to themselves or use human language to work through it, but one can point to a clear underlying model (principles, laws, and formulas) that are agnostic to the human language they're using to work through them. I realize some people believe (and it could be) that ultimately it really is the same process. Either the LLM does have such a model encoded implicitly in all those numbers or human thought using those principles and formulas is the same kind of statistical walk that the LLM is doing. At the very least, that seems far from clear. This seems reflected in the results like the OP's.
  - andsoitis 11mo ago
    > Intelligence is the ability to model, predict, and influence one’s future LLM's do pattern match and predict on textual symbols. Humans brains pattern match and predict beyond mere text. LLMs also do not learn in the moment, which I would argue is a sign of lack of intelligence.
encyclopedism 11mo ago
That's a very difficult question to answer. It's an open problem in academia. To tease out something often it can be useful to approach problems from the opposite end. For example what is NOT the way in which human cognition works? We know how LLM's function, humans certainly do not function in a similar fashion. For one I can reason well enough that next year is 2026 without having most all human literary output fed to me. It's amazing how much the human mind does with so little information.
- deleted 11mo ago
  [deleted]
ReplicantMaker 11mo ago
Human cognition comes bundled with subjective experience. There is no mechanism known, even in principle, that explains the taste of strawberry. We have no justifiable reasons to believe that our cognition is in any way similar to a bunch of matrix multiplications.
12_throw_away 11mo ago
Animal cognition is comprised of many intricate, quasi-redundant, deeply coupled systems that, among other things, can learn, form memories, interact with its environment, and grow. It is not remotely comparable to a computational neural network in any sense except that they both include "neural" in their jargon, albeit to mean vastly different things.

zahlman 11mo ago

> LLM's do not think, understand, reason, reflect, comprehend and they never shall. ... It's amazing the results that LLM's are able to acheive. ... it also makes sense as to why it would, just look at the volume of human knowledge Not so much amazing as bewildering that certain results are possible in spite of a lack of thinking etc. I find it highly counterintuitive that simply referencing established knowledge would ever get the correct answer to novel problems, absent any understanding of that knowledge.

andsoitis 11mo ago
> simply referencing established knowledge would ever get the correct answer to novel problems, absent any understanding of that knowledge. What is a concrete example of this?
- Dilettante_ 11mo ago
  Coding seems like the most prominent example.
  - andsoitis 11mo ago
    Can you tell us more?
    - keeda 11mo ago
      Unless everybody is writing the same code to solve the same exact problems over and over again, by definition LLMs are solving novel problems every time somebody prompts them for code. Sure, the fundamental algorithms and data structures and dependencies would be the same, but they would be composed in novel ways to address unique use-cases, which describes approximately all of software engineering. If you want to define "novel problems" as those requiring novel algorithms and data structures etc, well, how often do humans solve those in their day-to-day coding?
      - TingPing 11mo ago

Zambyte 11mo ago

Related research: https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection