4 ms·

Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to

by layer8 1y ago

Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to come up with the insight or solution? For all we know, it’s an unconscious process. Would we call it “thinking”? The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking. When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all. I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.

bithead 1y ago
Do LLMs ever ask for you to clarify something you said in a way a person who doesn't quite understand what you said will do?
- willmarch 1y ago
  Yes, often
- savolai 1y ago
  Yeah, as someone who has gained a lot of interaction skills by playing with the constructivist learning ennvironment called the enneagram, I can attest that it much resembles behaviour characteristic of certain enneatypes.
- brabel 1y ago
  What now, two minutes using one and you are going to get that!
- Workaccount2 1y ago
  Sometimes I think people leveraging criticisms of LLMs used ChatGPT 3 years ago and haven't touched one since, except for asking how many r's are in strawberry a year and a half ago.
creer 1y ago
Interesting, you think the associations your brain comes up with during sleep are NOT thinking?
- keiferski 1y ago
  I assume most people agree with that framing. “Thinking” generally implies a level of consciousness or self-awareness. “Let me think about this.” “I have to think on it.” My brain regulates all sorts of processes unconsciously, like breathing, for example. I don’t treat those as “thinking,” so I don’t know why other unconscious brain activity would be either.
  - creer 1y ago
    I don't know that most people think about that at all. For me, I have learned not to dismiss the remuninating, reordering, reframing that my brain performs without me deliberately reasoning line by line about it. Limiting my brain to deliberate line by line reasoning would be very limiting and costly. "Thinking" to me is very much NOT just conscious reasoning. So much of what I think is not done consciously. Indeed "let me think about it" is often simply giving my brain time to "sit on it", for another expression - only after which will I have enough mind time on the various alternatives for a worthwhile conscious decision.
- madaxe_again 1y ago
  I’d say it’s more like annealing. Sort, filter, compress, integrate. Updates your models for the next morning, which is why the answer is there when it wasn’t before.
petralithic 1y ago
> Would we call it “thinking”? Yes I would.
AnIrishDuck 1y ago
> Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility. > When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all. One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1]. I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas. As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence. 1. https://en.wikipedia.org/wiki/Why_We_Sleep https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/ https://www.aiweirdness.com/
madaxe_again 1y ago
Perhaps this is an artefact of instantiation - when you talk with an LLM, the responding instance is just that - it comes into being, inhales your entire chat history, and then continues like the last chap, finishes its response, and dies. The continuity is currently an illusion.
lordnacho 1y ago
> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all. Much like speaking to a less experienced colleague, no? They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.
popalchemist 1y ago
There is simply put no ongoing process and no feedback loop. The model does not learn. The cognition ends when the inference cycle ends. It's not thinking, it just produces output that looks similar to the output of thinking. But the process by which it does that is wholly unreleated.
- ako 1y ago
  Most AI tooling is shipped with a feedback loop around the LLM. The quality of Claude Code for example lies in the feedback loop it provides on your code. Maybe the LLM itself isn't thinking, but the Agent which ships an LLM plus feedback loop definitely shows thinking qualities. Just now in an debugging session with claude code: * let me read this file... * let me read this file... * I think there's a caching issue with the model after dropping the module. Let me check if there's a save or reload needed after DROP MODULE. First, let me verify something: * creates a bash/javascript script to verify its assumption * runs the script (after review and approval) * Aha! I found the problem! Look at the output... How is this not thinking?
  - popalchemist 1y ago
    without getting into theory of mind it's a bit difficult to elaborate, and I don't have the time or the will for that. But the short version is that thinking is interconnected with BEING as well as will, and the Agent has neither, in a philosophically formal sense. The agent is deterministically bound. So it is a fancy Rube Goldberg machine that outputs letters in a way that creates the impression of thought, but it is not thought, in the same way that some birds can mimic human speech without even the slightest hint as to the words' or sentences' meaning, underlying grammar, connotations, subtext, context, intended use, likely effect, etc. Is speech speech if the speaker has no concept whatsoever of said speech's content, and can not use it to actualize itself? I'd say no. It's mimicry, but not speech. So that means speech is something more than just its outward aspect - the words. It is the relation of something invisible, some inner experience known only to the speaker, VIA the words. Whereas a gorilla who learns sign language to communicate and use that communication to achieve aims which have direct correlation with its sense of self - that's thought in the Cogito, Ergo Sum sense of the word. Thought as commonly concieved by the layman is a sort of isolated phenomenon that is mechanical in nature and can be judged by its outward effects; whereas in the philosophical tradition defining thought is known to be one of the hard questions for its mysterious qualia of being interconnected with will and being as described above. Guess I gave you the long answer. (though, really, it could be much longer than this.) The Turing Test touches on this distinction between the appearance of thought and actual thought. The question goes all the way down to metaphysics; some (such as myself) would say that one must be able to define awareness (what some call consciousness - though I think that term is too loaded) before you can define thought. In fact that is at the heart of the western philosophical tradition; and the jury consensus remains elusive after all these thousands of years.

by layer8 1y ago

bithead 1y ago

Do LLMs ever ask for you to clarify something you said in a way a person who doesn't quite understand what you said will do?

willmarch 1y ago
Yes, often
savolai 1y ago
Yeah, as someone who has gained a lot of interaction skills by playing with the constructivist learning ennvironment called the enneagram, I can attest that it much resembles behaviour characteristic of certain enneatypes.
brabel 1y ago
What now, two minutes using one and you are going to get that!
Workaccount2 1y ago
Sometimes I think people leveraging criticisms of LLMs used ChatGPT 3 years ago and haven't touched one since, except for asking how many r's are in strawberry a year and a half ago.

creer 1y ago

Interesting, you think the associations your brain comes up with during sleep are NOT thinking?

keiferski 1y ago
I assume most people agree with that framing. “Thinking” generally implies a level of consciousness or self-awareness. “Let me think about this.” “I have to think on it.” My brain regulates all sorts of processes unconsciously, like breathing, for example. I don’t treat those as “thinking,” so I don’t know why other unconscious brain activity would be either.
- creer 1y ago
  I don't know that most people think about that at all. For me, I have learned not to dismiss the remuninating, reordering, reframing that my brain performs without me deliberately reasoning line by line about it. Limiting my brain to deliberate line by line reasoning would be very limiting and costly. "Thinking" to me is very much NOT just conscious reasoning. So much of what I think is not done consciously. Indeed "let me think about it" is often simply giving my brain time to "sit on it", for another expression - only after which will I have enough mind time on the various alternatives for a worthwhile conscious decision.
madaxe_again 1y ago
I’d say it’s more like annealing. Sort, filter, compress, integrate. Updates your models for the next morning, which is why the answer is there when it wasn’t before.

petralithic 1y ago

> Would we call it “thinking”? Yes I would.

AnIrishDuck 1y ago

> Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility. > When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all. One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1]. I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas. As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence. 1. https://en.wikipedia.org/wiki/Why_We_Sleep https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/ https://www.aiweirdness.com/

madaxe_again 1y ago

Perhaps this is an artefact of instantiation - when you talk with an LLM, the responding instance is just that - it comes into being, inhales your entire chat history, and then continues like the last chap, finishes its response, and dies. The continuity is currently an illusion.

lordnacho 1y ago

> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all. Much like speaking to a less experienced colleague, no? They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.

popalchemist 1y ago

There is simply put no ongoing process and no feedback loop. The model does not learn. The cognition ends when the inference cycle ends. It's not thinking, it just produces output that looks similar to the output of thinking. But the process by which it does that is wholly unreleated.

ako 1y ago
Most AI tooling is shipped with a feedback loop around the LLM. The quality of Claude Code for example lies in the feedback loop it provides on your code. Maybe the LLM itself isn't thinking, but the Agent which ships an LLM plus feedback loop definitely shows thinking qualities. Just now in an debugging session with claude code: * let me read this file... * let me read this file... * I think there's a caching issue with the model after dropping the module. Let me check if there's a save or reload needed after DROP MODULE. First, let me verify something: * creates a bash/javascript script to verify its assumption * runs the script (after review and approval) * Aha! I found the problem! Look at the output... How is this not thinking?
- popalchemist 1y ago
  without getting into theory of mind it's a bit difficult to elaborate, and I don't have the time or the will for that. But the short version is that thinking is interconnected with BEING as well as will, and the Agent has neither, in a philosophically formal sense. The agent is deterministically bound. So it is a fancy Rube Goldberg machine that outputs letters in a way that creates the impression of thought, but it is not thought, in the same way that some birds can mimic human speech without even the slightest hint as to the words' or sentences' meaning, underlying grammar, connotations, subtext, context, intended use, likely effect, etc. Is speech speech if the speaker has no concept whatsoever of said speech's content, and can not use it to actualize itself? I'd say no. It's mimicry, but not speech. So that means speech is something more than just its outward aspect - the words. It is the relation of something invisible, some inner experience known only to the speaker, VIA the words. Whereas a gorilla who learns sign language to communicate and use that communication to achieve aims which have direct correlation with its sense of self - that's thought in the Cogito, Ergo Sum sense of the word. Thought as commonly concieved by the layman is a sort of isolated phenomenon that is mechanical in nature and can be judged by its outward effects; whereas in the philosophical tradition defining thought is known to be one of the hard questions for its mysterious qualia of being interconnected with will and being as described above. Guess I gave you the long answer. (though, really, it could be much longer than this.) The Turing Test touches on this distinction between the appearance of thought and actual thought. The question goes all the way down to metaphysics; some (such as myself) would say that one must be able to define awareness (what some call consciousness - though I think that term is too loaded) before you can define thought. In fact that is at the heart of the western philosophical tradition; and the jury consensus remains elusive after all these thousands of years.