4 ms·

What I don’t like about chains of thoughts (2023)

56 points by jxmorris12 11mo ago

deleted 11mo ago
[deleted]
deleted 11mo ago
[deleted]
marcus_holmes 11mo ago
This is Dual Process Theory [0] otherwise known as Fast vs Slow thinking, or System 1 and System 2 thinking. Humans are the only known organism to do System 2 (which doesn't mean we're the only ones that do it, just that we don't know if whales do it), but System 2 is what the author is talking about when they refer to Chains of Thought. System 1 is what they're referring to when they talk about Messi reacting to an unusual situation on the field. Related anecdote: I tested myself for ADHD by taking amphetamines. I normally think by intuitive leaps from point to point, without doing the intermediate steps consciously. I found that during this experience my System 2 thinking was fast enough to follow and I actually experienced proper chains of thought. Or I was whizzing my tits off and hallucinated the whole thing. Not sure yet. I should repeat the experiment. [0] https://en.wikipedia.org/wiki/Dual_process_theory https://en.wikipedia.org/wiki/Dual_process_theory
- protocolture 11mo ago
  Dont know why you are getting the downvotes this is the best comment here.
  - coldtea 11mo ago
    Haven't many of the fast/slow thinking claims, as popularized by the best-selling Daniel Kahneman's book, been debunked?
    - protocolture 11mo ago
      Sounds like great fodder for a just as informative reply.
    - Libidinalecon 11mo ago
      The irony is I am pretty sure you believe this because of some kind of heuristic, but I am not sure what that heuristic would be called. Priming has issues with repeatability in experiments. To believe that "debunks" the entirety of heuristics and bias is completely absurd. The author literally won a Nobel prize for Prospect Theory. I think that book sucks personally. Just an incredibly dull book to read. The online social contagion that it is a work of fiction might actually be more interesting.
      - coldtea 11mo ago
        >The irony is I am pretty sure you believe this because of some kind of heuristic, but I am not sure what that heuristic would be called Maybe we could call it the "if a book bases it claims by citing a few papers, and those specific papers turn out to be bogus, then its claims are unsubstantiated" heuristic
- dns_snek 11mo ago
  > I tested myself for ADHD by taking amphetamines. [...] Or I was whizzing my tits off and hallucinated the whole thing. Not sure yet. You can't test yourself for ADHD by taking amphetamines because they have profound effects on everyone - there's a reason why stimulants are some of the most popular recreational drugs. They can make everyone feel smarter, more productive, like you're on top of the world, especially at recreational doses. When it comes to therapeutic doses for treating ADHD, the effects of the medication should be very subtle, you shouldn't "feel it working" like you've just taken that pill from Limitless. It can produce some euphoria or other recreational effects initially but they're supposed to disappear within a couple of days or weeks, otherwise your dosage is considered to be too high.
- runelk 11mo ago
  > I tested myself for ADHD by taking amphetamines. You cannot test yourself for ADHD - you need to do the actual work that is required for a proper diagnose. And you cannot test yourself for anything at all by taking amphetamines. Seriously. You were using it recreationally. When you post stuff like this online, people that believe what you write could get hurt.
albert_e 11mo ago
> chains of thoughts Pedantic maybe -- but does this need two plurals?
- 4b11b4 11mo ago
  There's lots of typos like this in the article which I find quite nice for some reason
  - jdthedisciple 11mo ago
    Reason being the human touch as opposed to AI one perhaps
    - 4b11b4 11mo ago
      Intentional typos and deliberately misspelling words.. I have thought... is going to have to become a style. Everyone will have their own way of fudging it up. Sort of like the painters who came after the "Realism" era.
      - albert_e 11mo ago
        Interestingly Sam Altman is among those who write without using capital letters -- which LLMs don't do (unless of course specifically prompted to)
tintor 11mo ago
"Obviously no, this IMO proves that we humans can reason efficiently without an inner speech." Well, no, it proves that Messi can reason efficiently without an inner speech.
crystal_revenge 11mo ago
Decoder only LLMs are Markov chains with sophisticated models of the state space. Anyone familiar with Hamiltonian Markov Chains will know that for good results you need a warm up period so that you're sampling from the typical set which is the area with generally the highest probability density in the distribution (not necessary the high propbability/maximum likelihood). I have spent a lot of time experimenting with Chain of Thought professionally and I have yet to see any evidence to suggest that what's happening with CoT is any more (or less) than this. If you let the model run a bit longer it enters a region close to the typical set and when it's ready to answer you have a high probability of getting a good answer. There's absolutely no "reasoning" going on here, except that some times sampling from the typical set near the region of your answer is going to look very similar to how human reason before coming up with an answer.
- 4b11b4 11mo ago
  I wish it got called "scaffolding" instead
  - eru 11mo ago
    When you adopt the probability distribution point of view, this is often called 'burn-in'. See eg the usage in https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_al...
  - xwolfi 11mo ago
    That sounds a lot like bad marketing. Chain of thoughts is better, it makes you think the thing is thinking !
- dhampi 11mo ago
  I don't understand the analogy. If I'm using an MCMC algorithm to sample a probability distribution, I need to wait for my Markov chain to converge to a stationary distribution before sampling, sure. But in no way is 'a good answer' a stationary state in the LLM Markov chain. If I continue running next-token prediction, I'm not going to start looping.
cubefox 11mo ago
Interesting that he came to this conclusion (CoT should be done in latent space) well before the release of OpenAI's o1, which made explicit CoT reliable in the first place. At the time the blog post was written, CoT was only achieved via a "reason step by step" instruction, which was highly error prone compared to modern o1-like reasoning. (And before InstructGPT/ChatGPT, it was achieved by prompting the model with "let me reason step by step".)
- jdthedisciple 11mo ago
  Fair to say he demonstrates good CoT reasoning capabilities?
  - cubefox 11mo ago
    Isn't this a form of intuitive intelligence which doesn't rely on "reasoning"? To me "reasoning" sounds like intentionally trying to solve some explicit problem, while another form of ... insight is the ability to figure out that something is a problem in the first place. That's, by the way, something LLMs are very much not good at. They possess a superhuman amount of knowledge covering all areas of academia, including math, science, philosophy, engineering, computer science, social sciences and so on, but that doesn't cause them to come up with novel hypotheses and theories. Something that would be easy for a smart human even with a fraction of the academic knowledge of an LLM.
AlexCoventry 11mo ago
There's a good deal of work on latent reasoning models, these days. Here is a recent survey paper. https://arxiv.org/abs/2507.06203 https://arxiv.org/abs/2507.06203
moralestapia 11mo ago
Yann LeCun tis saying that some Paul Halpern from X alleges that Albert Einstein (the one from the stories!) had a conversation with some psychologist more than a hundred years ago where those words were uttered. If true, this is somehow meant to be taken as a rule/law that language and thoughts are fundamentally different things. No wonder why he got sacked, lol.

56 points by jxmorris12 11mo ago

deleted 11mo ago

[deleted]

deleted 11mo ago

[deleted]

marcus_holmes 11mo ago

This is Dual Process Theory [0] otherwise known as Fast vs Slow thinking, or System 1 and System 2 thinking. Humans are the only known organism to do System 2 (which doesn't mean we're the only ones that do it, just that we don't know if whales do it), but System 2 is what the author is talking about when they refer to Chains of Thought. System 1 is what they're referring to when they talk about Messi reacting to an unusual situation on the field. Related anecdote: I tested myself for ADHD by taking amphetamines. I normally think by intuitive leaps from point to point, without doing the intermediate steps consciously. I found that during this experience my System 2 thinking was fast enough to follow and I actually experienced proper chains of thought. Or I was whizzing my tits off and hallucinated the whole thing. Not sure yet. I should repeat the experiment. [0] https://en.wikipedia.org/wiki/Dual_process_theory https://en.wikipedia.org/wiki/Dual_process_theory

protocolture 11mo ago
Dont know why you are getting the downvotes this is the best comment here.
- coldtea 11mo ago
  Haven't many of the fast/slow thinking claims, as popularized by the best-selling Daniel Kahneman's book, been debunked?
  - protocolture 11mo ago
    Sounds like great fodder for a just as informative reply.
  - Libidinalecon 11mo ago
    The irony is I am pretty sure you believe this because of some kind of heuristic, but I am not sure what that heuristic would be called. Priming has issues with repeatability in experiments. To believe that "debunks" the entirety of heuristics and bias is completely absurd. The author literally won a Nobel prize for Prospect Theory. I think that book sucks personally. Just an incredibly dull book to read. The online social contagion that it is a work of fiction might actually be more interesting.
    - coldtea 11mo ago
      >The irony is I am pretty sure you believe this because of some kind of heuristic, but I am not sure what that heuristic would be called Maybe we could call it the "if a book bases it claims by citing a few papers, and those specific papers turn out to be bogus, then its claims are unsubstantiated" heuristic
dns_snek 11mo ago
> I tested myself for ADHD by taking amphetamines. [...] Or I was whizzing my tits off and hallucinated the whole thing. Not sure yet. You can't test yourself for ADHD by taking amphetamines because they have profound effects on everyone - there's a reason why stimulants are some of the most popular recreational drugs. They can make everyone feel smarter, more productive, like you're on top of the world, especially at recreational doses. When it comes to therapeutic doses for treating ADHD, the effects of the medication should be very subtle, you shouldn't "feel it working" like you've just taken that pill from Limitless. It can produce some euphoria or other recreational effects initially but they're supposed to disappear within a couple of days or weeks, otherwise your dosage is considered to be too high.
runelk 11mo ago
> I tested myself for ADHD by taking amphetamines. You cannot test yourself for ADHD - you need to do the actual work that is required for a proper diagnose. And you cannot test yourself for anything at all by taking amphetamines. Seriously. You were using it recreationally. When you post stuff like this online, people that believe what you write could get hurt.

albert_e 11mo ago

> chains of thoughts Pedantic maybe -- but does this need two plurals?

4b11b4 11mo ago
There's lots of typos like this in the article which I find quite nice for some reason
- jdthedisciple 11mo ago
  Reason being the human touch as opposed to AI one perhaps
  - 4b11b4 11mo ago
    Intentional typos and deliberately misspelling words.. I have thought... is going to have to become a style. Everyone will have their own way of fudging it up. Sort of like the painters who came after the "Realism" era.
    - albert_e 11mo ago
      Interestingly Sam Altman is among those who write without using capital letters -- which LLMs don't do (unless of course specifically prompted to)

tintor 11mo ago

"Obviously no, this IMO proves that we humans can reason efficiently without an inner speech." Well, no, it proves that Messi can reason efficiently without an inner speech.

crystal_revenge 11mo ago

Decoder only LLMs are Markov chains with sophisticated models of the state space. Anyone familiar with Hamiltonian Markov Chains will know that for good results you need a warm up period so that you're sampling from the typical set which is the area with generally the highest probability density in the distribution (not necessary the high propbability/maximum likelihood). I have spent a lot of time experimenting with Chain of Thought professionally and I have yet to see any evidence to suggest that what's happening with CoT is any more (or less) than this. If you let the model run a bit longer it enters a region close to the typical set and when it's ready to answer you have a high probability of getting a good answer. There's absolutely no "reasoning" going on here, except that some times sampling from the typical set near the region of your answer is going to look very similar to how human reason before coming up with an answer.

4b11b4 11mo ago
I wish it got called "scaffolding" instead
- eru 11mo ago
  When you adopt the probability distribution point of view, this is often called 'burn-in'. See eg the usage in https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_al...
- xwolfi 11mo ago
  That sounds a lot like bad marketing. Chain of thoughts is better, it makes you think the thing is thinking !
dhampi 11mo ago
I don't understand the analogy. If I'm using an MCMC algorithm to sample a probability distribution, I need to wait for my Markov chain to converge to a stationary distribution before sampling, sure. But in no way is 'a good answer' a stationary state in the LLM Markov chain. If I continue running next-token prediction, I'm not going to start looping.

cubefox 11mo ago

Interesting that he came to this conclusion (CoT should be done in latent space) well before the release of OpenAI's o1, which made explicit CoT reliable in the first place. At the time the blog post was written, CoT was only achieved via a "reason step by step" instruction, which was highly error prone compared to modern o1-like reasoning. (And before InstructGPT/ChatGPT, it was achieved by prompting the model with "let me reason step by step".)

jdthedisciple 11mo ago
Fair to say he demonstrates good CoT reasoning capabilities?
- cubefox 11mo ago
  Isn't this a form of intuitive intelligence which doesn't rely on "reasoning"? To me "reasoning" sounds like intentionally trying to solve some explicit problem, while another form of ... insight is the ability to figure out that something is a problem in the first place. That's, by the way, something LLMs are very much not good at. They possess a superhuman amount of knowledge covering all areas of academia, including math, science, philosophy, engineering, computer science, social sciences and so on, but that doesn't cause them to come up with novel hypotheses and theories. Something that would be easy for a smart human even with a fraction of the academic knowledge of an LLM.

AlexCoventry 11mo ago

There's a good deal of work on latent reasoning models, these days. Here is a recent survey paper. https://arxiv.org/abs/2507.06203 https://arxiv.org/abs/2507.06203

moralestapia 11mo ago

Yann LeCun tis saying that some Paul Halpern from X alleges that Albert Einstein (the one from the stories!) had a conversation with some psychologist more than a hundred years ago where those words were uttered. If true, this is somehow meant to be taken as a rule/law that language and thoughts are fundamentally different things. No wonder why he got sacked, lol.