5 ms·

I agree with everything you say except this: > You can make any modern LLM explain its reasoning You can make any modern LLM create a plausible, self-consiste

I agree with everything you say except this: > You can make any modern LLM explain its reasoning You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".

dist-epoch 2mo ago
This was beautifully shown by asking a model to explain how it added two numbers together (something like 45+21), and it told a plausible story, when in fact they showed it was some rotation on a helix living in some internal manifold. Like asking a human "how did you catch that fast ball coming at you?"
- markasoftware 2mo ago
  It could be that the rotation in the helix manifold whatever is a low level representation of the logical steps (carry the 2, add the next column,...) it's describing. The point stands that the explanation it generates doesn't necessarily in all cases reflect what it "actually did" but your counterexample doesn't hold.
  - dist-epoch 2mo ago
    There have been multiple studies on how LLMs add numbers. They use the "Clock" algorithm, the "Pizza" algorithm, a few other ones. > All networks we study implement the same simple neuron model in their first-layer MLPs: degree-1 sinusoidal fits in layer 1, with deeper layers combining into degree-2 sinusoidal interactions. https://neurips.cc/virtual/2025/loc/san-diego/133808 https://neurips.cc/virtual/2025/loc/san-diego/133808 https://arxiv.org/abs/2502.00873 https://arxiv.org/abs/2502.00873 When you ask them they don't mention these at all, they give you high-school math: https://chatgpt.com/share/6a82afdd-872c-83eb-aad7-622d27f2dcdb https://chatgpt.com/share/6a82afdd-872c-83eb-aad7-622d27f2dc... > did you use the "cos" or "sin" function at all during this addition computation? > No. There is no need for trigonometric functions like sin or cos. The computation only uses basic arithmetic and place-value reasoning. Of course, if someone were implementing arithmetic in a computer, it is theoretically possible to express addition using extremely complicated formulas involving sin and cos. But in the reasoning I described, no trigonometric functions were involved at all. I simply decomposed the numbers into hundreds and smaller parts and added them.
kentonv 2mo ago
Tangent: This is often true of humans as well. We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.
- angry_octet 2mo ago
  And we all know some people that rationalise poor choices and misbehavior, hide their mistakes, etc, to an unacceptable degree. Sometimes the individual knows they are rationalising but continues anyway, other times they seem incapable of seeing that. When you ask people who are rationalising poor behaviour about the scenario, but it is someone else doing it, they may arrive at a better answer. Can we use multiple LLMs to achieve self criticism and critical thinking?
  - efilife 2mo ago
    isn't this happening already? There's the concept called "thinking" where the models talks with itself before giving you the final answer
    - angry_octet 2mo ago
      It does multiple rounds of feedback which is called 'thinking' but whether that is critical thinking is unknown.
- glenstein 2mo ago
  Tangent on the tangent: I think that's true in a minority of cases and in a majority of AI cases. Though in principle I think it should be possible for an LLM to have access to and faithfully represent its own reasoning.
  - margalabargala 2mo ago
    On the contrary, I would argue that it's true in a totality of AI cases. To your point, I agree that nominally there should be a way to give conceptual names to paths of weights, and when answering a question, notice which weights were and were not applied and retrospect on that. That's not what reasoning traces as they currently exist are, though.

dist-epoch 2mo ago

This was beautifully shown by asking a model to explain how it added two numbers together (something like 45+21), and it told a plausible story, when in fact they showed it was some rotation on a helix living in some internal manifold. Like asking a human "how did you catch that fast ball coming at you?"

markasoftware 2mo ago
It could be that the rotation in the helix manifold whatever is a low level representation of the logical steps (carry the 2, add the next column,...) it's describing. The point stands that the explanation it generates doesn't necessarily in all cases reflect what it "actually did" but your counterexample doesn't hold.
- dist-epoch 2mo ago
  There have been multiple studies on how LLMs add numbers. They use the "Clock" algorithm, the "Pizza" algorithm, a few other ones. > All networks we study implement the same simple neuron model in their first-layer MLPs: degree-1 sinusoidal fits in layer 1, with deeper layers combining into degree-2 sinusoidal interactions. https://neurips.cc/virtual/2025/loc/san-diego/133808 https://neurips.cc/virtual/2025/loc/san-diego/133808 https://arxiv.org/abs/2502.00873 https://arxiv.org/abs/2502.00873 When you ask them they don't mention these at all, they give you high-school math: https://chatgpt.com/share/6a82afdd-872c-83eb-aad7-622d27f2dcdb https://chatgpt.com/share/6a82afdd-872c-83eb-aad7-622d27f2dc... > did you use the "cos" or "sin" function at all during this addition computation? > No. There is no need for trigonometric functions like sin or cos. The computation only uses basic arithmetic and place-value reasoning. Of course, if someone were implementing arithmetic in a computer, it is theoretically possible to express addition using extremely complicated formulas involving sin and cos. But in the reasoning I described, no trigonometric functions were involved at all. I simply decomposed the numbers into hundreds and smaller parts and added them.

kentonv 2mo ago

Tangent: This is often true of humans as well. We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.

angry_octet 2mo ago
And we all know some people that rationalise poor choices and misbehavior, hide their mistakes, etc, to an unacceptable degree. Sometimes the individual knows they are rationalising but continues anyway, other times they seem incapable of seeing that. When you ask people who are rationalising poor behaviour about the scenario, but it is someone else doing it, they may arrive at a better answer. Can we use multiple LLMs to achieve self criticism and critical thinking?
- efilife 2mo ago
  isn't this happening already? There's the concept called "thinking" where the models talks with itself before giving you the final answer
  - angry_octet 2mo ago
    It does multiple rounds of feedback which is called 'thinking' but whether that is critical thinking is unknown.
glenstein 2mo ago
Tangent on the tangent: I think that's true in a minority of cases and in a majority of AI cases. Though in principle I think it should be possible for an LLM to have access to and faithfully represent its own reasoning.
- margalabargala 2mo ago
  On the contrary, I would argue that it's true in a totality of AI cases. To your point, I agree that nominally there should be a way to give conceptual names to paths of weights, and when answering a question, notice which weights were and were not applied and retrospect on that. That's not what reasoning traces as they currently exist are, though.