> The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this k

> The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason. Tiny models are sometimes used out of necessity, but the results often don't hold up at scale, or in different scenarios. If you train a tiny transformer to use a fraction of its capacity to perform an algorithm simple enough that you could design the transformer weights by hand, the results might generalize to natural-language reasoning, but they easily might just not. So it is hard to generalize even to something like GPT-3. But once you get to a modern LLM, the nature of the chain of thought is also very different. First of all, the CoT can be very long and encode reasoning with long chains of serial dependencies. Filler tokens don’t increase the limit on serial computation, particularly in the dots example where the output is a single token (as opposed to real language model which can somewhat substitute for missing CoT by reasoning within the answer itself). Also, a modern LLM gains much of its reasoning capability in post-training, where it learns which CoT tokens to emit rather than just relying on pre-existing chains of thought in its training data. The post-training involves reinforcement learning where the grader ignores the CoT and only grades the final result. The only things tying the CoT to natural language are (1) the starting point (pre-training data) containing natural-language CoT, and (2) the CoT being produced using the same weights as the final output. On one hand this could potentially make the CoT less faithful to the model's reasoning, since nothing directly punishes the model for, say, coming up with its own private meanings for words within the CoT. But on the other hand, this incentivizes the model to make good use of the CoT. To the extent that the CoT is a "parallel construction" (i.e. a chain of reasoning that has to be coherent-looking but differs from how the model is actually coming up with the answer), coming up with this duplicate reasoning is both a waste of computation and a big opportunity cost. The model should learn to instead use the CoT tokens for information that will be useful to it later. But in practice models do keep producing readable CoT, suggesting that the CoT does reflect the real thinking process to a significant extent (probably more so after post-training than before). The dots paper has no post-training, so it doesn’t encounter this dynamic at all.