This is how I imagine the human brain works. In principle we think iteratively, do one small computation, memorize the result, then do the next step based on t

This is how I imagine the human brain works. In principle we think iteratively, do one small computation, memorize the result, then do the next step based on the input and the result from the previous step. If things get too complicated, we grab some paper and write down intermediate results on external storage. But fundamentally we memorize some raw facts and otherwise learn algorithms that allow us doing new things by combining existing abilities. Ignoring memory limitations humans are Turing-complete. As we repeatedly solve similar tasks, we develop optimizations, the brain learns to perform a couple of steps in one go instead of going through all the individual steps. When children learn to read, to write or to do math, they learn to do it in small steps, they draw and recognize letters and numbers by looking at and drawing individual lines and circles. When they learn to draw lines or circles they first make very deliberate movements with the pen - up, down, left, right - before drawing a circle becomes a primitive. When they learn to add, they break numbers down into digits and only memorize how to add those, larger numbers are added iteratively digit by digit. With time and practice the brain learns to do many of those things in one step, you read and write letters or words, not deliberately draw individual strokes. You will still fall back to this if you have to deal with characters from a unknown alphabet. You will no longer perform additions digit by digit for small or commonly used numbers but you will fall back to this for big numbers. When you learn to drive, you first very deliberately look at traffic signs, think about turning the steering wheel this way or that way while parking or think about switching on the turn signals. Again with practice many of those things become automatic and you no longer have to think about them. So overall I think the brain is just following sequential algorithms, combining primitives into more complex result. Then with practice common tasks get reimplemented in some kind of fixed function unit that can do a specific task - maybe under some limited circumstances and maybe only approximately - in one step and this can then be used as a new primitive where appropriate. Actually those fixed function units could internally use iteration but this iterative process would be hidden from the primary loop. Language models like ChatGPT seem to be more on the fixed function side, they see a set of tokens and spit out a response. The attention mechanism has some similarity with a loop executing an algorithm step by step, it just has a fixed number of iterations and has been unrolled. When prompted to reason in steps, this adds another loop with additional storage for intermediate results. The difference is that this process is externally visible and that the model must be instructed to do so instead of the model deciding on its own to solve the problem in steps and hiding those steps from the outside. For questions like adding or multiplying numbers or finding the parity of some long bit string the model has some neurons that could directly provide an answer and the answer will probably be correct for simple tasks but the longer the input the more likely it becomes that those neurons will not produce the correct answer. And the model does not know in which region those neurons can be trusted. But the model also knows the underlying algorithm to solve the task, say computing the parity of a long bit string or adding large numbers, and where the steps only involve computations that the model can perform correctly, i.e. adding two digits or XORing two bits. So you can get the correct result out of a model like ChatGPT if you can make it follow the iterative algorithm instead of it trying to give the sum of two 20 digit numbers directly. And I would guess you could - at least to some extend - use ChatGPT itself to rewrite and iteratively solve harder problems. The primary obstacle is probably that it is hard to tell what tasks the model can reliably perform correctly and what tasks must be broken down further.