That sounds about right. There is definitely nothing in the _design_ of the model or in the training procedure that is already and by itself intelligent, instea

That sounds about right. There is definitely nothing in the _design_ of the model or in the training procedure that is already and by itself intelligent, instead it is an intelligence extraction process. The intelligence is in the data and some of that apparently gets distilled into the model. So we should stop thinking that the intelligence is the (untrained) brain and realize that it has always been the text. I think it makes more sense to compare the current models to unconscious thought or intuition. So the result just kind of pops up as a result of a computation, just like unconscious thought pops up for humans as a result of a (hidden) computation. Humans do conscious reasoning by bringing their thoughts into memory and keeping them there, stretching the process out over time. In some sense LLMs being able to solve some things by being instructed to think step by steps reminds me of that. So perhaps adding some rumination will create a layer on top that appears closer to what we think of as thinking.