> Of course learning is a kind of search, but discovering structures that are good at learning is the whole game No, you missed the essential. I mentioned sear

> Of course learning is a kind of search, but discovering structures that are good at learning is the whole game No, you missed the essential. I mentioned search in the context of discovery, or in other words expanding knowledge. Training neural nets is also a search for the best parameters that fit the data, but it's secondary. Many architectures work, there have been a thousand variations for the transformer architectures and plenty of RNN-like approaches since 2017 when transformer was invented, and none of them is better than the current one or significantly worse. Also, considering human population, the number of neurons in the brain, synapses and wiring are very different at micro level from person to person, yet we all learn. The difference between the top 5% and bottom 5% humans is small compared with other species, for example. What makes a big difference between people is education, in other words experiences, or training data. To return to the original idea - AI that simply learns to imitate human text is capable only of remixing ideas. But an AI that actively explores can discover novel ideas, like AlphaZero and AlphaTensor. In both these cases search played a major role. So I was generalizing the concept of "search" across many levels of optimization, from protein folding to DNA and human intelligence. Search is essential for progress across the stack. Even network architecture evolves by search - with human researchers.