My guess is that by testing and introspecting a lot of small models trained on datasets and distilled and much as possible, there will eventually be a good unde

My guess is that by testing and introspecting a lot of small models trained on datasets and distilled and much as possible, there will eventually be a good understanding of the types of neural circuits that are effective for different types of representations or tasks. (Although presumably there is more than one way to accomplish any given task or representation efficiently.) I bet the hard part is creating compressed and decomposed representations that really encompass all of the common edge cases. And maintaining a good compression ratio without having conflicting uses for the same neuron. But with small networks, it seems like there will be a way to fully analyze, for example, how language is grounded in visual and spatial-temporal data. Maybe we can use another network to help come up with circuit factorings and connections for specific use cases given a Q&A or conversation dataset.