AFAIK it's still an active area of research, and evidence from Meta AI [0] suggests that size and quality of data can let smaller (not necessarily less sophisti

AFAIK it's still an active area of research, and evidence from Meta AI [0] suggests that size and quality of data can let smaller (not necessarily less sophisticated) models do amazing things. But a lot of the advancements we're seeing right now are the result of more sophisticated models [1], and one person is doing some interesting work [2] around achieving transformer-level performance with other architectures. So it's not completely settled if more data is the answer. But it has a significant impact. [0] https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta... [1] https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_... [2] https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM