>But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to

>But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing. What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained. You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space. It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum