3 ms·
That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room
by naasking 3y ago
That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room for improvement.
- famouswaffles 3y agoIt doesn't really suggest anything. Neither model will even close to saturation and all else equal, bigger models perform better in every way, including generalization.
- naasking 3y agoBut why do bigger models perform better? Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.
- famouswaffles 3y ago>But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing. What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained. You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space. It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum