4 ms·
This reminds me a little bit of the early 2000's where search engines would list the number of indexed pages on their homepage. For language models, does large
by xnx 5y ago
This reminds me a little bit of the early 2000's where search engines would list the number of indexed pages on their homepage. For language models, does large = good? I'm guessing the quality of the corpus matters as much.
- bpiche 5y agoI'm not sure it moves the needle on NLU/classification tasks very much, compared to models with many fewer parameters. But it does seem to make the NLG better, which is what Microsoft seems obsessed with lately.
- thunderbird120 5y ago>For language models, does large = good? The short answer is yes, the long answer is it's complicated. You could actually think of these models as a type of indexer because, at their heart, what they are doing is memorizing the training data and storing it in such a way that incomplete samples can be used as keys to extract complete samples. The magic happens because the models themselves (even the 100+ billion parameter ones) are nowhere near complex enough to actually store all of these possible key value pairs. Instead, the model has to compress its representation of the data which leads to generalization. Larger models can model more complexities which leads to better performance as long as your training dataset is sufficiently large and varied.
- ewheeler 5y agoMaybe? The Scaling Hypothesis[1] suggests that greater capabilities of intelligence may emerge from scaling up 'scalable architectures' to large sizes. GPT-3 exhibits 'meta-learning' capabilities that GPT-2 did not (like learning how to sum numbers)--probably just because its a 100x larger version of GPT-2. [1] https://www.gwern.net/Scaling-hypothesis https://www.gwern.net/Scaling-hypothesis