3 ms·
Here's one of the signals that makes me think quality matters more than quantity: https://twitter.com/karpathy/status/1797313173449764933 https://twitter.com/k
by simonw 2y ago
Here's one of the signals that makes me think quality matters more than quantity:
https://twitter.com/karpathy/status/1797313173449764933 https://twitter.com/karpathy/status/1797313173449764933
> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.
- whimsicalism 2y agoYes, it's true that LLMs will learn a lot faster from higher quality content - but they're still remarkably good at extracting signal from not very high-quality content. Further, you can generally at least bootstrap on shitty data a model that lets you pan through your data for the higher-quality gem.