4 ms·
> People tend to over-estimate the value of their data for training. AI labs are constantly looking for new sources of high quality data - but quality really ma
by whimsicalism 2y ago
> People tend to over-estimate the value of their data for training. AI labs are constantly looking for new sources of high quality data - but quality really matters to them. Random junk people feed into the models is right at the bottom of that quality list.
I do not think this is at all accurate. Sure quality is increasingly important, but that is just the pendulum shifting only slightly back from the fact that quantity is the primary thing you need.
- simonw 2y agoHere's one of the signals that makes me think quality matters more than quantity: https://twitter.com/karpathy/status/1797313173449764933 https://twitter.com/karpathy/status/1797313173449764933 > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.
- whimsicalism 2y agoYes, it's true that LLMs will learn a lot faster from higher quality content - but they're still remarkably good at extracting signal from not very high-quality content. Further, you can generally at least bootstrap on shitty data a model that lets you pan through your data for the higher-quality gem.