Yes, it's true that LLMs will learn a lot faster from higher quality content - but they're still remarkably good at extracting signal from not very high-quality

Yes, it's true that LLMs will learn a lot faster from higher quality content - but they're still remarkably good at extracting signal from not very high-quality content. Further, you can generally at least bootstrap on shitty data a model that lets you pan through your data for the higher-quality gem.