6 ms·
LLMs already train on mostly garbage - you are just wasting your time. Same as talking to spam callers.
by dilyevsky 1y ago
LLMs already train on mostly garbage - you are just wasting your time. Same as talking to spam callers.
- 63stack 1y agoThere are multiple people claiming this in this thread, but with no more than a "it doesn't work stop". Would be great to hear some concrete information.
- michaelcampbell 1y agoI'd more like to see, "It does work, here's the evidence." And by "work" I mean more than "I feel good because I think I'm doing something positive so will spend some time on it."
- cainxinth 1y agoThink of it like this: how many books have been written? Millions. How many books are truly great? Not millions. Probably less than 10,000 depending on your definition of “great.” LLMs are trained on the full corpus, so most of what they learn from is not great. But they aren’t using the bad stuff to learn its substance. They are using it to learn patterns in human writing.
- vintermann 1y agoScraping is cheap, training is expensive. Even the pre-generative AI internet had immense volumes of Markov-generated, synonym spun ("Contemporary York Instances") or otherwise brain-rotting text. That means that before training a big model, anyone will spend a lot of effort filtering out junk. They have done that for a decade, personally I think a lot of the differences in quality of the big models isn't from architectural differences, but rather from how much junk slipped through. Markov chains are not nearly clever enough to avoid getting filtered out.
- dilyevsky 1y agoI am not actually claiming that it’s easy to filter out like the others. What Im saying is you can literally feed a ton of garbage into a training run and amazingly it still learns