4 ms·
There are multiple people claiming this in this thread, but with no more than a "it doesn't work stop". Would be great to hear some concrete information.
by 63stack 11mo ago
There are multiple people claiming this in this thread, but with no more than a "it doesn't work stop". Would be great to hear some concrete information.
- michaelcampbell 11mo agoI'd more like to see, "It does work, here's the evidence." And by "work" I mean more than "I feel good because I think I'm doing something positive so will spend some time on it."
- cainxinth 11mo agoThink of it like this: how many books have been written? Millions. How many books are truly great? Not millions. Probably less than 10,000 depending on your definition of “great.” LLMs are trained on the full corpus, so most of what they learn from is not great. But they aren’t using the bad stuff to learn its substance. They are using it to learn patterns in human writing.
- vintermann 11mo agoScraping is cheap, training is expensive. Even the pre-generative AI internet had immense volumes of Markov-generated, synonym spun ("Contemporary York Instances") or otherwise brain-rotting text. That means that before training a big model, anyone will spend a lot of effort filtering out junk. They have done that for a decade, personally I think a lot of the differences in quality of the big models isn't from architectural differences, but rather from how much junk slipped through. Markov chains are not nearly clever enough to avoid getting filtered out.
- dilyevsky 11mo agoI am not actually claiming that it’s easy to filter out like the others. What Im saying is you can literally feed a ton of garbage into a training run and amazingly it still learns