Scraping is cheap, training is expensive. Even the pre-generative AI internet had immense volumes of Markov-generated, synonym spun ("Contemporary York Instance

Scraping is cheap, training is expensive. Even the pre-generative AI internet had immense volumes of Markov-generated, synonym spun ("Contemporary York Instances") or otherwise brain-rotting text. That means that before training a big model, anyone will spend a lot of effort filtering out junk. They have done that for a decade, personally I think a lot of the differences in quality of the big models isn't from architectural differences, but rather from how much junk slipped through. Markov chains are not nearly clever enough to avoid getting filtered out.