Adding to this, the Internet is extremely noisy and polluted with all manner of crap, this has been the status quo for over a decade. The notion that you could

Adding to this, the Internet is extremely noisy and polluted with all manner of crap, this has been the status quo for over a decade. The notion that you could just grab data off the internet completely unvetted, and then train a useful model off that data seems very unrealistic. To get anything useful out of real world crawl data, you need a lot of filtering and processing to select the good bits. This is true if you're training an AI or building a search engine.