My bet is many of these crawlers collect price matching, socio-political and other data. It is curious how it gets decided that all spiders crawl for training

My bet is many of these crawlers collect price matching, socio-political and other data. It is curious how it gets decided that all spiders crawl for training. And in fact the walled data is much more interesting, and particularly Reddit, X, and FB data where we still have indications of human or at least correct data lives. These cannot be poisoned that easy.