I am also interested in this! I really want to know who is doing this scraping, and why are they doing it badly relative to search-engines? The argument that I'

I am also interested in this! I really want to know who is doing this scraping, and why are they doing it badly relative to search-engines? The argument that I've heard is that search engines have a vested interest in keeping sites up, whereas ML companies do not, but that seems a little too mechanistic -- some of the mistakes in load aren't good for the scrapers resources, as well as the scraped. My working theory is that rather than the big AI companies, this is mainly a rash of Asmall I startups (globally) who don't know any better and are just writing poor tools, and also think that "unique data" gives them an advantage. So this is more about a flood of capital to (frankly) incompetent groups, that will empty out when the boom busts. But I can see the arguments on both sides. It would be great to know the facts!