3 ms·

I really don't get why we got here. The big 5 providers have enough money to hire/poach some scraping/indexing experts. They could avoid lots of grief and issue

I really don't get why we got here. The big 5 providers have enough money to hire/poach some scraping/indexing experts. They could avoid lots of grief and issues by just indexing in a non stupid way. Even if they still ignored robots.txt, they could fly under the radar with a few well placed exceptions. But now they're risking losing easy access to a lot of the internet by just doing a really bad job of a thing they really rely on. I get that it could be incompetence in one or two places, but as far as I know all the big providers fail here... so am I missing something obvious? Is it really just greed and vc money burning scrapefest?

gruez 2y ago
>The big 5 providers have enough money to hire/poach some scraping/indexing experts. [...] >I get that it could be incompetence in one or two places, but as far as I know all the big providers fail here... Is there any indication that the scraping traffic is from the "the big providers"? The article doesn't mention this.
- dannyobrien 2y ago
  I am also interested in this! I really want to know who is doing this scraping, and why are they doing it badly relative to search-engines? The argument that I've heard is that search engines have a vested interest in keeping sites up, whereas ML companies do not, but that seems a little too mechanistic -- some of the mistakes in load aren't good for the scrapers resources, as well as the scraped. My working theory is that rather than the big AI companies, this is mainly a rash of Asmall I startups (globally) who don't know any better and are just writing poor tools, and also think that "unique data" gives them an advantage. So this is more about a flood of capital to (frankly) incompetent groups, that will empty out when the boom busts. But I can see the arguments on both sides. It would be great to know the facts!

gruez 2y ago

>The big 5 providers have enough money to hire/poach some scraping/indexing experts. [...] >I get that it could be incompetence in one or two places, but as far as I know all the big providers fail here... Is there any indication that the scraping traffic is from the "the big providers"? The article doesn't mention this.

dannyobrien 2y ago
I am also interested in this! I really want to know who is doing this scraping, and why are they doing it badly relative to search-engines? The argument that I've heard is that search engines have a vested interest in keeping sites up, whereas ML companies do not, but that seems a little too mechanistic -- some of the mistakes in load aren't good for the scrapers resources, as well as the scraped. My working theory is that rather than the big AI companies, this is mainly a rash of Asmall I startups (globally) who don't know any better and are just writing poor tools, and also think that "unique data" gives them an advantage. So this is more about a flood of capital to (frankly) incompetent groups, that will empty out when the boom busts. But I can see the arguments on both sides. It would be great to know the facts!