3 ms·

Proposition: 99% of scraping use cases are eliminated if the scraper agrees to subsequently abide by the target's terms of service.

• jazoom 10y ago
  Googlebot doesn't abide by 99% of websites' terms of use.
  • simplyluke 10y ago
    But that's "different" because they've built a $600bn company off it.
    • jazoom 10y ago
      More that the websites actually want to be found by someone.
      • camus2 10y ago
        The over-reliance on google search is both a blessing and a curse for the web. Today, google IS basically the web, a centralized version of it.
  • piker 10y ago
    I agree with the implication here, but (1) it doesn't actually rebut the proposition and (2) some significant fraction of websites crawled by Googlebot have signed up on Google's webmaster tools, which includes its own ToS that likely governs.