3 ms·

Well that's an interesting reversal. Back when I was a kid, websites would have terms of services that their users were expected to follow. Now, web scrapers

by klyrs 3y ago

Well that's an interesting reversal. Back when I was a kid, websites would have terms of services that their users were expected to follow. Now, web scrapers have terms of... consumption? Time to make a webring of Markov generated garbage to feed the bots...

scarface_74 3y ago
Did Google say they aren’t respecting robots.txt files?
- palata 3y ago
  Does anyone respect robots.txt files?
  - scarface_74 3y ago
    Any history of Google not respecting it?
    - tteggel 3y ago
      Google does not respect robots.txt if a page is linked to from other sites https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
      - djmips 3y ago
        That doesn't say the content but only that it is indexed.