4 ms·

I might be thinking about this the wrong way, but, scrapers as described by this article are more-or-less targeted web crawlers, right? Don't we have a de-fact

• FastedCoyote 6y ago
  It's trivial to change the User Agent.
  • dafoex 6y ago
    It is, but what I'm suggesting is that a scraper identifies itself as such in the user agent and complies with the robots exclusion standard. Google's crawlers recognise that `googlebot` - among other strings - refers to them, and a scraper should behave similarly if it doesn't want to be seen as malicious. I think scrapers should be legal, but just like crawlers, they should comply with a directive that says they aren't welcome. And on the topic of trivial workarounds, it's even more trivial to just ignore the robots.txt and scrape away, but that's what malicious bots performing questionable activities do. One would hope a newsbot is being civil and not behaving in a shady manner, but alas, I don't for one moment think any of these news publishers have even heard of a robots.txt