3 ms·

It's funny how scraping can be illegal yet Google and Microsoft and Yahoo and DuckDuckGo, and every other search engine have a monopoly on it. It's legal for th

• Google234 6y ago
  You can block them in your robot.txt if you desire... it’s not the same thing, websites want to be indexed
• dafoex 6y ago
  Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.