4 ms·
Web scraping is legal. And if you run a website and want to opt-out then simply add a robots.txt. The standard way of preventing bots for 30 years.
by threeseed 2y ago
Web scraping is legal.
And if you run a website and want to opt-out then simply add a robots.txt.
The standard way of preventing bots for 30 years.
- mdhb 2y agoHow are people supposed to block it when they stole all the data first and then only after that point they decide to even tell anyone what user agent they need to block and how they are planning to exploit your work for their profit.
- threeseed 2y agoYou just have a rule that says block everything except crawlers: A, B, C. Also the AppleBot was known about before it appeared in Siri.
- immibis 2y agoSo you expect all websites to block FoobarSearch so it never gets off the ground and becomes a big search engine that people know to unblock. Then FoobarSearch learns to ignore robots.txt wildcards, and we're back at square one. IIRC this happened to DDG or Bing.
- threeseed 2y agoWebsites have always had the ability to precisely control who has access to their content. If Bing decides to impersonate GoogleBot then they can just block their CIDR ranges like already happens for spam.