4 ms·
I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search p
by whazor 1y ago
I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.
- Loic 1y agoNot necessarily, I have a website with 95% (maybe even more) of the traffic generated by crawlers. If some of them are behaving badly, it is fair to exclude them with my robots.txt. But of course, the ones behaving badly tend to not respect the robots.txt, so you end up banning the IP or IP block. And here, I am a nice guy, the crawler must really be a piece of crap for me to start to block.
- throawayonthe 1y agothe parent comment is talking about allow-listing (aka 'whitelisting') just a few crawlers from like, google
- whazor 1y agoDeny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt. But allow-listing particular crawlers only is collusion.
- ToucanLoucan 1y agoThis rather bluntly runs up against the fact that permitting crawling is an expense the web operator is taking on, ergo, receiving that content is by definition a privilege not a right.
- p3rls 1y agoI think my team has spent about a quarter of this year's dev time on defending against scrapers. Their arrogance is the stuff of ancient greek dramas.
- lucb1e 1y agoI don't know if that's a reply at me or a general remark, but yes, I never understood why you'd include a few big names and ban the rest for example. That's just screaming for anticompetitiveness. I don't know if my mention of robots.txt sounded like I do this, but I do not