4 ms·
The normal way to exclude a crawler is to use a robots.txt file: http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html Any ethical cr
by smacktoward 13y ago
The normal way to exclude a crawler is to use a robots.txt file: http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
Any ethical crawler will respect exclusions defined in that file. The article doesn't mention trying this method before jumping to mod_rewrite, though.
So -- does Semalt's crawler look for and follow robots.txt? As long as it does, they're doing what they should be doing to let site owners opt out of crawling.
EDIT: Found a page on their web site where you can enter a domain to opt it out of crawling: http://semalt.com/project_crawler.php http://semalt.com/project_crawler.php
No instructions for how to opt out via robots.txt, though. That's a big omission. Anyone who's going to do mass crawling needs to support robots.txt.
- RyJones 13y agoFirst comment on OP says they ignore robots.txt
- lingben 13y agoyeah, crawlers don't work like that or rather, ethical companies don't release crawlers that don't follow robot.txt instructions semalt can not fix this by offering an 'opt-out' while ignoring robot.txt very stupid of them because they are shooting themselves in the foot by this very simple but MASSIVE breach of standards of conduct