3 ms·
I understand the motivation too. My request is that there should me some option for crawlers to apply to be whitelisted. I know Yelp asks you to contact them if
by palakchokshi 13y ago
I understand the motivation too. My request is that there should me some option for crawlers to apply to be whitelisted. I know Yelp asks you to contact them if you want to crawl them and they'll decide on a per case basis.
It would be nice if there were standards that sites could implement to whitelist crawlers, create crawl schedules for whitelisted crawlers, something on the lines of robots.txt but with attributes that told a specific crawler when and how frequently they can crawl.
- timr 13y agoIt wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.
- palakchokshi 13y agoSo a "legit" crawler has no recourse but to resort to implementing technology that'll circumvent anti-crawler code? Can't say I didn't try to go legit :)
- deleted 13y ago[deleted]
- timr 13y agoWell, no: you ask, and if they say no, you don't get to crawl. The rules don't change because you think they're unjust. If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.
- palakchokshi 13y agoAgreed. I would rather not go down a route of building my product and then have a bunch of cease and desist orders taking away a lot of my search results. Asking nicely is the next step for me.