5 ms·
So a "legit" crawler has no recourse but to resort to implementing technology that'll circumvent anti-crawler code? Can't say I didn't try to go legit :)
by palakchokshi 13y ago
So a "legit" crawler has no recourse but to resort to implementing technology that'll circumvent anti-crawler code? Can't say I didn't try to go legit :)
- deleted 13y ago[deleted]
- timr 13y agoWell, no: you ask, and if they say no, you don't get to crawl. The rules don't change because you think they're unjust. If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.
- palakchokshi 13y agoAgreed. I would rather not go down a route of building my product and then have a bunch of cease and desist orders taking away a lot of my search results. Asking nicely is the next step for me.