3 ms·
DDG uses Bing's search API. That's why you see Yelp on DDG. I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the s
by timr 13y ago
DDG uses Bing's search API. That's why you see Yelp on DDG.
I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the site, we'd have been regularly DOSed by robots. I don't (and didn't) like the precedent, but I understand some of the motivation.
- palakchokshi 13y agoI understand the motivation too. My request is that there should me some option for crawlers to apply to be whitelisted. I know Yelp asks you to contact them if you want to crawl them and they'll decide on a per case basis. It would be nice if there were standards that sites could implement to whitelist crawlers, create crawl schedules for whitelisted crawlers, something on the lines of robots.txt but with attributes that told a specific crawler when and how frequently they can crawl.
- timr 13y agoIt wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.
- palakchokshi 13y agoSo a "legit" crawler has no recourse but to resort to implementing technology that'll circumvent anti-crawler code? Can't say I didn't try to go legit :)
- deleted 13y ago[deleted]
- timr 13y agoWell, no: you ask, and if they say no, you don't get to crawl. The rules don't change because you think they're unjust. If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.
- palakchokshi 13y agoAgreed. I would rather not go down a route of building my product and then have a bunch of cease and desist orders taking away a lot of my search results. Asking nicely is the next step for me.