4 ms·
you are right. I don't want to make a better search engine than Google's. I do want to provide more personalized results than Google's. To do that I need to giv
by palakchokshi 13y ago
you are right. I don't want to make a better search engine than Google's. I do want to provide more personalized results than Google's. To do that I need to give my users a basic result set and then take input from their behavior to provide a more personalized experience. I do want to be able to search over a limited set of domains without hitting a 10000 req/day limit.
I typed "sushi" on duckduckgo.com and saw a result from Yelp. If duckduckgo.com used Yelp's API to get that result then you can imagine the complexity of maintaining API connections to hundreds of services like Yelp that do not allow crawling. However if duckduckgo.com crawled/scraped Yelp to get that result then that's against Yelp's TOS which would cause Yelp to send a cease and desist order to duckduckgo.com.
IMHO any crawler that drives traffic back to the original site (like Google, Bing, duckduckgo, etc) should be allowed to crawl the site but Yelp's TOS prevents that.
- dm2 13y agoDuckDuckGo lets developers build the extensions for them, which is very clever. http://duckduckhack.com/ http://duckduckhack.com/ Plus, the privacy promotion from them helped drive developers to their sites. Restrictions on what people can do on the internet is a slippery slope.
- timr 13y agoDDG uses Bing's search API. That's why you see Yelp on DDG. I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the site, we'd have been regularly DOSed by robots. I don't (and didn't) like the precedent, but I understand some of the motivation.
- palakchokshi 13y agoI understand the motivation too. My request is that there should me some option for crawlers to apply to be whitelisted. I know Yelp asks you to contact them if you want to crawl them and they'll decide on a per case basis. It would be nice if there were standards that sites could implement to whitelist crawlers, create crawl schedules for whitelisted crawlers, something on the lines of robots.txt but with attributes that told a specific crawler when and how frequently they can crawl.
- timr 13y agoIt wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.
- palakchokshi 13y agoSo a "legit" crawler has no recourse but to resort to implementing technology that'll circumvent anti-crawler code? Can't say I didn't try to go legit :)
- deleted 13y ago[deleted]
- timr 13y agoWell, no: you ask, and if they say no, you don't get to crawl. The rules don't change because you think they're unjust. If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.
- palakchokshi 13y agoAgreed. I would rather not go down a route of building my product and then have a bunch of cease and desist orders taking away a lot of my search results. Asking nicely is the next step for me.