7 ms·
Duck Duck Go written in Perl
- draegtun 16y agoAnd here are the HN comments on that Duck Duck Go architecture post: http://news.ycombinator.com/item?id=525048 http://news.ycombinator.com/item?id=525048
- jackowayed 16y agoThe perl code isn't a full search engine though. He uses Yahoo! Build Your Own Search Service, so that perl code is just taking the results from there and throwing results out, reordering results, getting the zero-click info, etc. Yahoo does all the crawling.
- sp332 16y agoSearching is not the same as indexing. DDG will give different results from Bing search, even though they use the same index. Edit: DDG does some crawling in addition to using the Bing index.
- Keyframe 16y agoI wonder if DDG could benefit from http://www.80legs.com/ http://www.80legs.com/ ?
- epi0Bauqu 16y agoI do my own crawling & indexing in addition to using third-party APIs. It's a hybrid approach where I focus on the places I think I add the most value.
- zaphar 16y agoThat's cool. I've used DDG a few times in the past and generally like so far. I had no idea it was done in perl though. It's nice to see new perl apps getting some press. I wonder if they use any of the newer CPAN libs like Moose and family in the architecture?
- draegtun 16y agoIt's nice to see new perl apps getting some press Some Perl press as appeared on HN before. For eg: http://news.ycombinator.com/item?id=565152 http://news.ycombinator.com/item?id=565152 (though sometimes the Perl community have done their best to keep head low on this one :)
- sdesol 16y agoAs for Moose, he (DDG author) answered this here http://www.reddit.com/r/IAmA/comments/bbqw7/i_am_the_founder_of_a_search_engine_duck_duck_go/ http://www.reddit.com/r/IAmA/comments/bbqw7/i_am_the_founder... Having written an indexer in Perl for my current startup product, I really can't see the need for using something like Moose. I'm biased though, as I'm not a big fan of OO programming. I gave it a try in Perl years ago but it really sucked. OO programming that is, but I guess Moose was designed to fix this, but it's a little too late for me. I was taught C in school and I've learned to live without OO.
- zaphar 16y agoI can't say I disagree with him on the dislike for OO. However if you are going to do OO in perl Moose is the way to do it.
- someone_here 16y agoMoose is nice, but making your own objects can be pretty cool too. Try blessing a function ;)
- mkramlich 16y agoi had pictured something elegant like Lisp. but no it was just hacked together in Perl? Somebody needs to link to the famous XKCD strip. :)
- piaresquared 16y agoYou mean this one? :) http://xkcd.com/224/ http://xkcd.com/224/
- gojomo 16y agoFrom some early discussion (and their current job listing), search engine startup Blekko also has a significant Perl component: http://blekko.com/jobs.html http://blekko.com/jobs.html
- jsrfded 16y agoYes, blekko has a full web crawler, indexer, and some of the query processing code in perl. Some of the innards and query execution are in C++ for speed. Generally we tried to delay the conversion of a component to C++ as late as possible, since it makes it so much harder to change and iterate on. We call it "pouring cement on the code". For some things it's absolutely necessary to drop to C for speed, others it doesn't make a lot of difference since you're bound by other constraints like I/O.
- draegtun 16y agoProbably the biggest profile Perl (and Catalyst) public app in last few years would be the BBC iPlayer: http://www.bbc.co.uk/blogs/bbcinternet/2008/12/iplayer_day_performance_tricks.html http://www.bbc.co.uk/blogs/bbcinternet/2008/12/iplayer_day_p...
- dragonquest 16y agoActually, YouPorn is powered by Perl and Catalyst and regularly makes the top 25 domains. Not sure if it beats the BBC main site though.
- draegtun 16y agoYep! http://news.ycombinator.com/item?id=1501063 http://news.ycombinator.com/item?id=1501063 :)
- deno 16y agoHow can he use Bing as a backend? I read their API's terms of service and it forbides you from changing search results in any way. “(c) modify, filter, obscure, or replace the text, images, or other content of Bing results, including by changing the order in which Bing results appear (but this limitation will not apply to Bing results of type "Web"), intermixing Bing results with search results from other sources, or intermixing with Bing results any other content so that the other content appears to be part of Bing results;” I was quite interested in using Bing API myself but that does seem too restrictive. I hope you can prove me wrong!
- gorm 16y ago> but this limitation will not apply to Bing results of type "Web") Isn't this what he is using?
- irq11 16y agoThat parenthetical only applies to the clause about result ordering. if you read the whole TOS, it's quite clear that Bing doesn't want you to do what DDG appears to be doing. In particular, mixing results from different services is a no-no in most search APIs.
- sandee 16y agoThanks. Good Post. This is great example that good application and complex systems can be written in perl as well. On a side note, i saw the previous post on DDG, where discussion was about how DDG does not store any private information. It makes sense in the context, that since he leverages upon other search engines , his cost of running is low. So he can afford to ignore user information otherwise used for commercial purpose by others.
- loewenskind 16y ago>This is great example that good application and complex systems can be written in perl as well. You could write "good application and complex systems" in machine language if you work hard enough. What are you trying to say?