11 ms·
DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
- account_taken 14y ago1 million doesn't seem that big a number on the internet?
- jules 14y agoOn average that's around 11.6 requests per second. That's about 500 million CPU cycles per request on single low end CPU. Of course DDG probably isn't CPU bound, but it does give an indication that 11 requests per second is not a lot relative to the raw power of modern hardware.
- yegg 14y agoWe also do about 12M API requests per day: http://duckduckgo.com/traffic.html http://duckduckgo.com/traffic.html
- jules 14y agoNot sure why yegg's (Gabriel Weinberg) comment was killed. It says: > We also do about 12M API requests per day: http://duckduckgo.com/traffic.html http://duckduckgo.com/traffic.html
- tedunangst 14y agoSame comment was posted in another thread.
- gummydude 14y agoTheir system centered around APIs, hashmap and msg encode/decode are CPU bound.
- account_taken 14y agoDoesn't google do around 3 billion searches/day? I was under the impression DuckDuckGo was a bigger player.
- abbert 14y agoWhen will google block DDG?
- hnriot 14y agothe bad press among techheads might make it not worth their bothering. Remember the people that use ddg are the candidate pool for most hiring positions at google. regular people use use google/bing etc
- runarb 14y agoDDG don't rely on Google. There results comes from there own systems and Yahoo (through BOSS), embed.ly, WolframAlpha, EntireWeb, Bing, Yandex, and Blekko according to http://help.duckduckgo.com/customer/portal/articles/216399-sources http://help.duckduckgo.com/customer/portal/articles/216399-s...
- moheeb 14y agoWell DuckDuckGo searches their own stock and then points you towards google for anything DuckDuckGo doesn't have in stock. Google should be paying them commission, if anything.
- afsina 14y agoCustom Perl solution O_o. Ok seriously, no matter what, DuckDuckGo is still just a front end of some search engines.
- JoachimSchipper 14y agoDon't be so nasty.
- afsina 14y agoOk I was harsh, but I wonder if what I said was wrong. I do not understand the over protective behavior over DuckDuckGo news and brand in HN.
- pestaa 14y agoHN is helping Davids against Goliaths and I consider it a very good thing.
- badgar 14y agoWhat happened to merit?
- irahul 14y agoIt's not about being protective over DDG. You are being an asshole again and again in this thread. By your admission, you don't have any experience with search engines, and yet you have definitive opinion about everything related to it.
- davidpayne11 14y ago1) He never called you an asshole. He just posted his views about DDG. 2) If you are offended by what he said, it doesn't give you the right to call him an asshole. For example, I know many people whose projects you didn't complete on time and who think you are a rotten dick for rightly being one. But I didn't call you a rotten dick, did I? And also, stop defending DDG like a pussy that's everywhere, dude.
- gary4gar 14y agoDDG is using dynamic language like Perl[1] to achieve high-scalability which is slowest of all languages. This proves languages don't matter much, its all about architecture. I guess its time to stop worrying about performance of your programming language & start building better high-salable architectures. [1] Slowest of all languages http://benchmarksgame.alioth.debian.org/u32/which-programs-are-fastest.php?gcc=on&gpp=on&ifc=on&java=on&v8=on&hipe=on&jruby=on&php=on&python3=on&yarv=on&perl=on&calc=chart http://benchmarksgame.alioth.debian.org/u32/which-programs-a...
- afsina 14y agoThis is simply not true. Not every problem can be solved with caching. If DuckDuckGo would be a real search engine, they would not even touch with a 10 foot pole to Perl for any non trivial algorithm/function. edit: fix the name
- randomstring 14y agoUh, hello!? Blekko is a web scale search engine that is built using Perl. It's not the language that makes handling big data slow. It's the algorithms and how you move the bits around.
- rieter 14y agoUhm, no. Google can't be implemented in Perl.
- afsina 14y agoWell, sure if you want to use 1000 servers instead of 100, go ahead and use it. No wonder Facebook noticed this and desperately trying to compile Php to C. Perhaps if Blekko ever gets popular we will see if their choice will bite them. Also an anecdotal example. Two months ago a friend of mine wrote a rather complex algorithm in Perl (He is very fluent with it). It was a novel sentence alignment algorithm using Gibbs sampling. Algorithm was hard to make parallel and not cache friendly. He needed to wait more than a day for training the system in a server. Well, long story short, with his help another developer converted it to Java in short time it worked around 200 times faster. So there. Moving bits around did not help Perl here at all.
- hexonexxon 14y agostartpage.com seems to have a better front end, is faster and the results actually what you're searching for. haven't dug into their security to see if it really is "the world's most private search engine"
- dublinben 14y agoStartPage just serves you Google results through a proxy. If you're trying to avoid Google, this is pointless.
- temes 14y agoMost people are just trying to avoid Google tracking. Startpage claims to not even record your IP address. And it was hosted outside of the USA, but it seems that isn't the case anymore...
- umenline 14y agowhere can i find info on how the ddg crawler is build?
- knowaveragejoe 14y agoRegarding the "specialized semantic data", isn't Google already offering similar stuff and/or in the position to offer more? The only example that comes to mind is searching for word definitions, but I'm sure there's more.
- bifrost 14y agoI've gotta say, I love DDG, I rarely ever use Big-G for anything anymore. If I had to guess, it saves me a couple hours a week!
- wiremine 14y agoJust curious, as a non-DDG user: what features are you using that are saving that much time?
- orta 14y agoThe major for me is being able to do meta-queries that will be sent to the relevant website, e.g. "!maps 401 broadway to Canal St Station" or "!wa how many stars are in the galaxy" http://duckduckgo.com/bang.html http://duckduckgo.com/bang.html
- lutz 14y agoBut Google does that already, without needing to type !maps...
- nonamegiven 14y agoYou get better (short, sane and email-friendly) URLs if you access google searches via DDG bang. !gm = google maps, !g = google search, !gi = google images, etc.
- badgar 14y agoSo we have DDG as a better solution when you're e-mailing links to google searches. That doesn't add up to hours a week.
- pbhjpbhj 14y agoI do the same sort of queries but just use the search keywords feature of Firefox - so that's surely even faster as DDG isn't acting as intermediary to interpret the search request and redirect. That said I think the competition DDG has provided the other search providers has been a benefit to me.
- fourstar 14y ago"Front-end development uses a lot low level JavaScript. Thinking of moving from YUI to jQuery." So they are using node? I can't think of YUI being "low level javascript" at all.
- photorized 14y agoFor your own crawler, what do you estimate the size of your index (# of objects)? Just curious how your own data compares to the rest of your sources (Bing, Yandex, Blekko, etc)
- pjungwir 14y agoSurprised to hear that crawling still runs out of the basement. For a general-purpose search engine, what kind of bandwidth does that require? Or is it manageable only because DDG proxies so many searches to other search engines? Also, doesn't it seem inefficient to have crawlers here and indexes there?
- robotmay 14y agoI believe yegg has a pretty awesome internet connection in his basement: http://www.gabrielweinberg.com/blog/2011/12/duckduckgo-used-to-run-out-of-my-basement.html http://www.gabrielweinberg.com/blog/2011/12/duckduckgo-used-...
- orangethirty 14y agoI run a couple of boxes that are crawl only for Nuuton right out of my office. I do have a good internet connection. But its a good way to save on infrastructure.
- boyter 14y agoI don't suppose you have a blog or something for that? My hobby/interest is search engines and I love reading about peoples experiences getting them running.
- orangethirty 14y agoThere is one but I have been too busy building it that have not posted anything yet. You can always get in touch through email. I like chatting with other hackers.
- boyter 14y agoThat's a pity. Just for the record for anyone looking at this space here are some links to get started, http://www.yioop.com/blog.php http://www.gigablast.com/rants.html http://queue.acm.org/detail.cfm?id=988407 http://blog.procog.com/ http://www.thebananatree.org/ http://blog.blekko.com/ I will even suggest my own small implementation (created purely for SEO value but it does work) http://searchco.de/blog/view/code-for-a-search-engine-in-php-part-1/ BTW email sent.
- kimmel 14y agoGood god, the comments in this thread really show how douchy the HN community can be. Tons of anecdotal stories with no evidence. Obvious outsiders trying to image what it is like on the inside (all the guesses about how DDG works) and of course no facts to back anything up. I would say this pandering and uninformed behaviour is not common on HN, but it is. People who engage in language flame wars simply do not understand that when you become a good programmer languages do not matter only the platforms matter.
- hammerzeit 14y agoLeaving aside one trolly user, most of the conversation here is either referencing data mentioned in the article or asking questions/having discussions about semi-related points. I'd hardly call the argument over language speed a 'flame war' by any internet standard I've ever seen. There are probably better examples than this one to pull the 'HN commenters suck' card. The conversation here is mostly civil and trying to learn things, one troll aside.
- rehack 14y agoAgree. The choice of perl actually does surprise people, genuinely. As somewhere else mentioned, even blekko uses perl. So obviously these people know what they are doing. But that said, choosing anything interpreted does come to haunt you later on, in some way or the other. We run Java on Jettys for most of the things for our app (traffic is around 60k/70k in a day, so much less in comparison). But even with this traffic, we need to use 2 large EC2 instances in the day time. And mysteriously the Jettys keep going out of memory every now and then.* I am sure the same thing if done in C++ would need only 1 large EC2 instance. And it will also help the latency a bit, as a parallel gain. At present am analyzing the cost/benefit of such a move. Inputs are welcome. * With Java its always the memory which hurts you first. Latency wise, not much of a difference, in most cases. Edit: Down vote? Surprised. Why??
- burrows 14y ago> People who engage in language flame wars simply do not understand that when you become a good programmer languages do not matter only the platforms matter. Guess you're building the next killer app in brainfuck?
- druiid 14y agoSeems like a mostly straightforward arch. I have only one question (and it's a genuine question as I'm not sure under what circumstances it's being used and if there is an issue that was being run up against). My question is why is Solr replication being run using a 'custom' solution rather than cloud/Zookeeper or similar, or even just the standard master/slave arch if the data was small enough?
- pepr 14y agoI'd be very interested to hear a ballpark estimate how much data they're storing in their PostgreSQL instances. Any word on that?