5 ms·

Duck Duck Go Architecture

72 points by epi0Bauqu 18y ago

deleted 18y ago
[deleted]
- e1ven 18y ago
  epi0Bauqu, a HN reader and creator of DDG is understandably proud of his work, so he submits articles when he does something he thinks is cool. Pretty reasonable to my eye, as long as he's submitting stuff he's proud of and may be interesting, not just trying to get people to look at his site.
- jimbokun 18y ago
  Skimming the headlines of articles on the front page right now, this is more Hackerish, in my view, than several others on the list. He discusses what he is building, and how he's doing it.
jwilliams 18y ago
Cool article. Couple of queries :-) It mentions that static content is served from nginx, but then mentions S3 later on as well. Why not just Amazon for all the static content? Mentions nginx being more stable than Apache - you mean no-crash stable, or more consistent under load?
- epi0Bauqu 18y ago
  Thanks! I use S3 to serve images that are loaded after the initial page load (incidentally, using YUI's ImageLoader: http://developer.yahoo.com/yui/imageloader/ http://developer.yahoo.com/yui/imageloader/). I use nginx to serve static files, e.g. HTML, as well as images that get loaded on page load, e.g. logo. Why not S3 for the latter? Too slow. Amazon's (relatively) new content network may be faster, though I haven't tried it yet. Wrt nginx being more stable, both. I've crashed Apache as well as pseudo-crashed it (process still running, but not serving right). nginx hasn't crashed yet. It's also more consistent (and faster) under load. However, in Apache's defense, it does a lot more, e.g. mod_perl vs nginx proxy to FastCGI.
  - jwilliams 18y ago
    Cool - well I'm on Apache now, on a very undersized server - so I guess I'll have to give nginx a shot. Not sure how you'd extract more speed from your current setup. The natural next step would be doing work in parallel - this something you do/have considered? If you do try the Amazon CDN, let us know how you go :)
    - epi0Bauqu 18y ago
      I use multiple worker processes in nginx (2 x # of cores), which work in parallel. Then FCGI::Engine is configured to run multiple processes as well. Finally, I use the external JS and image files (in part) to speed up the client-side via parallel processing of requests. (Note that IE7 and below limit concurrent connections per host by default often to 2: http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx).
almost 18y ago
I've seen Duck Duck Go when you posted before and didn't think much of the idea (hey, I can use Wikipedia!) Just had a look again and I now think it's pretty awesome. Just for a really speedy "what the hell is this?" type of query it looks absolutely amazing. I love the AJAX extra search links, makes it all feel really snappy. Works great on the iPhone as well, I've added an icon to my home screen. You might want to consider making an app for the app store (you could pretty much just make an app that loads the site as it is). Have you considered adding an API? I currently have my Emacs set up to do a dictionary lookup (using the dictd protocol) on a word when I hit C-cl. I'd love to add Duck Duck Go results to the buffer this displays!
- epi0Bauqu 18y ago
  Thanks! Yeah, we've gotten a lot better since then. On the zero-click info (what is x?) we use lots of sources now beyond Wikipedia (e.g. Crunchbase) as well as have a bunch of new features (http://duckduckgo.com/about.html http://duckduckgo.com/about.html). We also have both an iPhone app and an API :) iPhone App: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=301375111&mt=8 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftwa... API info: http://duckduckgo.com/api.html http://duckduckgo.com/api.html
  - almost 18y ago
    Whoops, should have spotted both of those :p Both look good though!
strlen 18y ago
Question: why are you going with FastCGI over mod_perl? Issues with mod_perl debugging/memory leaks? Desire to use nginx over Apache for higher throughput? The big advantage -- to me -- of mod_perl is the power of having your code live within Apache's process space. I wonder if mod_perl works with event_mpm, yet.
- epi0Bauqu 18y ago
  d) All of the above. Initially I didn't think anything could beat mod_perl speed, but after benchmarking, this setup is just as fast. It's also more stable, and uses less memory. And yes, also it is easier to debug and I was getting memory leaks on mod_perl. Don't get me wrong, I love mod_perl. I used it at my last company and on many other projects before this one. And I still use it for other stuff. Last I looked I think you can run mod_perl with event_mpm but I didn't like this: "The current Apache::DBI should be usable under threaded mpm, though it doesn't share connections across threads. Each Perl interpreter has its own cache, just like in the prefork mpm." (http://perl.apache.org/docs/2.0/user/performance/mpm.html http://perl.apache.org/docs/2.0/user/performance/mpm.html)
globalrev 18y ago
do you make money?
- epi0Bauqu 18y ago
  No ads on the site yet, but in general, search ads make money.
njharman 18y ago
I'm not exactly a search connoisseur or even pay much attention to startups but wow. This is one of the first search innovations I've found that was any good, in fact it's great. I've just replaced my google FF keyword, 'g', with duckduckgo link. I'll try it out for a week or so and see if I still think it's as awesome as I do now.
- epi0Bauqu 18y ago
  Thanks! Please email me your thoughts after the week--always looking for real feedback from real users. Also note we have a FF extension: https://addons.mozilla.org/en-US/firefox/addon/10536 https://addons.mozilla.org/en-US/firefox/addon/10536
brandnewlow 18y ago
This passed my "does this not suck?" test. Good work. A search engine that does not suck and is not named Google.
- kingkongrevenge 18y ago
  gigablast finds stuff for me that others miss.
tdonia 18y ago
complete side note - but your favicon has a lot of aliasing when it's on a dark background (i.e. as it is when you have ddg in an inactive chrome tab). the circle looks nice on white, but you may want to adjust it so it reads a little better against other mattes. aside from that tiny-minor issue, ddg seems cool & the overview of the architecture is very clear. i'm going to give it a try as my default chrome search engine & figure out just how behaviorally indoctrinated google has me and how your architecture holds up as more people use the service.
- epi0Bauqu 18y ago
  Thanks! I will look into the favicon issue. Please send me an email after you try it for a while with your thoughts.
rozim 18y ago
I must have missed something - what technologies are behind the actual search? Lucene/Nutch/Xapian?? thx, Dave
- epi0Bauqu 18y ago
  All proprietary Perl.

72 points by epi0Bauqu 18y ago

deleted 18y ago

[deleted]

e1ven 18y ago
epi0Bauqu, a HN reader and creator of DDG is understandably proud of his work, so he submits articles when he does something he thinks is cool. Pretty reasonable to my eye, as long as he's submitting stuff he's proud of and may be interesting, not just trying to get people to look at his site.
jimbokun 18y ago
Skimming the headlines of articles on the front page right now, this is more Hackerish, in my view, than several others on the list. He discusses what he is building, and how he's doing it.

jwilliams 18y ago

Cool article. Couple of queries :-) It mentions that static content is served from nginx, but then mentions S3 later on as well. Why not just Amazon for all the static content? Mentions nginx being more stable than Apache - you mean no-crash stable, or more consistent under load?

epi0Bauqu 18y ago
Thanks! I use S3 to serve images that are loaded after the initial page load (incidentally, using YUI's ImageLoader: http://developer.yahoo.com/yui/imageloader/ http://developer.yahoo.com/yui/imageloader/). I use nginx to serve static files, e.g. HTML, as well as images that get loaded on page load, e.g. logo. Why not S3 for the latter? Too slow. Amazon's (relatively) new content network may be faster, though I haven't tried it yet. Wrt nginx being more stable, both. I've crashed Apache as well as pseudo-crashed it (process still running, but not serving right). nginx hasn't crashed yet. It's also more consistent (and faster) under load. However, in Apache's defense, it does a lot more, e.g. mod_perl vs nginx proxy to FastCGI.
- jwilliams 18y ago
  Cool - well I'm on Apache now, on a very undersized server - so I guess I'll have to give nginx a shot. Not sure how you'd extract more speed from your current setup. The natural next step would be doing work in parallel - this something you do/have considered? If you do try the Amazon CDN, let us know how you go :)
  - epi0Bauqu 18y ago
    I use multiple worker processes in nginx (2 x # of cores), which work in parallel. Then FCGI::Engine is configured to run multiple processes as well. Finally, I use the external JS and image files (in part) to speed up the client-side via parallel processing of requests. (Note that IE7 and below limit concurrent connections per host by default often to 2: http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx).

almost 18y ago

I've seen Duck Duck Go when you posted before and didn't think much of the idea (hey, I can use Wikipedia!) Just had a look again and I now think it's pretty awesome. Just for a really speedy "what the hell is this?" type of query it looks absolutely amazing. I love the AJAX extra search links, makes it all feel really snappy. Works great on the iPhone as well, I've added an icon to my home screen. You might want to consider making an app for the app store (you could pretty much just make an app that loads the site as it is). Have you considered adding an API? I currently have my Emacs set up to do a dictionary lookup (using the dictd protocol) on a word when I hit C-cl. I'd love to add Duck Duck Go results to the buffer this displays!

epi0Bauqu 18y ago
Thanks! Yeah, we've gotten a lot better since then. On the zero-click info (what is x?) we use lots of sources now beyond Wikipedia (e.g. Crunchbase) as well as have a bunch of new features (http://duckduckgo.com/about.html http://duckduckgo.com/about.html). We also have both an iPhone app and an API :) iPhone App: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=301375111&mt=8 http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftwa... API info: http://duckduckgo.com/api.html http://duckduckgo.com/api.html
- almost 18y ago
  Whoops, should have spotted both of those :p Both look good though!

strlen 18y ago

Question: why are you going with FastCGI over mod_perl? Issues with mod_perl debugging/memory leaks? Desire to use nginx over Apache for higher throughput? The big advantage -- to me -- of mod_perl is the power of having your code live within Apache's process space. I wonder if mod_perl works with event_mpm, yet.

epi0Bauqu 18y ago
d) All of the above. Initially I didn't think anything could beat mod_perl speed, but after benchmarking, this setup is just as fast. It's also more stable, and uses less memory. And yes, also it is easier to debug and I was getting memory leaks on mod_perl. Don't get me wrong, I love mod_perl. I used it at my last company and on many other projects before this one. And I still use it for other stuff. Last I looked I think you can run mod_perl with event_mpm but I didn't like this: "The current Apache::DBI should be usable under threaded mpm, though it doesn't share connections across threads. Each Perl interpreter has its own cache, just like in the prefork mpm." (http://perl.apache.org/docs/2.0/user/performance/mpm.html http://perl.apache.org/docs/2.0/user/performance/mpm.html)

globalrev 18y ago

do you make money?

epi0Bauqu 18y ago
No ads on the site yet, but in general, search ads make money.

njharman 18y ago

I'm not exactly a search connoisseur or even pay much attention to startups but wow. This is one of the first search innovations I've found that was any good, in fact it's great. I've just replaced my google FF keyword, 'g', with duckduckgo link. I'll try it out for a week or so and see if I still think it's as awesome as I do now.

epi0Bauqu 18y ago
Thanks! Please email me your thoughts after the week--always looking for real feedback from real users. Also note we have a FF extension: https://addons.mozilla.org/en-US/firefox/addon/10536 https://addons.mozilla.org/en-US/firefox/addon/10536

brandnewlow 18y ago

This passed my "does this not suck?" test. Good work. A search engine that does not suck and is not named Google.

kingkongrevenge 18y ago
gigablast finds stuff for me that others miss.

tdonia 18y ago

complete side note - but your favicon has a lot of aliasing when it's on a dark background (i.e. as it is when you have ddg in an inactive chrome tab). the circle looks nice on white, but you may want to adjust it so it reads a little better against other mattes. aside from that tiny-minor issue, ddg seems cool & the overview of the architecture is very clear. i'm going to give it a try as my default chrome search engine & figure out just how behaviorally indoctrinated google has me and how your architecture holds up as more people use the service.

epi0Bauqu 18y ago
Thanks! I will look into the favicon issue. Please send me an email after you try it for a while with your thoughts.

rozim 18y ago

I must have missed something - what technologies are behind the actual search? Lucene/Nutch/Xapian?? thx, Dave

epi0Bauqu 18y ago
All proprietary Perl.