4 ms·

Surprised to hear that crawling still runs out of the basement. For a general-purpose search engine, what kind of bandwidth does that require? Or is it manageab

Surprised to hear that crawling still runs out of the basement. For a general-purpose search engine, what kind of bandwidth does that require? Or is it manageable only because DDG proxies so many searches to other search engines? Also, doesn't it seem inefficient to have crawlers here and indexes there?

robotmay 14y ago
I believe yegg has a pretty awesome internet connection in his basement: http://www.gabrielweinberg.com/blog/2011/12/duckduckgo-used-to-run-out-of-my-basement.html http://www.gabrielweinberg.com/blog/2011/12/duckduckgo-used-...
orangethirty 14y ago
I run a couple of boxes that are crawl only for Nuuton right out of my office. I do have a good internet connection. But its a good way to save on infrastructure.
- boyter 14y ago
  I don't suppose you have a blog or something for that? My hobby/interest is search engines and I love reading about peoples experiences getting them running.
  - orangethirty 14y ago
    There is one but I have been too busy building it that have not posted anything yet. You can always get in touch through email. I like chatting with other hackers.
    - boyter 14y ago
      That's a pity. Just for the record for anyone looking at this space here are some links to get started, http://www.yioop.com/blog.php http://www.gigablast.com/rants.html http://queue.acm.org/detail.cfm?id=988407 http://blog.procog.com/ http://www.thebananatree.org/ http://blog.blekko.com/ I will even suggest my own small implementation (created purely for SEO value but it does work) http://searchco.de/blog/view/code-for-a-search-engine-in-php-part-1/ BTW email sent.
      - ersii 14y ago
        These are all great links to read if you'd like to dig deeper into search engine land. Thanks for the searchco.de articles, they were nice to read (Saw them in the previous searchco.de HN) :-)
        boyter 14y ago
        Thanks. I collect any that come up since its really such a dark area. I should probably write a blog post about it since I have a few more I have since dug up. Glad you like the articles :) it was something I had been writing for weeks and then finally got my act together and finished it.

robotmay 14y ago

I believe yegg has a pretty awesome internet connection in his basement: http://www.gabrielweinberg.com/blog/2011/12/duckduckgo-used-to-run-out-of-my-basement.html http://www.gabrielweinberg.com/blog/2011/12/duckduckgo-used-...

orangethirty 14y ago

I run a couple of boxes that are crawl only for Nuuton right out of my office. I do have a good internet connection. But its a good way to save on infrastructure.

boyter 14y ago
I don't suppose you have a blog or something for that? My hobby/interest is search engines and I love reading about peoples experiences getting them running.
- orangethirty 14y ago
  There is one but I have been too busy building it that have not posted anything yet. You can always get in touch through email. I like chatting with other hackers.
  - boyter 14y ago
    That's a pity. Just for the record for anyone looking at this space here are some links to get started, http://www.yioop.com/blog.php http://www.gigablast.com/rants.html http://queue.acm.org/detail.cfm?id=988407 http://blog.procog.com/ http://www.thebananatree.org/ http://blog.blekko.com/ I will even suggest my own small implementation (created purely for SEO value but it does work) http://searchco.de/blog/view/code-for-a-search-engine-in-php-part-1/ BTW email sent.
    - ersii 14y ago
      These are all great links to read if you'd like to dig deeper into search engine land. Thanks for the searchco.de articles, they were nice to read (Saw them in the previous searchco.de HN) :-)
      - boyter 14y ago
        Thanks. I collect any that come up since its really such a dark area. I should probably write a blog post about it since I have a few more I have since dug up. Glad you like the articles :) it was something I had been writing for weeks and then finally got my act together and finished it.