3 ms·

Honestly I think people are completely underestimating the difficulty of a good search engine I suspect that is not difficult so much as expensive. Boutiqu

Honestly I think people are completely underestimating the difficulty of a good search engine I suspect that is not difficult so much as expensive. Boutique search engines pop up all the time here on HN, but they can't compete fairly against Google, without the resources to crawl a billion webpages day after day.

charcircuit 5y ago
Google also keeps a copy of the "entire" web in RAM to search it faster.
- 1_player 5y ago
  Source? But it's definitely possible nowadays when RAM is cheap and modern Google makes the Internet feel incredibly smaller than it was 20 years ago. At the time you would get tons of results from a myriad of small blogs, forums, niche websites. Nowadays it's Pinterest, blogspam and more SEO optimised algorithmic crap. If you're lucky you get a forum result that might actually be relevant. Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.
  - charcircuit 5y ago
    It looks like I was mistaken. They don't keep all of the documents in RAM, but I'm pretty sure they keep the search index (100 petabytes) in RAM. I'm struggling to find a source but I remember hearing it in a talk given by a Googler on YouTube. I also read somewhere that moving their whole index to RAM is how they brought their response times down from a couple of seconds to a couple hundred milliseconds.
  - cromwellian 5y ago
    Here, let Jeff Dean explain how it works (and why this is a complex problem that a lot of the naive 'build a better search engine than Google' miss) https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8a05a5e500dbf05.pdf https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8... The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.
    - charcircuit 5y ago
      A link to the talk for those curious https://www.youtube.com/watch?v=IQuSuyUVkoE https://www.youtube.com/watch?v=IQuSuyUVkoE

charcircuit 5y ago

Google also keeps a copy of the "entire" web in RAM to search it faster.

1_player 5y ago
Source? But it's definitely possible nowadays when RAM is cheap and modern Google makes the Internet feel incredibly smaller than it was 20 years ago. At the time you would get tons of results from a myriad of small blogs, forums, niche websites. Nowadays it's Pinterest, blogspam and more SEO optimised algorithmic crap. If you're lucky you get a forum result that might actually be relevant. Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.
- charcircuit 5y ago
  It looks like I was mistaken. They don't keep all of the documents in RAM, but I'm pretty sure they keep the search index (100 petabytes) in RAM. I'm struggling to find a source but I remember hearing it in a talk given by a Googler on YouTube. I also read somewhere that moving their whole index to RAM is how they brought their response times down from a couple of seconds to a couple hundred milliseconds.
- cromwellian 5y ago
  Here, let Jeff Dean explain how it works (and why this is a complex problem that a lot of the naive 'build a better search engine than Google' miss) https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8a05a5e500dbf05.pdf https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8... The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.
  - charcircuit 5y ago
    A link to the talk for those curious https://www.youtube.com/watch?v=IQuSuyUVkoE https://www.youtube.com/watch?v=IQuSuyUVkoE