4 ms·

>... compared to google search which you'd never be able to host locally That's something I haven't thought about for a long time. How big is the Google index,

• raincole 3y ago
  > over 100,000,000 gigabytes Accodrding to Google: https://www.google.com/search/howsearchworks/how-search-works/organizing-information https://www.google.com/search/howsearchworks/how-search-work...
• hansvm 3y ago
  Probably petabytes for the text index and exabytes for the whole index (including images and video). The entire textual internet is hundreds of terabytes uncompressed, and indexes tend to balloon storage size to make serving tailored queries faster and cheaper. That idea shifts a bit with images/video where vector/ML techniques dominate; you get a reduction in quality, but quality is good enough (mostly, probably, we think), and you have within a small constant multiple one way or the other of the underlying dataset size for the index size (as opposed to text, where 100x-1000x one way or another isn't uncommon). If the index is less than petabytes then it's impossible to serve many long tail queries even just based on text content. IME those queries _have_ gotten progressively worse results the last few years, so maybe the techniques have changed, but when the money printer is "search" and a petabyte of data is a fraction of a full-time engineer in cost to store, I doubt they would have cut costs that aggressively.
• jdshaffer 3y ago
  That reminds me of the time my friend downloaded the ENTIRE Yahoo! search database to a 1.44MB floppy. I think that was back in 1993. laugh