2 ms·
Probably petabytes for the text index and exabytes for the whole index (including images and video). The entire textual internet is hundreds of terabytes uncomp
by hansvm 3y ago
Probably petabytes for the text index and exabytes for the whole index (including images and video). The entire textual internet is hundreds of terabytes uncompressed, and indexes tend to balloon storage size to make serving tailored queries faster and cheaper. That idea shifts a bit with images/video where vector/ML techniques dominate; you get a reduction in quality, but quality is good enough (mostly, probably, we think), and you have within a small constant multiple one way or the other of the underlying dataset size for the index size (as opposed to text, where 100x-1000x one way or another isn't uncommon).
If the index is less than petabytes then it's impossible to serve many long tail queries even just based on text content. IME those queries _have_ gotten progressively worse results the last few years, so maybe the techniques have changed, but when the money printer is "search" and a petabyte of data is a fraction of a full-time engineer in cost to store, I doubt they would have cut costs that aggressively.