3 ms·
50TB on localhost is awesome! I wonder what the amount of new content created was between 2018 and 2019.
by caro_douglos 7y ago
50TB on localhost is awesome! I wonder what the amount of new content created was between 2018 and 2019.
- StillBored 7y agoBut I doubt you really need it all in RAM for a small localized single user search engine. In that case you back most of the index with NVMe which at the prices I saw on black friday is probably less than $10k using consumer grade QLC flash drives when combined with cheap x1 pcie expansion boards/etc. The 10PB of disk is also quite reachable given its possible to buy bulk 10T disks at $150 each. Bottom line, I did some of these calculations a couple years ago because I was interested in a topic based search engine that only indexed for certain topics and basically tossed any crawler results that didn't appear to fit the subject matter. So, while the web is a lot bigger than when google started, storage and compute is also a lot cheaper. A web search engine that specialized in say cooking recipes might be entirely doable on a fairly limited budget.