Search engines have been coming up lately, so maybe this is a good a place as any to discuss some back of the envelope calculations. Let's say we wanted to rec

Search engines have been coming up lately, so maybe this is a good a place as any to discuss some back of the envelope calculations. Let's say we wanted to recreate the web index made by Google. How much cost and engineering would it take? Estimating the size of the web from worldwidewebsize.com [0], this is estimated at around 50 billion (5010^9). The average web page size looks to be on the order of 1.5 Mb (1.510^6). The nominal cost of hard disk space is about $0.02 / Gb [2]. So, roughly, that's 75 exabytes of data (~7510^15). At a cost of $0.02 / Gb that gives roughly $1.5M just to buy the hardware to store (a significant fraction of?) the web. The Hutter prize exists [3], so maybe there's some confidence that we only need to actually store 1/10 of that, so around $150k in costs. For perspective, that's 10 multi millionaire silicon valley types donating about $150k each, 100 "engineer types" at $15k each or 1000 to 10,000 pro-active citizens at $1.5k to $150 each (just* for the hard disk space, discounting energy, bandwidth and other operating costs). If we try to extrapolate lowering hard disk space costs and take the price halving to be about 2.5 years with a current (pessimistic?) cost of $0.02/Gb, that's about 10-15 years before a petabyte scale hard drive is available to the consumer for $1000. From my perspective, I would ask "why hasn't a decentralized search index been created and/or is in wide use?". My guess is that figuring out a robust enough system that's cheap enough is still out of reach. $150 might not seem like a lot, but you have to convince 10k people to devote energy just to search. Put another way, when does the landscape change enough so that decentralized search is a viable option? My guess is that when people can store a significant fraction of the web locally for nominal cost is the determining factor. Maybe some great compression and/or AI sentiment analysis can be done to bootstrap and maybe some type of financial incentives can help solve this issue, but my bet these will only provide a light push in the right direction and the needed technology is the underlying cheap disk space. As a side note, the worldwidewebsize.com [0] shows the number of indexed pages by Google holding pretty constant over a five year period with a sharp decline somewhere in 2020. I wonder if this is the method of estimation or if Google has changed something significant in their back end to alter their search engine and storage. [0] https://www.worldwidewebsize.com/ https://www.worldwidewebsize.com/ [1] https://www.pingdom.com/blog/webpages-are-getting-larger-every-year-and-heres-why-it-matters/#:~:text=According%20to%20the%20HTTP%20Archive,steadily%20increased%20over%20the%20years https://www.pingdom.com/blog/webpages-are-getting-larger-eve.... [2] https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ [3] http://prize.hutter1.net/ http://prize.hutter1.net/