4 ms·
I think the words "minimally viable" are being ignored here. I think OP's point is, assume you only have 10/100 terabytes of space and limited compute ability
by Closi 5y ago
I think the words "minimally viable" are being ignored here.
I think OP's point is, assume you only have 10/100 terabytes of space and limited compute ability - how would you approach the problem? I assume 90% of google's searches probably come from less than 1% of their total index, not to mention that Google is also keeping full cached versions of the whole website including images.
- adtac 5y agoI just eyeballed my browser history from the last 2-3 days and I'd estimate 15% is current/latest news related, some 25% is programming related, 15% is e-commerce stuff, the rest random crap. I'd imagine 10-100 TB can easily serve all of _my_ search space (even the links I didn't look at on page 10) from the past few years, but that's the thing -- it's just my search space. How do you serve the rest of the world? I wish I knew the answer :)
- foxfluff 5y agoWell Google doesn't know the answer either. Results are complete trash when you try find something niche or in a local language. The challenge isn't to index the entire web, it's to index the useful parts of it, and I think an index covering most of the useful web can be seeded quite easily with some community effort.
- deleted 5y ago[deleted]