I don’t think so. If there are 5 billion pages on the internet, one could have a compressed 1KB-per-page index of the entire internet on a single consumer-grade

I don’t think so. If there are 5 billion pages on the internet, one could have a compressed 1KB-per-page index of the entire internet on a single consumer-grade computer in their bedroom in 2020. Size of the web should not be an issue, in my opinion :(