You do have a point. However, the internet today is very different from its early days. Its content is much more centralized. In the past, Google search result

You do have a point. However, the internet today is very different from its early days. Its content is much more centralized. In the past, Google search results will point to some good websites you never heard of and are very low traffic. But nowadays, the first couple pages are all popular websites with huge traffic even when they are low quality on the content. You can only find low traffic but good content websites in comments from social sites like reddit and HN. So it seems to me, to build a usable search engine, you only need to index most popular websites.