We said the same thing, in one way or another, about indexing and searching the web -- privacy/anonymity, spam/seo, algorithms, bias, corporations.... So today

We said the same thing, in one way or another, about indexing and searching the web -- privacy/anonymity, spam/seo, algorithms, bias, corporations.... So today one can build their own crawler and search engine with great opensource software, yet a vast majority is querying a web index through third parties as the computing power required (well, mostly storage and networking) remains brutal, and then all those nitty-picketty details about crawling one needs to painstakingly figure out. Even if by Moore's law we get that great power in our own computers, enough to run a today's chatGPT clone, there will always be a commercial version that is 100x more awesome out there built by actors with immense compute power backed by rivers of corporate money.