3 ms·
I agree with you in that I cannot imagine a search startup to successfully compete with Google in the current situation, but for different reasons: I think the
by weinzierl 6y ago
I agree with you in that I cannot imagine a search startup to successfully compete with Google in the current situation, but for different reasons:
I think the index is not the biggest challenge. Yes, the internet was small when Google started and is enormous now, but there has also been a lot of concentration in the sense that the most requested content is hosted on fewer and fewer sites. Also, the expectations changed. In the beginning, the idea was to index the whole web, and search engines bragged about the size of their index. Nowadays, it's all quality over quantity, and index size is probably not the most critical factor.
Apart from that: Building a decently sized index is a major undertaking, but it is not impossible. While the web has grown, technology also has advanced, and there are multiple companies that actively maintain a search engine scale index. SEO companies, like ahrefs or majestic, do it for their purposes, and there is also Common Crawl.
The real challenge, I believe, is that Google has access to a ton of engagement data. Google Analytics, AdSense, Google Fonts, Google Public DNS, and probably other services provide Google with excellent and hard to manipulate metrics about sites' popularity.
To successfully compete with Google, a startup would have to find an ethical alternative for this. I believe that building on unethically sourced engagement data from Browsers and Browser extensions will backfire in the long run. The Brave search engine is an example of this. The best chance to succeed here have the large CDN companies, like Cloudflare, because they could harvest some of the data in a non-invading way.
> [..] and let's face it, in the English language, it's pretty much all Google with Bing a distant second
I'd argue that the non-English world is, even more, all Google because it is the only search engine that works appropriately for non-English queries as well as for local search results. Baidu in China is the only real exception to this. Even in Russian-speaking countries, Google's market share is higher than their (decent) local search engine Yandex.