The 2005 Google model only made sense in the 2005 internet. Google had the luck to become a search monopoly, and they quickly created Chrome to ensure that no o

The 2005 Google model only made sense in the 2005 internet. Google had the luck to become a search monopoly, and they quickly created Chrome to ensure that no one would ever switch away from Google search, so they could maintain the monopoly. Now that Google exists, you can't create another one. There's only room for one. Another thing is the rise of "content sites", like this one (Hacker News). I'm sure YCombinator doesn't like getting hit by dozens of crawlers. The impulse to ban everything that crawls except (Google|Bing|Baidu|VK) is too great. A lot of alternative suggestions are being thrown into this discussion. Let me throw in mine: Reverse the concept of the "crawler". Instead of following links around the internet randomly, require sites to register with you and request to be crawled and/or submit a sitemap. It would be hard to get started, but once something like this gained momentum, I believe that there's room for several of these reverse-search-engines to compete.