I did my Ph.D. in information retrieval and had studied the workings of the 3 major search engines from the inside during various points about a decade ago. Wi

I did my Ph.D. in information retrieval and had studied the workings of the 3 major search engines from the inside during various points about a decade ago. Without giving away any trade secrets, part of it is the power of search behavior data (for example, what you click on after a search, how long before you go back to the search page, etc.). The more popular your search engine, the more search behavior data you have, which makes the search results better especially for tail queries (the less common searches). So then you attract more searchers, and it's a cycle that is hard for a newcomer to break into. It's why academia has given up on competing with commercial search engines in the web space, and why alternative search engines are just begging at the table right now.