Having one crawler would not be desirable as it'd involve choosing which pages get crawled and how often for freshness. Point taken about sites being less avail

Having one crawler would not be desirable as it'd involve choosing which pages get crawled and how often for freshness. Point taken about sites being less available to crawl due to the bot whitelist approach but it is not a huge issue, though some large sites like Facebook go for a whitelist only approach. Regarding those other data sources, the more privacy conscious engines would perhaps object to particular kinds of data but agreed that the kind of scale and various sources that Google has at its disposal, does give it an advantage when it comes to search. There are alternatives beyond Google and Bing who do index the web albeit at the moment are smaller in scale, and likely outnumbered by all the Bing clones. There is a high barrier to entry wrt cost but the technical/capital hurdles of crawling are not so high as compared to ranking and indexing.