Three easy reasons search engine owners /might/ pay for a full copy of the web crawl: -Faster. You don't have the latency of millions of HTTP connections, but

Three easy reasons search engine owners /might/ pay for a full copy of the web crawl: -Faster. You don't have the latency of millions of HTTP connections, but instead a single download. (Or a few dozen. Or a van full of hard drives.) -Easier. The problem of crawling quickly but politely has been handled for you. The reading of sitemaps has been handled for you. The problem of deciding how deep to crawl, and when to write off a subsite as an endless black hole, has been handled for you. Etc. -Predictable. Figuring out, in advance, how much it is going to cost you to crawl some/all of the web is, to say the least, tricky. Buying a copy with a known price tag provides a measure of certainty. Of course, I am leaving out the potential pitfalls, but the point is there /are/ arguments in favor of buying a copy of the web (and then building your own index).