However, you need the correct subset of the 15 billion pages, which is only fractionally easier, and in some ways harder: google can just grab everything, and

However, you need the correct subset of the 15 billion pages, which is only fractionally easier, and in some ways harder: google can just grab everything, and then pull semantics out. If you want to leverage your limited domain, you need to be able to be able to have semantics in your indexer/crawler, otherwise you're going to end up having to index everything anyway. One exception, of course, is in genuinely finite-domain search engines, like Octopart. There, you know exactly where to send your indexer, so you can be very efficient.