The code for doing the indexing (at least by default) could be built right in to the web server, so it'd just be a matter of enabling an option in Apache or the

The code for doing the indexing (at least by default) could be built right in to the web server, so it'd just be a matter of enabling an option in Apache or the like. It would be pretty easy to verify whether or not the index is accurate with a small random sample of pages on the site, and then penalize / exclude (or do a de-novo crawl) for those sites not providing a legit index.