No doubt Google will come up with a way for search engines to authenticate to the sites they are spidering, which will be just "Open" enough that all their comp

No doubt Google will come up with a way for search engines to authenticate to the sites they are spidering, which will be just "Open" enough that all their competitors can implement it for free (but not have any control over the evolution of the technology). Sites would just have to decide which search engines they want to allow to spider them, which will probably be a hardcoded list in a webserver config file somewhere, with the associated public keys. If we're lucky, webserver package maintainers will try to keep these lists up to date, and Google may even contribute some funds for the development effort, doing just enough to stop this system from triggering an anti-trust investigation.