I would imaging reciprosity and level playing field would require Google's robots.txt to be an open free-for-all. I mean, since we're talking about embracing th

I would imaging reciprosity and level playing field would require Google's robots.txt to be an open free-for-all. I mean, since we're talking about embracing the open web. I Googled "is Google crawlable" and similar and that doesn't seem to be a thing. I checked the robots.txt and that seems to disallow a lot of stuff. Search, news, groups, images, etc. [1] Google does a great job of taking the open web, repackaging it, slapping it up behind a (crawling) wall, extracting maximum cash from the result. It will always be able to suck in your content "think of the users!" but that'll generally be a 1-way street. Yes there are API's, but that's still a different relationship to "we crawl whatever we want, we put any content we want in our search results, because users". Unrelated: Chrome (which I use and love) also had the added benefit of reducing the amount of money they'd be paying to Firefox for the search deal. Not a terrible outcome. It might even have paid for itself right there, but that's utter speculation. [1] http://www.google.com/robots.txt http://www.google.com/robots.txt - seems to be far more disallow than allow