He's really talking about an API. The NYTimes has one. I do think there's a business model there. What if the NYTimes shut out Google in its robots.txt file

He's really talking about an API. The NYTimes has one. I do think there's a business model there. What if the NYTimes shut out Google in its robots.txt file but then offered up full "firehose" access to it's content via the api for a price? Nobody pays for content. Everybody pays for data. Publishers should build solid APIs, shut out the spiders, and then charge search engines, aggregators etc. for full, rights-free access to their stuff.