@pg I've started checking out Gnip, http://www.gnipcentral.com http://www.gnipcentral.com. Their API makes it very easy for sites to publish their data to the g

@pg I've started checking out Gnip, http://www.gnipcentral.com http://www.gnipcentral.com. Their API makes it very easy for sites to publish their data to the gnip service. They are pretty hardcore about "hit us as much as you want" to get data. Twitter, Digg, Flickr, and a number of other services are publishing to them. Might be a good way for HN to avoid getting hammered by crawlers and still let the hacker-types slurp the data.