I would think that HN wouldn't really care about someone crawling them as long as they were respectful about it. Also, as far as writing crawlers/scrapers goes

I would think that HN wouldn't really care about someone crawling them as long as they were respectful about it. Also, as far as writing crawlers/scrapers goes, a header-handling error is one of the least annoying things you can run into.