3 ms·

Can't speak for pg, but you should probably follow the site's robots.txt file's rules: http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots

by yahelc 15y ago

Can't speak for pg, but you should probably follow the site's robots.txt file's rules: http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt It sets a crawl-delay of 30 seconds.

FaceKicker 15y ago
wget does this automatically, in case grandparent doesn't know. I'm not going to try it (because I'm not going to analyze the results myself, so it would be wasteful), but I think "wget -r news.ycombinator.com" would take care of this... It will not visit external links without the --span-hosts parameter (I believe).
- deleted 15y ago
  [deleted]
__rkaup__ 15y ago
How come that comes up with a Content-Type: text/html header?