3 ms·
If pg wouldn't mind me scraping the site, in a manner that would not impact the sites performance, I will write an app to collect this data and publish it to th
by d0ne 15y ago
If pg wouldn't mind me scraping the site, in a manner that would not impact the sites performance, I will write an app to collect this data and publish it to the public domain.
- yahelc 15y agoCan't speak for pg, but you should probably follow the site's robots.txt file's rules: http://news.ycombinator.com/robots.txt http://news.ycombinator.com/robots.txt It sets a crawl-delay of 30 seconds.
- FaceKicker 15y agowget does this automatically, in case grandparent doesn't know. I'm not going to try it (because I'm not going to analyze the results myself, so it would be wasteful), but I think "wget -r news.ycombinator.com" would take care of this... It will not visit external links without the --span-hosts parameter (I believe).
- deleted 15y ago[deleted]
- __rkaup__ 15y agoHow come that comes up with a Content-Type: text/html header?
- bootload 15y ago"... If pg wouldn't mind me scraping the site, in a manner that would not impact the sites performance ..." you don't have to scape the site, RTFM. The official 'hnsearch' api will most likely yield the info you need ~ http://www.hnsearch.com/api http://www.hnsearch.com/api
- d0ne 15y agoThanks for point that out. I did not know their was an api for querying data on HN.