5 ms·

Ask HN: Is there a database dump of HN?

10 points by daenz 7y ago

I'm interested in running some analysis on posts/comments, and I would prefer not to scrape the site with a bot. Does HN offer a periodic database dump? If not, can it be considered? I'm sure I'm not the only one who would be interested in it.

verdverm 7y ago
BigQuery has a copy you can run queries against. One of their provided, shared datasets.
- daenz 7y ago
  This seems to be the dataset you are referring to https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news?pli=1 https://console.cloud.google.com/marketplace/details/y-combi... Hasn't been updated in awhile but better than nothing.
  - verdverm 7y ago
    yeah, that's the one, i wondered about updates
  - harrisreynolds 7y ago
    It is fairly up to date. Last stories were added on 3/3/2020 and today is 3/31/2020. Not sure how frequently it updates... maybe monthly? See screenshot here. https://www.dropbox.com/s/6h7qk04s0b4mggj/Screenshot%202020-03-31%2014.09.14.png?dl=0 https://www.dropbox.com/s/6h7qk04s0b4mggj/Screenshot%202020-...
    - daenz 7y ago
      Oh cool, thanks. I mis-interpreted the "last updated" date on the BigQuery dataset page.
itronitron 7y ago
hn.algolia.com has an API for getting JSON records... https://hn.algolia.com/api https://hn.algolia.com/api