7 ms·
Where can I get a data dump of all HN comments?
I'm looking for a data dump of HN comments. It looks like Hacker News API[0] had one and disabled it. I haven't been able to find any other. It doesn't really need to be up to date -- anything in the last ~3 years would be fine.
[0] http://api.ihackernews.com/
- acosmism 13y agoScrape it
- ivan_ah 13y agoSurely there's a better way than hitting the HN servers O(7169040) times... Does anyone have an old scrape they can post somewhere or as a torrent? Note, for my purposes bag-of-words (i.e. word counts) would be enough.
- rahimnathwani 13y agobag-of-words (i.e. word counts) would be enough Are you doing sentiment analysis?
- ivan_ah 13y agoSorry for the delay in replying: I didn't see your comment. I'm working on topic modelling (Latent Dirichlet Allocation in particular). It's for a business, but I also want to solve one of my problems: reading HN addiction ;)
- tptacek 13y agoI think you just suggested that scraping HN was a constant time operation.
- deleted 13y ago[deleted]
- GrahamsNumber 13y ago10/10 would scrape
- sheff 13y agoInstead of scraping HN directly, have you looked at the HN Search API, from the old HN search engine ? ( https://www.hnsearch.com/api https://www.hnsearch.com/api ) The current search engine looks like it also has an API ( https://hn.algolia.com/api https://hn.algolia.com/api ). Presumably they have a full dataset on hand, which you may be able to get if you contact them.