7 ms·

Downloading All of Hacker News Posts and Comments

136 points by sytelus 12y ago

ivan_ah 12y ago
Awesome, this will be super useful for running ML experiments on the HN stories. Previously, Max Woolf worked on this http://minimaxir.com/2014/02/hacking-hacker-news/ http://minimaxir.com/2014/02/hacking-hacker-news/ via https://news.ycombinator.com/item?id=7291531 https://news.ycombinator.com/item?id=7291531
- minimaxir 12y ago
  I did, and with that, I had also included that Python code in GitHub, which uses the same implementation as the OP: https://github.com/minimaxir/hacker-news-download-all-stories https://github.com/minimaxir/hacker-news-download-all-storie... Fun fact: At 10,000 calls/hour and 1,000 objects per request, you can download all stories AND comments in less than an hour. As an aside: I tried to use ML on Hacker News stories and have had exactly zero success. (i.e. the predictive models are not statistically significantly better than the NIR)
  - jaredsohn 12y ago
    If you prefer the Python approach, I have forked minimaxir's code to work for comments: https://github.com/jaredsohn/hacker-news-download-all-comments https://github.com/jaredsohn/hacker-news-download-all-commen... I have a few other things in the works related to the hnsearch API (had them almost ready for release a few months ago but then I got distracted); this post is persuading me to finish them up soon. :)
- karpathy 12y ago
  I did similar analysis here http://karpathy.ca/myblog/?p=559 http://karpathy.ca/myblog/?p=559 but it's a little less clean presentation. I'm quite excited about this data release- there are many interesting ML models that could be trained here. One I hacked on previously on my own data was a comment ranker which uses a ranking loss to rank comments consistent with their observed order in data (which roughly reflects their number of upvotes, I believe). In principle I think it could be converted to a browser extension that gives a score for how well received your comment will be conditioned on the parent comment, as you write it in the text box. One of the main issues I ran into when I hacked a bit on it was space complexity, since you need to keep all the word embeddings (usually on order of 50-200D / word) around in memory of the extension, and there are many words.
- snippyhollow 12y ago
  I also did topic modeling (LDA) and other experiments on HN, code for fetching all posts and comments, and for topic modeling, can be found here: https://github.com/SnippyHolloW/HN_stats https://github.com/SnippyHolloW/HN_stats (lein run // look at hn.py)
  - ciex 12y ago
    What were your results? What did you do it for?
LukeB_UK 12y ago
Your "Fork me on Github" banner covers your hamburger menu icon.
howlett 12y ago
Your "Stories Download URL" is the same as the "Comments Download URL".
- nekopa 12y ago
  Easy fix, just change comments in the url to stories.
  - jaredsohn 12y ago
    I imagine the point of that comment was to suggest that the OP fixes it to make it easier for others.
- sytelus 12y ago
  Fixed. Thanks for pointing this out.
nekopa 12y ago
It's strange to see 1.3m stories and only 5.8m comments. I look forward to examining the data to see how many stories have 0 comments.
- meritt 12y ago
  Lots and lots of stories never make it anywhere near the front page. Mostly spam.
  - nekopa 12y ago
    But I think the spam is caught and doesn't register through the api...
    - minimaxir 12y ago
      Stories which are dead will not show in the API, that's correct. But the proportion of submissions that make it to the front page / get any comments is very low. (<10%)
  - ternaryoperator 12y ago
    "Mostly spam." No. Most stories in "new" get zero comments and are not spam.
  - sytelus 12y ago
    Most stories are not spam, they are just posted at wrong time. To give you an example, this story itself was posted on last Friday evening PST (https://news.ycombinator.com/item?id=7825146 https://news.ycombinator.com/item?id=7825146). It got just one upvote and 0 comments. Exact same story with exact same page content on exact same domain was posted on Monday (today) afternoon PST and it got 80+ upvotes, 30+ comments and got on frontpage for more than 6 hours! A lot of stories are like this. HN ranking algorithm isn't perfect.
danso 12y ago
Danke...on my 1,000th day (which was quite a few days ago), I wanted to do an analysis of how my upvoting/submission habits changed...that is, over the 3+ years I've been on HN, did my interests in hacking and languages diversify? But in terms of what I upvoted, it's hard to tell without the entire post set whether my interests changed, or the composition of HN's submissions. Also, it's great to be able to filter through and find all of the highly-upvoted stories that I've missed out, and programmatically push them to Pinboard. Thanks for this.
SergeyHack 12y ago
Have you considered sharing the data as a torrent? The current FileDropper speed is quite low (~200KB/s) P.S. Thanks for the data, you saved a good amount of our time.
- pmalynin 12y ago
  magnet:? xt=urn:btih:A3E2200A9A99906476E2E88CA002477219A3C2C3&dn=HN&tr= udp%3a%2f%2ftracker.openbittorrent.com%3a80%2fannounce&tr=udp% 3a%2f%2ftracker.publicbt.com%3a80%2fannounce&tr=udp%3a%2f%2ftr acker.ccc.de%3a80%2fannounce
  - baddox 12y ago
    999.9 megabytes?
  - pmalynin 12y ago
    954 mb
  - DanBC 12y ago
    PLEASE add four spaces before very long unbroken strings.
voltagex_ 12y ago
> Content Blocked (content_filter_denied) > Content Category: "Suspicious" Which is an odd catch-all category. It may be a keyword match from the domain. That sucks. Anyone got the github link?
- SergeyHack 12y ago
  https://github.com/sytelus/HackerNewsData https://github.com/sytelus/HackerNewsData
simonwalton 12y ago
Great work! Does anyone know if there is anything similar for Slashdot?
- frik 12y ago
  The /. comments with its tags "Interesting", "Funny", "Insightful", "Informative", "Flamebait" can be very helpful for machine learning purposes.
deleted 12y ago
[deleted]
- voltagex_ 12y ago
  I'm seeding those, plus the combined 7z file that's linked in the same GitHub issue. Edit: scratch that, can't get any peers/DHT response - can you check if you're announcing that torrent at all?
- pmalynin 12y ago
  Seems redundant, as you can already choose what file you want in the magnet link I posted. This only harms both swarms.
bertrandom 12y ago
I uploaded it to Internet Archive: https://archive.org/details/HackerNewsStoriesAndCommentsDump https://archive.org/details/HackerNewsStoriesAndCommentsDump Not sure two giant JSON files is the best format for this, but I used jp: http://www.paulhammond.org/jp/ http://www.paulhammond.org/jp/ to browse through it.
- sytelus 12y ago
  Thank you! I was trying to do that but couldn't find information on their policy about accepting and hosting such data. I've added links to this as well as Torrent links other folks had sent.
mimighost 12y ago
Very helpful. I was looking for dataset like this recently. Now there it is. Great work.
agibsonccc 12y ago
Thanks for the dataset! I've been wanting this corpus for a while. I have a few ideas I want to run with this.
deleted 12y ago
[deleted]
sytelus 12y ago
Just wanted to start a thread on some project ideas for this data: * Discover geek friendly WordPress themes and plugins by analyzing CSS in stories posted on HN. * My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). * Find out abandonment rate of veteran users. * Find out undiscovered great stories that didn't got in to frontpage because algorithm deficiency in HN (for example, get links posted by people with 10K+ karma but without upvotes.
- guernica 12y ago
  Construct phrases like Yoda, shall I. Unmask this Identity, never you shall.
  - jacquesm 12y ago
    Oh come on George, stop it.
- jacquesm 12y ago
  > My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). I'd bet that you're not the only one who thought of that. And for less money I'd bet that someone has beat you to it.
  - Flenser 12y ago
    yup :) Except I've been doing it manually as I see comments/commenters that interest me and adding them to a greasemonkey script that decorates posts with icons identifying users and their affiliations. Does that make me evil? If I find some free time I might download this and run some extracts as sytelus suggested.
hopeless 12y ago
What licence is the HN content made available under? I'm trying to find some T & C's that I might have agreed to but I haven't them yet. I don't think you can assume that comments made here are free to copy, reuse, manipulate and republish elsewhere
naveen99 12y ago
would be nice if there was something similar for reddit.

136 points by sytelus 12y ago

Awesome, this will be super useful for running ML experiments on the HN stories. Previously, Max Woolf worked on this http://minimaxir.com/2014/02/hacking-hacker-news/ http://minimaxir.com/2014/02/hacking-hacker-news/ via https://news.ycombinator.com/item?id=7291531 https://news.ycombinator.com/item?id=7291531

minimaxir 12y ago
I did, and with that, I had also included that Python code in GitHub, which uses the same implementation as the OP: https://github.com/minimaxir/hacker-news-download-all-stories https://github.com/minimaxir/hacker-news-download-all-storie... Fun fact: At 10,000 calls/hour and 1,000 objects per request, you can download all stories AND comments in less than an hour. As an aside: I tried to use ML on Hacker News stories and have had exactly zero success. (i.e. the predictive models are not statistically significantly better than the NIR)
- jaredsohn 12y ago
  If you prefer the Python approach, I have forked minimaxir's code to work for comments: https://github.com/jaredsohn/hacker-news-download-all-comments https://github.com/jaredsohn/hacker-news-download-all-commen... I have a few other things in the works related to the hnsearch API (had them almost ready for release a few months ago but then I got distracted); this post is persuading me to finish them up soon. :)
karpathy 12y ago
I did similar analysis here http://karpathy.ca/myblog/?p=559 http://karpathy.ca/myblog/?p=559 but it's a little less clean presentation. I'm quite excited about this data release- there are many interesting ML models that could be trained here. One I hacked on previously on my own data was a comment ranker which uses a ranking loss to rank comments consistent with their observed order in data (which roughly reflects their number of upvotes, I believe). In principle I think it could be converted to a browser extension that gives a score for how well received your comment will be conditioned on the parent comment, as you write it in the text box. One of the main issues I ran into when I hacked a bit on it was space complexity, since you need to keep all the word embeddings (usually on order of 50-200D / word) around in memory of the extension, and there are many words.
snippyhollow 12y ago
I also did topic modeling (LDA) and other experiments on HN, code for fetching all posts and comments, and for topic modeling, can be found here: https://github.com/SnippyHolloW/HN_stats https://github.com/SnippyHolloW/HN_stats (lein run // look at hn.py)
- ciex 12y ago
  What were your results? What did you do it for?

LukeB_UK 12y ago

Your "Fork me on Github" banner covers your hamburger menu icon.

howlett 12y ago

Your "Stories Download URL" is the same as the "Comments Download URL".

nekopa 12y ago
Easy fix, just change comments in the url to stories.
- jaredsohn 12y ago
  I imagine the point of that comment was to suggest that the OP fixes it to make it easier for others.
sytelus 12y ago
Fixed. Thanks for pointing this out.

nekopa 12y ago

It's strange to see 1.3m stories and only 5.8m comments. I look forward to examining the data to see how many stories have 0 comments.

meritt 12y ago
Lots and lots of stories never make it anywhere near the front page. Mostly spam.
- nekopa 12y ago
  But I think the spam is caught and doesn't register through the api...
  - minimaxir 12y ago
    Stories which are dead will not show in the API, that's correct. But the proportion of submissions that make it to the front page / get any comments is very low. (<10%)
- ternaryoperator 12y ago
  "Mostly spam." No. Most stories in "new" get zero comments and are not spam.
- sytelus 12y ago
  Most stories are not spam, they are just posted at wrong time. To give you an example, this story itself was posted on last Friday evening PST (https://news.ycombinator.com/item?id=7825146 https://news.ycombinator.com/item?id=7825146). It got just one upvote and 0 comments. Exact same story with exact same page content on exact same domain was posted on Monday (today) afternoon PST and it got 80+ upvotes, 30+ comments and got on frontpage for more than 6 hours! A lot of stories are like this. HN ranking algorithm isn't perfect.

danso 12y ago

Danke...on my 1,000th day (which was quite a few days ago), I wanted to do an analysis of how my upvoting/submission habits changed...that is, over the 3+ years I've been on HN, did my interests in hacking and languages diversify? But in terms of what I upvoted, it's hard to tell without the entire post set whether my interests changed, or the composition of HN's submissions. Also, it's great to be able to filter through and find all of the highly-upvoted stories that I've missed out, and programmatically push them to Pinboard. Thanks for this.

SergeyHack 12y ago

Have you considered sharing the data as a torrent? The current FileDropper speed is quite low (~200KB/s) P.S. Thanks for the data, you saved a good amount of our time.

pmalynin 12y ago
magnet:? xt=urn:btih:A3E2200A9A99906476E2E88CA002477219A3C2C3&dn=HN&tr= udp%3a%2f%2ftracker.openbittorrent.com%3a80%2fannounce&tr=udp% 3a%2f%2ftracker.publicbt.com%3a80%2fannounce&tr=udp%3a%2f%2ftr acker.ccc.de%3a80%2fannounce
- baddox 12y ago
  999.9 megabytes?
- pmalynin 12y ago
  954 mb
- DanBC 12y ago
  PLEASE add four spaces before very long unbroken strings.

voltagex_ 12y ago

> Content Blocked (content_filter_denied) > Content Category: "Suspicious" Which is an odd catch-all category. It may be a keyword match from the domain. That sucks. Anyone got the github link?

SergeyHack 12y ago
https://github.com/sytelus/HackerNewsData https://github.com/sytelus/HackerNewsData

simonwalton 12y ago

Great work! Does anyone know if there is anything similar for Slashdot?

frik 12y ago
The /. comments with its tags "Interesting", "Funny", "Insightful", "Informative", "Flamebait" can be very helpful for machine learning purposes.

deleted 12y ago

[deleted]

voltagex_ 12y ago
I'm seeding those, plus the combined 7z file that's linked in the same GitHub issue. Edit: scratch that, can't get any peers/DHT response - can you check if you're announcing that torrent at all?
pmalynin 12y ago
Seems redundant, as you can already choose what file you want in the magnet link I posted. This only harms both swarms.

bertrandom 12y ago

I uploaded it to Internet Archive: https://archive.org/details/HackerNewsStoriesAndCommentsDump https://archive.org/details/HackerNewsStoriesAndCommentsDump Not sure two giant JSON files is the best format for this, but I used jp: http://www.paulhammond.org/jp/ http://www.paulhammond.org/jp/ to browse through it.

sytelus 12y ago
Thank you! I was trying to do that but couldn't find information on their policy about accepting and hosting such data. I've added links to this as well as Torrent links other folks had sent.

mimighost 12y ago

Very helpful. I was looking for dataset like this recently. Now there it is. Great work.

agibsonccc 12y ago

Thanks for the dataset! I've been wanting this corpus for a while. I have a few ideas I want to run with this.

deleted 12y ago

[deleted]

sytelus 12y ago

Just wanted to start a thread on some project ideas for this data: * Discover geek friendly WordPress themes and plugins by analyzing CSS in stories posted on HN. * My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). * Find out abandonment rate of veteran users. * Find out undiscovered great stories that didn't got in to frontpage because algorithm deficiency in HN (for example, get links posted by people with 10K+ karma but without upvotes.

guernica 12y ago
Construct phrases like Yoda, shall I. Unmask this Identity, never you shall.
- jacquesm 12y ago
  Oh come on George, stop it.
jacquesm 12y ago
> My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). I'd bet that you're not the only one who thought of that. And for less money I'd bet that someone has beat you to it.
- Flenser 12y ago
  yup :) Except I've been doing it manually as I see comments/commenters that interest me and adding them to a greasemonkey script that decorates posts with icons identifying users and their affiliations. Does that make me evil? If I find some free time I might download this and run some extracts as sytelus suggested.

hopeless 12y ago

What licence is the HN content made available under? I'm trying to find some T & C's that I might have agreed to but I haven't them yet. I don't think you can assume that comments made here are free to copy, reuse, manipulate and republish elsewhere

naveen99 12y ago

would be nice if there was something similar for reddit.