14 ms·

Awesome, this will be super useful for running ML experiments on the HN stories. Previously, Max Woolf worked on this http://minimaxir.com/2014/02/hacking-hac

by ivan_ah 12y ago

Awesome, this will be super useful for running ML experiments on the HN stories. Previously, Max Woolf worked on this http://minimaxir.com/2014/02/hacking-hacker-news/ http://minimaxir.com/2014/02/hacking-hacker-news/ via https://news.ycombinator.com/item?id=7291531 https://news.ycombinator.com/item?id=7291531

minimaxir 12y ago
I did, and with that, I had also included that Python code in GitHub, which uses the same implementation as the OP: https://github.com/minimaxir/hacker-news-download-all-stories https://github.com/minimaxir/hacker-news-download-all-storie... Fun fact: At 10,000 calls/hour and 1,000 objects per request, you can download all stories AND comments in less than an hour. As an aside: I tried to use ML on Hacker News stories and have had exactly zero success. (i.e. the predictive models are not statistically significantly better than the NIR)
- jaredsohn 12y ago
  If you prefer the Python approach, I have forked minimaxir's code to work for comments: https://github.com/jaredsohn/hacker-news-download-all-comments https://github.com/jaredsohn/hacker-news-download-all-commen... I have a few other things in the works related to the hnsearch API (had them almost ready for release a few months ago but then I got distracted); this post is persuading me to finish them up soon. :)
karpathy 12y ago
I did similar analysis here http://karpathy.ca/myblog/?p=559 http://karpathy.ca/myblog/?p=559 but it's a little less clean presentation. I'm quite excited about this data release- there are many interesting ML models that could be trained here. One I hacked on previously on my own data was a comment ranker which uses a ranking loss to rank comments consistent with their observed order in data (which roughly reflects their number of upvotes, I believe). In principle I think it could be converted to a browser extension that gives a score for how well received your comment will be conditioned on the parent comment, as you write it in the text box. One of the main issues I ran into when I hacked a bit on it was space complexity, since you need to keep all the word embeddings (usually on order of 50-200D / word) around in memory of the extension, and there are many words.
snippyhollow 12y ago
I also did topic modeling (LDA) and other experiments on HN, code for fetching all posts and comments, and for topic modeling, can be found here: https://github.com/SnippyHolloW/HN_stats https://github.com/SnippyHolloW/HN_stats (lein run // look at hn.py)
- ciex 12y ago
  What were your results? What did you do it for?

by ivan_ah 12y ago

minimaxir 12y ago

I did, and with that, I had also included that Python code in GitHub, which uses the same implementation as the OP: https://github.com/minimaxir/hacker-news-download-all-stories https://github.com/minimaxir/hacker-news-download-all-storie... Fun fact: At 10,000 calls/hour and 1,000 objects per request, you can download all stories AND comments in less than an hour. As an aside: I tried to use ML on Hacker News stories and have had exactly zero success. (i.e. the predictive models are not statistically significantly better than the NIR)

jaredsohn 12y ago
If you prefer the Python approach, I have forked minimaxir's code to work for comments: https://github.com/jaredsohn/hacker-news-download-all-comments https://github.com/jaredsohn/hacker-news-download-all-commen... I have a few other things in the works related to the hnsearch API (had them almost ready for release a few months ago but then I got distracted); this post is persuading me to finish them up soon. :)

karpathy 12y ago

I did similar analysis here http://karpathy.ca/myblog/?p=559 http://karpathy.ca/myblog/?p=559 but it's a little less clean presentation. I'm quite excited about this data release- there are many interesting ML models that could be trained here. One I hacked on previously on my own data was a comment ranker which uses a ranking loss to rank comments consistent with their observed order in data (which roughly reflects their number of upvotes, I believe). In principle I think it could be converted to a browser extension that gives a score for how well received your comment will be conditioned on the parent comment, as you write it in the text box. One of the main issues I ran into when I hacked a bit on it was space complexity, since you need to keep all the word embeddings (usually on order of 50-200D / word) around in memory of the extension, and there are many words.

snippyhollow 12y ago

I also did topic modeling (LDA) and other experiments on HN, code for fetching all posts and comments, and for topic modeling, can be found here: https://github.com/SnippyHolloW/HN_stats https://github.com/SnippyHolloW/HN_stats (lein run // look at hn.py)

ciex 12y ago
What were your results? What did you do it for?