3 ms·

we should do one for hacker news too.

by shaohua 13y ago

minimaxir 13y ago
It's much harder to get data from Hacker News, as HN doesn't have a public API (and HNSearch's API limits make it infeasable)
- jaredsohn 13y ago
  >(and HNSearch's API limits make it infeasable) I searched around and couldn't find information on API limits. Are these documented or just found via experiencing them? The Hacker News karma tracker posted this past January seems to be able to get the information that would be necessary for writing something like this.
  - minimaxir 13y ago
    HNSearch has a limitation of 1000 maximum results returned (see [1]). You can take a snapshot of the Front Page / New Posts every so often a la [2], but it's limited per the robots.txt. It's possible, but more time intensive (as in the linked post) [1]: http://api.thriftdb.com/api.hnsearch.com/items/_search?q=medium.com&limit=100&filter[fields][domain]=medium.com&start=1000 http://api.thriftdb.com/api.hnsearch.com/items/_search?q=med... [2]: http://mayank.lahiri.me/writing/hackernews/index.html http://mayank.lahiri.me/writing/hackernews/index.html
    - jaredsohn 13y ago
      Thanks for the information (including the links.) Based on other comments on this post, it looks like HNSearch and Reddit APIs have the same limitations, so doing something like this should be feasible for Hacker News. Also, another post indicates that the data for Reddit was done with only single 1000-limit queries, so the time-intensive method shouldn't be needed to produce equivalent results (although it could provide more accurate results.) Finally, for those looking for something like this for Hacker News right now, check out these sites: http://www.hntrends.com/ http://www.hntrends.com/, http://hntrends.jerodsanto.net/ http://hntrends.jerodsanto.net/, Hacker News hiring trends (https://github.com/adamw523/hackernewshires https://github.com/adamw523/hackernewshires)

3 ms·

we should do one for hacker news too.

by shaohua 13y ago

we should do one for hacker news too.

minimaxir 13y ago
It's much harder to get data from Hacker News, as HN doesn't have a public API (and HNSearch's API limits make it infeasable)
- jaredsohn 13y ago
  >(and HNSearch's API limits make it infeasable) I searched around and couldn't find information on API limits. Are these documented or just found via experiencing them? The Hacker News karma tracker posted this past January seems to be able to get the information that would be necessary for writing something like this.
  - minimaxir 13y ago
    HNSearch has a limitation of 1000 maximum results returned (see [1]). You can take a snapshot of the Front Page / New Posts every so often a la [2], but it's limited per the robots.txt. It's possible, but more time intensive (as in the linked post) [1]: http://api.thriftdb.com/api.hnsearch.com/items/_search?q=medium.com&limit=100&filter[fields][domain]=medium.com&start=1000 http://api.thriftdb.com/api.hnsearch.com/items/_search?q=med... [2]: http://mayank.lahiri.me/writing/hackernews/index.html http://mayank.lahiri.me/writing/hackernews/index.html
    - jaredsohn 13y ago
      Thanks for the information (including the links.) Based on other comments on this post, it looks like HNSearch and Reddit APIs have the same limitations, so doing something like this should be feasible for Hacker News. Also, another post indicates that the data for Reddit was done with only single 1000-limit queries, so the time-intensive method shouldn't be needed to produce equivalent results (although it could provide more accurate results.) Finally, for those looking for something like this for Hacker News right now, check out these sites: http://www.hntrends.com/ http://www.hntrends.com/, http://hntrends.jerodsanto.net/ http://hntrends.jerodsanto.net/, Hacker News hiring trends (https://github.com/adamw523/hackernewshires https://github.com/adamw523/hackernewshires)