5 ms·
FYI, the new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in
by minimaxir 13y ago
FYI, the new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in less than 2 hours without breaking the API request limit.
https://hn.algolia.com/api https://hn.algolia.com/api
- jrussino 13y ago"new Hacker News API" -> do you mean there's now an official API? I must have missed this news. Can you provide a link / additional info?
- sillysaurus3 13y agothe new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in less than 2 hours without breaking the API request limit. Which API? https://www.google.com/search?q=hacker+news+api https://www.google.com/search?q=hacker+news+api
- jared314 13y agoWith hnsearch.com being shutdown, I believe he is referring to: http://hn.algolia.com/api http://hn.algolia.com/api
- sillysaurus3 13y agoUsing that API, you could download 48,000 hacker news stories in 2 days, so if there have been less than 48,000 submissions, then what minimaxir said is true. But first you'd need to generate a list of all 48,000 story ids, and there seems to be no way to actually do that.
- minimaxir 13y agoNo need to generate all IDs beforehand. The search_by_date endpoint is fine. You have to paginate using the created_at_i parameter, not the page parameter. Also, you can set hitsPerPage = 1000. ;)
- sillysaurus3 13y agoWhen trying to access page 2 via that endpoint: http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=1000&page=2 http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsP... "you can only fetch the 1000 hits for this query, contact us to increase the limit" It was a nice try, but it did seem too good to be true.
- minimaxir 13y agoYou can paginate using the created_at_i parameter (edited OP) Just pass created_at_i<X, where X is the time stamp of the earliest submission. I was able to download 500k stories (i.e. about half of HN's 1.26M stories) before I ran into memory issues; I've fixed them and am downloading the rest.
- sillysaurus3 13y agoThat's incredible. Will you upload the raw database somewhere, please? If you make a torrent, I'll help seed it. Would you email me at sillysaurus3@gmail.com whenever it's ready?
- deleted 13y ago[deleted]
- deft 13y agoThis is old so what you're suggesting wouldn't be possible unless he updated it (which I don't think he has any reason to do).
- e15ctr0n 13y agoThere's a list of all the apps that have been built based on this API: http://hn.algolia.com/cool_apps http://hn.algolia.com/cool_apps
- pak 13y agoYup, I've switched over my Chrome extension [1] to use Algolia's API instead of HNSearch (which is shutting down), and so far, it seems to be working peachy. [1]: https://chrome.google.com/webstore/detail/hacker-news-sidebar/ngljhffenbmdjobakjplnlbfkeabbpma?hl=en-US https://chrome.google.com/webstore/detail/hacker-news-sideba...
- ehsanu1 13y agoWas this there when you posted?: RATE LIMITS We are limiting the number of API requests from a single IP to 1000 per hour. If you or your application has been blacklisted and you think there has been an error, please contact us.
- minimaxir 13y agoYes, that was there, and the math is still correct. You can query 1,000 stories per request, and do 1,000 requests per hour. That's 1M requests per hour. There are 1.26M Hacker News stories indexed by the API. :) EDIT: Finished downloading all the entries (and can confirm that 1.26M is indeed all of them). Took 3 hours due to a conservative wait period between each request to make sure I stayed within the limits.
- tripzilch 13y agoHow big is that dataset? Can you share it somehow?
- karangoeluw 13y agoI have written an API for HN: - Python module: https://github.com/karan/HackerNewsAPI https://github.com/karan/HackerNewsAPI - REST API: https://github.com/karan/HNify https://github.com/karan/HNify