3 ms·
How does this differ from the Algolia HN API in terms of data access? (https://hn.algolia.com/api https://hn.algolia.com/api) I was able to download all HN data
by minimaxir 12y ago
How does this differ from the Algolia HN API in terms of data access? (https://hn.algolia.com/api https://hn.algolia.com/api) I was able to download all HN data recently with ease using that endpoint. Authentication?
EDIT: After looking at the documentation there are two new aspects of the Firebase API not in the Algolia API:
1) Ability to see deleted/dead stories.
2) Endpoint for user data.
Question to kogir/dang: Has the "delay" field (Delay in minutes between a comment's creation and its visibility to other users) always been there?
- zuck9 12y agoAlgolia scrapes HN HTML and provides their own API from what it looks like.
- jaredsohn 12y agoThey don't scrape it. From the hnsearch about page: "Every 2 minutes, the last 1000 HN items (stories, comments, polls) are sent to Algolia's indexing API. Items from the last 48 hours are refreshed every hour."
- redox_ 12y agoCorrect, and we're planning to move to the new official API ASAP (instead of the legacy one -> it was not web crawler but far from perfect) for the indexing. Regarding the REST APIs, let's keep both for now :)
- deleted 12y ago[deleted]
- dang 12y agoThe 'delay' field has been there for years. I vaguely remember pg announcing it here when he added it, so we could probably dig up the exact date.
- gwern 12y agoI'm also curious whether it removes some of the limitations of the Algolia version; I wanted to download my content for some statistical analysis (notes at http://www.gwern.net/HN http://www.gwern.net/HN ) but I discovered that it seems there's some hard limits to how much of my data I can reach: https://github.com/algolia/hn-search/pull/36 https://github.com/algolia/hn-search/pull/36
- minimaxir 12y agoThere's a cheat to work around that limit: use the created_at_i parameter. Example: https://github.com/minimaxir/hacker-news-download-all-stories/blob/master/hacker_news_getallstories.py https://github.com/minimaxir/hacker-news-download-all-storie...
- gwern 12y agoI don't know Python, so I'm not sure what your source code is doing. At a guess, you've hacked together some sort of repeated queries thing with a time-window?
- minimaxir 12y ago> Get 1000 entries and process them. > Take the timestamp of the last entry. > Requery the API, asking for articles made before that timestamp Repeat.
- jaredsohn 12y agoIf you want to get data for a single user through Algolia's API via the commandline, you could also use https://github.com/jaredsohn/hnuserdownload https://github.com/jaredsohn/hnuserdownload. It uses the same technique as minimaxir's code (a post of his was the inspiration.)
- sospep 12y ago> "Delay" field post by pg on 2008-06-29 https://news.ycombinator.com/item?id=231024 https://news.ycombinator.com/item?id=231024