3 ms·
I'm also curious whether it removes some of the limitations of the Algolia version; I wanted to download my content for some statistical analysis (notes at http
by gwern 12y ago
I'm also curious whether it removes some of the limitations of the Algolia version; I wanted to download my content for some statistical analysis (notes at http://www.gwern.net/HN http://www.gwern.net/HN ) but I discovered that it seems there's some hard limits to how much of my data I can reach: https://github.com/algolia/hn-search/pull/36 https://github.com/algolia/hn-search/pull/36
- minimaxir 12y agoThere's a cheat to work around that limit: use the created_at_i parameter. Example: https://github.com/minimaxir/hacker-news-download-all-stories/blob/master/hacker_news_getallstories.py https://github.com/minimaxir/hacker-news-download-all-storie...
- gwern 12y agoI don't know Python, so I'm not sure what your source code is doing. At a guess, you've hacked together some sort of repeated queries thing with a time-window?
- minimaxir 12y ago> Get 1000 entries and process them. > Take the timestamp of the last entry. > Requery the API, asking for articles made before that timestamp Repeat.
- jaredsohn 12y agoIf you want to get data for a single user through Algolia's API via the commandline, you could also use https://github.com/jaredsohn/hnuserdownload https://github.com/jaredsohn/hnuserdownload. It uses the same technique as minimaxir's code (a post of his was the inspiration.)