4 ms·
When trying to access page 2 via that endpoint: http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=1000&page=2 http://hn.algolia.com/api/v1/sear
by sillysaurus3 13y ago
When trying to access page 2 via that endpoint: http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=1000&page=2 http://hn.algolia.com/api/v1/search_by_date?tags=story&hitsP...
"you can only fetch the 1000 hits for this query, contact us to increase the limit"
It was a nice try, but it did seem too good to be true.
- minimaxir 13y agoYou can paginate using the created_at_i parameter (edited OP) Just pass created_at_i<X, where X is the time stamp of the earliest submission. I was able to download 500k stories (i.e. about half of HN's 1.26M stories) before I ran into memory issues; I've fixed them and am downloading the rest.
- sillysaurus3 13y agoThat's incredible. Will you upload the raw database somewhere, please? If you make a torrent, I'll help seed it. Would you email me at sillysaurus3@gmail.com whenever it's ready?