2 ms·
Ask HN: Crawling historic HN - a project proposal/request
I've been attempting to check on some older items on HN in order to do some text analysis, and I want to be fair and play by the rules. PG has said that the "official" way to do this is now via the HN Search API, but in using that I've found (a) significant gaps, and (b) it doesn't go back beyond January 2010.
So I can use that to get some data, and then attempt to fill in the gaps.
However, I'm sure other people have done this before now, and are probably sitting on previous crawls and/or downloads. So the question is, can we create a project to make all historic HN data available via an easy-to-use interface.
Thoughts, suggestions and comments invited.