3 ms·

you don't need to scrape HN there's the public dataset in google BigQuery, I don't know if it's still updated regularly. edit: here's the link https://console

by gryn 4y ago

you don't need to scrape HN there's the public dataset in google BigQuery, I don't know if it's still updated regularly. edit: here's the link https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news https://console.cloud.google.com/marketplace/details/y-combi... they seems to have stopped updating around the later part of 2022, don't know why.

pfarrell 4y ago
To anyone interested, I noticed this stoppage as well. IIRC, the data in BigQuery stopped being updated Nov 16, 2022. I’ve got a complete dataset that merges in new content from the firebase dataset, then rectifies story points at a later time (since stories can amass points indefinitely). Comment here if you’re interested. I’m thinking of publishing it as a torrent, but don’t know if there’s enough interest.
- malborodog 4y ago
  +1
- codetrotter 4y ago
  > Comment here if you’re interested. I’m thinking of publishing it as a torrent, but don’t know if there’s enough interest. Please do :)
  - gbertb 4y ago
    Would be interested as well!
- zxexz 4y ago
  I would love that! I find HN useful to have backed up on my laptop, for freetext searching offline mostly.
klausbreyer 4y ago
I also learned this today. Thanks for sharing! :) Edit: But sadly it is not up to date. :)