3 ms·
I noticed the Hacker News dataset that was published to big query was now a year out of date. I have created an updated copy and made it available for download
by aaronhoffman 10y ago
I noticed the Hacker News dataset that was published to big query was now a year out of date.
I have created an updated copy and made it available for download.
(This is the last 10MM entries, I can add the rest if people are interested.)
- toomuchtodo 10y agoThanks for doing this Aaron! Something to consider for the future: if you use grab-site (https://github.com/ludios/grab-site https://github.com/ludios/grab-site), which is a close cousin of ArchiveTeam's ArchiveBot, it'll retrieve all of the content from HackerNews (making the requisite http request and receiving the response for each URL, all of which is stored in WARC/CDX files), and store it in WARC/CDX files which can be uploaded to the Internet Archive. Those can then be integrated into the Wayback Machine by an IA admin at a later date. Something to keep in mind!
- aaronhoffman 10y agoThanks - was not aware of grab-site, I'll check it out!