4 ms·

Ask HN: Has Hacker News stopped uploading its dataset in 2022?

• lulzx 3y ago
  Check https://news.ycombinator.com/item?id=38781031 https://news.ycombinator.com/item?id=38781031
  • wslh 3y ago
    There is no [official] answer there. Dang [1]? [1] https://news.ycombinator.com/user?id=dang https://news.ycombinator.com/user?id=dang
    • pvg 3y ago
      https://news.ycombinator.com/item?id=38796861 https://news.ycombinator.com/item?id=38796861
      • wslh 3y ago
        If it is a matter of [human] resources I can put engineers to work on this as an open data good.
        • pvg 3y ago
          You might have missed it in all the links but the data set and its maintenance is a Google thing not an HN thing so those are the people you need to get in touch with.
          • wslh 3y ago
            So I assume your feedback is official right? I understand now that it could be an interesting contribution to have a similar database to be used, for example, for machine learning? Do you agree?
            • pvg 3y ago
              "random internet busybody" official, yes. If you want to talk to HN people about it, you can email them, I'm just pointing out their description of the situation is already quite plain and public.
• deleted 3y ago
  [deleted]
• pvg 3y ago
  Email this stuff to the mods or search the site https://news.ycombinator.com/item?id=38781031 https://news.ycombinator.com/item?id=38781031
• cldellow 3y ago
  The graveyard that is Google's issue tracker has an abandoned ticket about this: https://issuetracker.google.com/issues/261579123 https://issuetracker.google.com/issues/261579123
• jerbear4328 3y ago
  That doesn't appear to be an official HN thing, it's some third party responsible for it. I see dang has said he contacted them but can't do anything about it. The API still works. https://news.ycombinator.com/item?id=38796861 https://news.ycombinator.com/item?id=38796861
  • wslh 3y ago
    I now understand that. I just thought it was trivial for HN to export the public database buit if not it can be retrieved from the API. Are there hard rate limits to use the API? And we can assume that everyone has a license tu publish the HN public data? In general, there is a disclaimer about that. Don't want to be sued though.
• xnx 3y ago
  You can now get Hacker News data in real time from the Hacker News API powered by Firebase: https://github.com/HackerNews/API https://github.com/HackerNews/API This is great (real time!), but also kind of a pain (38+ millions individual http requests to get the whole thing). Thankfully there's no authentication or apparent rate limiting. I fumbled my way through downloading the whole thing with curl. I screwed up a few times so made over 70 million requests in total. Toy analysis of the data I downloaded here: https://public.tableau.com/app/profile/isna/viz/HackerNewsDashboard_17042353296150/HackerNewsYearinReview2023 https://public.tableau.com/app/profile/isna/viz/HackerNewsDa...
• wslh 3y ago
  After one day of my post and reviewing these answers a simple query [0] gives a lot of responses. For example [1] and [2]. [0] Google: dump all hackernews site:github.com [1] https://github.com/ashish01/hn-data-dumps https://github.com/ashish01/hn-data-dumps [2] https://github.com/iOliverNguyen/hackernews-dump https://github.com/iOliverNguyen/hackernews-dump