5 ms·

Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 m

by mlhpdx 7mo ago

Static web content and dynamic data? > The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed. That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.

xandrius 7mo ago
I don't get what you meant with this comment.
- john_strinlai 7mo ago
  the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago. they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.
  - voxic11 7mo ago
    No that is the date at which the bulk archive ends and the 5 minute update files begin, so it should not be updated.
    - deleted 7mo ago
      [deleted]
voxic11 7mo ago
That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself." So to get all the data you need to grab the archive and all the 5 minute update files. archive data is here https://huggingface.co/datasets/open-index/hacker-news/tree/main/data https://huggingface.co/datasets/open-index/hacker-news/tree/... update files are here (I know that its called "today" but it actually includes all the update files which span multiple days at this point) https://huggingface.co/datasets/open-index/hacker-news/tree/main/today https://huggingface.co/datasets/open-index/hacker-news/tree/...
- john_strinlai 7mo ago
  >if you just would finish reading the paragraph probably uncalled for
  - fatty_patty89 7mo ago
    not really since original comment completely missed it
    - john_strinlai 7mo ago
      not to be "that guy" but it is pretty explicitly laid out in the guidelines, with an example and everything
      - mpalmer 7mo ago
        Then surely "little bit depressing this is still how we do things" is equally unwelcome
        john_strinlai 7mo ago
        you are certainly free to say that under the top-level comment with that quote. or email the mods about it. im not going to stop you.
- mlhpdx 7mo ago
  That paragraph doesn’t make it clear (to me) that it’s a snapshot with incremental updates. If that’s what it is. Sorry if my obtuse read offended. I just figured it was edge cached HTML, and less likely it was actually broken.
tamnd 7mo ago
That’s a silly bug in the "dynamic" README, fixing it now.

by mlhpdx 7mo ago

xandrius 7mo ago

I don't get what you meant with this comment.

john_strinlai 7mo ago
the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago. they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.
- voxic11 7mo ago
  No that is the date at which the bulk archive ends and the 5 minute update files begin, so it should not be updated.
  - deleted 7mo ago
    [deleted]

voxic11 7mo ago

That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself." So to get all the data you need to grab the archive and all the 5 minute update files. archive data is here https://huggingface.co/datasets/open-index/hacker-news/tree/main/data https://huggingface.co/datasets/open-index/hacker-news/tree/... update files are here (I know that its called "today" but it actually includes all the update files which span multiple days at this point) https://huggingface.co/datasets/open-index/hacker-news/tree/main/today https://huggingface.co/datasets/open-index/hacker-news/tree/...

john_strinlai 7mo ago
>if you just would finish reading the paragraph probably uncalled for
- fatty_patty89 7mo ago
  not really since original comment completely missed it
  - john_strinlai 7mo ago
    not to be "that guy" but it is pretty explicitly laid out in the guidelines, with an example and everything
    - mpalmer 7mo ago
      Then surely "little bit depressing this is still how we do things" is equally unwelcome
      - john_strinlai 7mo ago
        you are certainly free to say that under the top-level comment with that quote. or email the mods about it. im not going to stop you.
mlhpdx 7mo ago
That paragraph doesn’t make it clear (to me) that it’s a snapshot with incremental updates. If that’s what it is. Sorry if my obtuse read offended. I just figured it was edge cached HTML, and less likely it was actually broken.

tamnd 7mo ago

That’s a silly bug in the "dynamic" README, fixing it now.