4 ms·
Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.
by vovavili 7mo ago
Replacing an 11.6GB Parquet file every 5 minutes strikes me as a bit wasteful. I would probably use Apache Iceberg here.
- fabmilo 7mo agoWas thinking the same thing. probably once a day would be more than enough. if you really want a minute by minute probably a delta file from the previous day should be more than enough.
- ai-inquisitor 7mo agoIt's not doing that. If you look at the repository, it's adding a new commit with tiny parquet files every 5 minutes. This recent one only was a 20.9 KB parquet file: https://huggingface.co/datasets/open-index/hacker-news/commit/96a6747a06e954a45c3e385736e762e03c9b1c0a https://huggingface.co/datasets/open-index/hacker-news/commi... and the ones before it were a median of 5 KB: https://huggingface.co/datasets/open-index/hacker-news/tree/main/today/2026/03/18/19 https://huggingface.co/datasets/open-index/hacker-news/tree/... The bigger concern is how large the git history is going to get on the repository.
- vovavili 7mo agoThis makes more sense. I still wonder if the author isn't just effectively recreating Apache Iceberg manually here.
- tomrod 7mo agoAre they paying for the repo space, I wonder?
- cyanydeez 7mo agosomeones paying to keep name dropping Iceberg(tm)
- mulmen 7mo agoWeird accusation. Iceberg is an Apache project. I don’t think anyone gets paid when you use it so not sure what the benefit of shilling would be. It is just a table format that’s well suited for this purpose. I would expect any professional to make a similar recommendation.
- tamnd 7mo agoI intentionally kept it lightweight. Just Parquet files + simple partitioning + commits on Hugging Face. That already covers most of what I need, without introducing a heavier stack or extra dependencies. Also, I wanted something that is easy to consume anywhere. With this setup, you can point DuckDB or Polars directly at the data and start querying, no catalog or special tooling required.
- btown 7mo agoI recall that this became a big problem for the Homebrew project in terms of load on the repo, to the extent that Github asked them not to recommend/default-enable shallow clones for their users: https://github.com/Homebrew/brew/issues/15497#issuecomment-1568062267 https://github.com/Homebrew/brew/issues/15497#issuecomment-1... This is likely to be lower traffic, and the history should (?) scale only linearly with new data, so likely not the worst thing. But it's something to be cognizant of when using SCM software in unexpected ways!
- roncesvalles 7mo agoHow would shallow clone be more stressful for GitHub than a regular clone?
- enchilada 7mo agoShallow clones (and the resulting lack of shared history data) break many assumptions that packfile optimisations rely on. See also: https://github.com/orgs/Homebrew/discussions/225 https://github.com/orgs/Homebrew/discussions/225
- sureglymop 7mo agoSo they are sharding by time/day? I have a similar project right now where I am scraping a dataset that is only ever offering the current state. I am trying to preserve the history of this dataset and was thinking of using the same strategy. If anyone has experience or pointers in how to best add time as a dimension to an existing generic dataset, I'd love to read about it.
- zerocrates 7mo ago"The dataset is organized as one Parquet file per calendar month, plus 5-minute live files for today's activity. Every 5 minutes, new items are fetched from the source and committed directly as a single Parquet block. At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory." So it's not really one big file getting replaced all the time. Though a less extreme variation of that is happening day to day.
- tomrod 7mo agoParquet is a very efficient storage approach. Data interfaces tend to treat paths as partitions, if logical.