3 ms·
The article, at least, doesn't say anything about HDF5 not be chunked. I recommended Zarr on the basis that threading seems to work much better and compression
by itamarst 7y ago
The article, at least, doesn't say anything about HDF5 not be chunked.
I recommended Zarr on the basis that threading seems to work much better and compression options are better.
- marsokod 7y agoHow is compression better with zarr? Pytables has been supporting quite a lot of compression format for hdf5 for quite a while. I've been using it with zstandard without any issue so far.
- jofer 7y agoThe article doesn't say HDF5 isn't chunked, but that's the impression a reader might get if they don't already know better. It's certainly not the author's fault -- just an easy mistake for a reader to make with the flow of the article. It's more that I keep hearing that statement ("Zarr is chunked and HDF5 isn't") in the wild a lot. As for threading, yeah, HDF5 can be a bit cumbersome there. I'm not sure I'd agree on compression, though. HDF5 supports fully arbitrary compression, after all, it's just the client reading the data also needs to have the compression filter you're using installed. Zarr is often used with BLOSC, and that was originally developed specifically for HDF5, FWIW.
- itamarst 7y agoI'm the original author. If the reader might get the wrong impression then I need to fix the text, that's my fault not theirs. So I'll do that. My impression from the talk linked in the article was that HDF5 doesn't do BLOSC by default, and that in general it's much easier to plug new things in, add caching, etc..