3 ms·
The article doesn't say HDF5 isn't chunked, but that's the impression a reader might get if they don't already know better. It's certainly not the author's fau
by jofer 7y ago
The article doesn't say HDF5 isn't chunked, but that's the impression a reader might get if they don't already know better. It's certainly not the author's fault -- just an easy mistake for a reader to make with the flow of the article.
It's more that I keep hearing that statement ("Zarr is chunked and HDF5 isn't") in the wild a lot.
As for threading, yeah, HDF5 can be a bit cumbersome there.
I'm not sure I'd agree on compression, though. HDF5 supports fully arbitrary compression, after all, it's just the client reading the data also needs to have the compression filter you're using installed. Zarr is often used with BLOSC, and that was originally developed specifically for HDF5, FWIW.
- itamarst 7y agoI'm the original author. If the reader might get the wrong impression then I need to fix the text, that's my fault not theirs. So I'll do that. My impression from the talk linked in the article was that HDF5 doesn't do BLOSC by default, and that in general it's much easier to plug new things in, add caching, etc..