5 ms·
There seems to be a common misconception that HDF5 somehow isn't chunked like zarray is... HDF5 will happily chunk things internally and treat them like a sing
by jofer 7y ago
There seems to be a common misconception that HDF5 somehow isn't chunked like zarray is... HDF5 will happily chunk things internally and treat them like a single large dataset. It's an index of chunks under the hood. It's _very_ much meant to encourage and allow chunking large datasets transparently.
The big advantage of zarray is when reading/writing to a cloud bucket, where having a single file containing chunks/indicies is impractical and you'd rather work with direct references to objects in a bucket.
Otherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support. If you're working in a situation where it makes sense to have things on disk or some sort of NFS share, use HDF5. If you're working with objects in a cloud bucket, you'll incur additional overhead with HDF5, as you'll have to read its table of indices, then make range requests to each chunk. Zarr is optimized for the cloud use case.
- itamarst 7y agoThe article, at least, doesn't say anything about HDF5 not be chunked. I recommended Zarr on the basis that threading seems to work much better and compression options are better.
- marsokod 7y agoHow is compression better with zarr? Pytables has been supporting quite a lot of compression format for hdf5 for quite a while. I've been using it with zstandard without any issue so far.
- jofer 7y agoThe article doesn't say HDF5 isn't chunked, but that's the impression a reader might get if they don't already know better. It's certainly not the author's fault -- just an easy mistake for a reader to make with the flow of the article. It's more that I keep hearing that statement ("Zarr is chunked and HDF5 isn't") in the wild a lot. As for threading, yeah, HDF5 can be a bit cumbersome there. I'm not sure I'd agree on compression, though. HDF5 supports fully arbitrary compression, after all, it's just the client reading the data also needs to have the compression filter you're using installed. Zarr is often used with BLOSC, and that was originally developed specifically for HDF5, FWIW.
- itamarst 7y agoI'm the original author. If the reader might get the wrong impression then I need to fix the text, that's my fault not theirs. So I'll do that. My impression from the talk linked in the article was that HDF5 doesn't do BLOSC by default, and that in general it's much easier to plug new things in, add caching, etc..
- deleted 7y ago[deleted]
- MichaelSalib 7y agoOtherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support. Absolutely not. HDF5 is an awful format with terrible implementations. For example, try writing a python program with multiple threads where each thread writes to a different HDF5 file. This should just work -- there's no concurrent access. And yet it doesn't because HDF5 implementations are piles of ancient C code that use lots of global state. There's no technical reason for this; one could easily store all the state needed in a per-file object. But back in the day, software eng standards were lower (especially for scientists) and HDF5 changes at a glacial place. I've been bitten by this particular bug, but you really have to wonder: given how poorly it speaks to the software engineering behind HDF5 implementations, what else is broken in the code or specifications? If you're working in a situation where it makes sense to have things on disk or some sort of NFS share, use HDF5. If you're working with objects in a cloud bucket, you'll incur additional overhead with HDF5, as you'll have to read its table of indices, then make range requests to each chunk. Zarr is optimized for the cloud use case. When last I looked, there were no open source HDF5 implementations that were smart enough to do range requests to cloud hosted hdf files. Has this changed?
- benibela 7y agoI thought you could not write parallel anything in Python because of the GIL
- MichaelSalib 7y agoNo, you can certainly write in parallel, despite the GIL. The GIL makes this inefficient if your work is CPU-bound, but for IO-bound workloads it can be fine.
- xen0 7y agoBut the HDF5 library does not really support multi-threading at all. Compiling the library with the "threading" option just locks around every API call, so you're back to a single thread whenever you enter (compiling without it will just crash your program). And the library does quite a lot of work when you call into it; chunk lookup, decompression, and type conversions all happen behind that lock. You can use the "direct chunk access" functions (H5Dread_chunk?) to bypass a lot of that work and do it yourself, so you get back to using multiple threads again, and that can be a big win, but having to do it sucks, and I don't think h5py exposes this functionality at all.