4 ms·
HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/mov
by shoyer 11y ago
HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/ http://cyrille.rossant.net/moving-away-hdf5/
My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain.
This is especially annoying when using HDF5's built-in compression, which hogs a lot of CPU. Inter-process communication is slower than reading from SSDs, so that isn't a great alternative:
http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidt...
There's a lot to be said for file formats that you can simply memory map, and that's exactly what Feather/Arrow are. Building out-of-core workflows on top of should be a joy.
Wes -- does the Python library for Feather already release the GIL?
^ you can use and/or compile HDF5 with a global lock, but the underlying library still isn't thread safe.
- masklinn 11y ago> Wes -- does the Python library for Feather already release the GIL? Checking out the code: https://github.com/wesm/feather/blob/master/python/feather/libfeather.pxd#L40 https://github.com/wesm/feather/blob/master/python/feather/l... So the actual reading from and writing to disk should be with a released GIL if I'm reading this correctly. The conversion to and from arrays or dataframes holds the GIL.
- wesm 11y agoConcurrent writes would not be possible but parallel reads definitely are. Haven't put any multithreading work into the library yet but it shouldn't be a large task.