4 ms·

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and us

by jzwinck 11y ago

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if the Feather API is preferable?

andreasvc 11y ago
I found it quite frustrating to use HDF5. It does not handle variable-length strings well (very common). In Pandas, categoricals and MultiIndex are not supported. I found that settling for CSV and pickle is more reliable & robust. Also, HDF5 basically implements a hierarchical file system, which is overengineering IMO.
- nhatbui 11y ago
  Are you talking about a different type of MultiIndex? http://pandas.pydata.org/pandas-docs/stable/advanced.html http://pandas.pydata.org/pandas-docs/stable/advanced.html
  - andreasvc 11y ago
    That's the one. Don't recall the specifics, but combinations of MultiIndex as index and columns and the other features were not supported with HDF5.
chubot 11y ago
HDF5 apparently only has one implementation, which is ridiculously bloated: https://news.ycombinator.com/item?id=10858189 https://news.ycombinator.com/item?id=10858189
deadmutex 11y ago
HDF5 is not thread-efficient. https://www.hdfgroup.org/hdf5-quest.html#tsafe https://www.hdfgroup.org/hdf5-quest.html#tsafe So, python packages like h5py do not even try to release the GIL. This makes working with HDF5 very annoying in python (when using multiple threads).
wesm 11y ago
Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code (< 2KLOC in the core library) and a correspondingly minimal API. It's a simple file format with excellent performance, and we wanted to make it as easy as possible for people to use Feather. There is also the Apache Arrow factor -- integration between the Arrow memory representation and R and Python tools will have a lot of ecosystem benefits, so one of the goals of Feather is to reconcile Python's and R's metadata requirements with the "official" Arrow metadata so that we can move around data frames with very low overhead.
- jacobolus 11y ago
  Is there a plan to add other language implementations (or C implementation wrappers)? I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python. I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.
  - hadley 11y ago
    It's just a matter of someone implementing the protocol. It's not a huge amount of work for an experienced js/matlab programmer.
    - jacobolus 11y ago
      Is there a spec somewhere, or is the existing implementation the spec? Edit: https://github.com/wesm/feather/blob/master/doc/FORMAT.md https://github.com/wesm/feather/blob/master/doc/FORMAT.md Seems a bit sparse/incomplete still (as would be expected for a brand new project).
- IndianAstronaut 11y ago
shoyer 11y ago
HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/ http://cyrille.rossant.net/moving-away-hdf5/ My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain. This is especially annoying when using HDF5's built-in compression, which hogs a lot of CPU. Inter-process communication is slower than reading from SSDs, so that isn't a great alternative: http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidt... There's a lot to be said for file formats that you can simply memory map, and that's exactly what Feather/Arrow are. Building out-of-core workflows on top of should be a joy. Wes -- does the Python library for Feather already release the GIL? ^ you can use and/or compile HDF5 with a global lock, but the underlying library still isn't thread safe.
- masklinn 11y ago
  > Wes -- does the Python library for Feather already release the GIL? Checking out the code: https://github.com/wesm/feather/blob/master/python/feather/libfeather.pxd#L40 https://github.com/wesm/feather/blob/master/python/feather/l... So the actual reading from and writing to disk should be with a released GIL if I'm reading this correctly. The conversion to and from arrays or dataframes holds the GIL.
- wesm 11y ago
  Concurrent writes would not be possible but parallel reads definitely are. Haven't put any multithreading work into the library yet but it shouldn't be a large task.

by jzwinck 11y ago

andreasvc 11y ago

I found it quite frustrating to use HDF5. It does not handle variable-length strings well (very common). In Pandas, categoricals and MultiIndex are not supported. I found that settling for CSV and pickle is more reliable & robust. Also, HDF5 basically implements a hierarchical file system, which is overengineering IMO.

nhatbui 11y ago
Are you talking about a different type of MultiIndex? http://pandas.pydata.org/pandas-docs/stable/advanced.html http://pandas.pydata.org/pandas-docs/stable/advanced.html
- andreasvc 11y ago
  That's the one. Don't recall the specifics, but combinations of MultiIndex as index and columns and the other features were not supported with HDF5.

chubot 11y ago

HDF5 apparently only has one implementation, which is ridiculously bloated: https://news.ycombinator.com/item?id=10858189 https://news.ycombinator.com/item?id=10858189

deadmutex 11y ago

HDF5 is not thread-efficient. https://www.hdfgroup.org/hdf5-quest.html#tsafe https://www.hdfgroup.org/hdf5-quest.html#tsafe So, python packages like h5py do not even try to release the GIL. This makes working with HDF5 very annoying in python (when using multiple threads).

wesm 11y ago

Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code (< 2KLOC in the core library) and a correspondingly minimal API. It's a simple file format with excellent performance, and we wanted to make it as easy as possible for people to use Feather. There is also the Apache Arrow factor -- integration between the Arrow memory representation and R and Python tools will have a lot of ecosystem benefits, so one of the goals of Feather is to reconcile Python's and R's metadata requirements with the "official" Arrow metadata so that we can move around data frames with very low overhead.

jacobolus 11y ago
Is there a plan to add other language implementations (or C implementation wrappers)? I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python. I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.
- hadley 11y ago
  It's just a matter of someone implementing the protocol. It's not a huge amount of work for an experienced js/matlab programmer.
  - jacobolus 11y ago
    Is there a spec somewhere, or is the existing implementation the spec? Edit: https://github.com/wesm/feather/blob/master/doc/FORMAT.md https://github.com/wesm/feather/blob/master/doc/FORMAT.md Seems a bit sparse/incomplete still (as would be expected for a brand new project).
IndianAstronaut 11y ago

shoyer 11y ago

HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/ http://cyrille.rossant.net/moving-away-hdf5/ My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain. This is especially annoying when using HDF5's built-in compression, which hogs a lot of CPU. Inter-process communication is slower than reading from SSDs, so that isn't a great alternative: http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidt... There's a lot to be said for file formats that you can simply memory map, and that's exactly what Feather/Arrow are. Building out-of-core workflows on top of should be a joy. Wes -- does the Python library for Feather already release the GIL? ^ you can use and/or compile HDF5 with a global lock, but the underlying library still isn't thread safe.

masklinn 11y ago
> Wes -- does the Python library for Feather already release the GIL? Checking out the code: https://github.com/wesm/feather/blob/master/python/feather/libfeather.pxd#L40 https://github.com/wesm/feather/blob/master/python/feather/l... So the actual reading from and writing to disk should be with a released GIL if I'm reading this correctly. The conversion to and from arrays or dataframes holds the GIL.
wesm 11y ago
Concurrent writes would not be possible but parallel reads definitely are. Haven't put any multithreading work into the library yet but it shouldn't be a large task.