35 ms·

Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback reall

by wesm 11y ago

Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code (< 2KLOC in the core library) and a correspondingly minimal API. It's a simple file format with excellent performance, and we wanted to make it as easy as possible for people to use Feather. There is also the Apache Arrow factor -- integration between the Arrow memory representation and R and Python tools will have a lot of ecosystem benefits, so one of the goals of Feather is to reconcile Python's and R's metadata requirements with the "official" Arrow metadata so that we can move around data frames with very low overhead.

jacobolus 11y ago
Is there a plan to add other language implementations (or C implementation wrappers)? I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python. I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.
- hadley 11y ago
  It's just a matter of someone implementing the protocol. It's not a huge amount of work for an experienced js/matlab programmer.
  - jacobolus 11y ago
    Is there a spec somewhere, or is the existing implementation the spec? Edit: https://github.com/wesm/feather/blob/master/doc/FORMAT.md https://github.com/wesm/feather/blob/master/doc/FORMAT.md Seems a bit sparse/incomplete still (as would be expected for a brand new project).
IndianAstronaut 11y ago
How does this contrast with the new Dask library in Python?
- wesm 11y ago
  Dask is a compute framework. So you could use dask to create lots of Feather files, then perform computations.

by wesm 11y ago

jacobolus 11y ago

Is there a plan to add other language implementations (or C implementation wrappers)? I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python. I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.

hadley 11y ago
It's just a matter of someone implementing the protocol. It's not a huge amount of work for an experienced js/matlab programmer.
- jacobolus 11y ago
  Is there a spec somewhere, or is the existing implementation the spec? Edit: https://github.com/wesm/feather/blob/master/doc/FORMAT.md https://github.com/wesm/feather/blob/master/doc/FORMAT.md Seems a bit sparse/incomplete still (as would be expected for a brand new project).

IndianAstronaut 11y ago

How does this contrast with the new Dask library in Python?

wesm 11y ago
Dask is a compute framework. So you could use dask to create lots of Feather files, then perform computations.