7 ms·

Both Wes and I (project authors) will be tracking this thread in case you have questions!

by hadley 11y ago

_Wintermute 11y ago
As one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?
- peatmoss 11y ago
  I mentioned it elsewhere in the thread, but libdataframe.c seems like a pretty natural continuation to me. EDIT: What I DON'T know is how much libdataframe would look than libsqlite.
- hadley 11y ago
  Yes! Anything that is mostly C/C++ for performance could be shared between R and Python now that we have an easy interchange format.
maxerickson 11y ago
In discussing Apache Arrow in the context of Python and R, we wanted to see if we could use the insights from feather to design a very fast file format for storing data frames that could be used by both languages. Thus, the Feather format was born. Is the feather in insights from feather the right word? It reads awkwardly to me, which could just be me lacking context.
- hadley 11y ago
  Should be insights from _arrow_
data_scientist 11y ago
Great idea! Some questions: - Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather? - Feather is "not for long term data storage". Will it be standardize in a distant future? - Do you plan to integrate it into Pandas?
- wesm 11y ago
  Feather doesn't support a number of pandas features (but these are features that R doesn't really support, either): https://github.com/wesm/feather/blob/master/python/README.md https://github.com/wesm/feather/blob/master/python/README.md (not comprehensive)
- hadley 11y ago
  Feather currently doesn't support recursive/hierarchical data structures, like lists in R. That'll be added in the future though. We'll definitely standardise in the future so you can feel confident using it in the long term. I have no plans to integrate it with pandas, but I'm sure Wes does ;)
  - infinite8s 11y ago
    Will you take a storage approach for nested data similar to Parquet?
deleted 11y ago
[deleted]

Both Wes and I (project authors) will be tracking this thread in case you have questions!

by hadley 11y ago

Both Wes and I (project authors) will be tracking this thread in case you have questions!

As one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?

peatmoss 11y ago
I mentioned it elsewhere in the thread, but libdataframe.c seems like a pretty natural continuation to me. EDIT: What I DON'T know is how much libdataframe would look than libsqlite.
hadley 11y ago
Yes! Anything that is mostly C/C++ for performance could be shared between R and Python now that we have an easy interchange format.

maxerickson 11y ago

In discussing Apache Arrow in the context of Python and R, we wanted to see if we could use the insights from feather to design a very fast file format for storing data frames that could be used by both languages. Thus, the Feather format was born. Is the feather in insights from feather the right word? It reads awkwardly to me, which could just be me lacking context.

hadley 11y ago
Should be insights from _arrow_

data_scientist 11y ago

Great idea! Some questions: - Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather? - Feather is "not for long term data storage". Will it be standardize in a distant future? - Do you plan to integrate it into Pandas?

wesm 11y ago
Feather doesn't support a number of pandas features (but these are features that R doesn't really support, either): https://github.com/wesm/feather/blob/master/python/README.md https://github.com/wesm/feather/blob/master/python/README.md (not comprehensive)
hadley 11y ago
Feather currently doesn't support recursive/hierarchical data structures, like lists in R. That'll be added in the future though. We'll definitely standardise in the future so you can feel confident using it in the long term. I have no plans to integrate it with pandas, but I'm sure Wes does ;)
- infinite8s 11y ago
  Will you take a storage approach for nested data similar to Parquet?

deleted 11y ago

[deleted]