3 ms·

There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files

There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files no other library understands in an intuitive way. They mostly recommend "to_csv" with all the usual problems of CSV. NumPy does have an official documented format which is straightforward to read and write from other languages if you write some code yourself, but still doesn't have very wide support outside of Python.

dijksterhuis 3y ago
Parquet is a binary file format and is very widely supported. I think there’s even an Excel plug-in for parquet these days.
- radarsat1 3y ago
  > I think there’s even an Excel plug-in for parquet these days. While CSV comes with plenty of known problems, its one advantage, which apparently remains a significant one, is that it doesn't require any plugins or special software support at all, just a text editor. It seems that's a big enough feature that no attempt to fully replace it has really worked out. Amazing when you think about it, the power of plain text encoding. Of course I agree that it's often not the best choice but somehow it remains incredibly useful and can even be hard to make arguments against it on a team project because it just kind or works well enough in a lot of cases and is simply a lowest common denominator when people can't agree on what format to use.
benglish11 3y ago
Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.
- aldanor 3y ago
  The sole fact that pandas, while trying to catch up to polars, now sort-of supports arrow as it's in-memory backend doesn't make it a "pandas dataframe binary format" in any possible way.
  - ayhanfuat 3y ago
    The whole point of Arrow is that it is language agnostic; so of course it is not a "pandas binary format". However, pandas didn't start supporting Arrow "to try to catch up to polars". The creator of pandas, Wes McKinney, came up with the Arrow idea because of the interoperability limitations he came across while working with pandas and the big data ecosystem. Supporting Arrow has been in the works for a long time: https://wesmckinney.com/blog/pandas-and-apache-arrow/ https://wesmckinney.com/blog/pandas-and-apache-arrow/ Pandas is an older and larger library with less maneuver capability and millions of users relying on things not being broken.

dijksterhuis 3y ago

Parquet is a binary file format and is very widely supported. I think there’s even an Excel plug-in for parquet these days.

radarsat1 3y ago
> I think there’s even an Excel plug-in for parquet these days. While CSV comes with plenty of known problems, its one advantage, which apparently remains a significant one, is that it doesn't require any plugins or special software support at all, just a text editor. It seems that's a big enough feature that no attempt to fully replace it has really worked out. Amazing when you think about it, the power of plain text encoding. Of course I agree that it's often not the best choice but somehow it remains incredibly useful and can even be hard to make arguments against it on a team project because it just kind or works well enough in a lot of cases and is simply a lowest common denominator when people can't agree on what format to use.

benglish11 3y ago

Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.

aldanor 3y ago
The sole fact that pandas, while trying to catch up to polars, now sort-of supports arrow as it's in-memory backend doesn't make it a "pandas dataframe binary format" in any possible way.
- ayhanfuat 3y ago
  The whole point of Arrow is that it is language agnostic; so of course it is not a "pandas binary format". However, pandas didn't start supporting Arrow "to try to catch up to polars". The creator of pandas, Wes McKinney, came up with the Arrow idea because of the interoperability limitations he came across while working with pandas and the big data ecosystem. Supporting Arrow has been in the works for a long time: https://wesmckinney.com/blog/pandas-and-apache-arrow/ https://wesmckinney.com/blog/pandas-and-apache-arrow/ Pandas is an older and larger library with less maneuver capability and millions of users relying on things not being broken.