6 ms·

Isn’t the Python data frame format becoming an established format? I mean, there’s already commercial products that talk it natively.

by moomin 3y ago

jzwinck 3y ago
There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files no other library understands in an intuitive way. They mostly recommend "to_csv" with all the usual problems of CSV. NumPy does have an official documented format which is straightforward to read and write from other languages if you write some code yourself, but still doesn't have very wide support outside of Python.
- dijksterhuis 3y ago
  Parquet is a binary file format and is very widely supported. I think there’s even an Excel plug-in for parquet these days.
  - radarsat1 3y ago
    > I think there’s even an Excel plug-in for parquet these days. While CSV comes with plenty of known problems, its one advantage, which apparently remains a significant one, is that it doesn't require any plugins or special software support at all, just a text editor. It seems that's a big enough feature that no attempt to fully replace it has really worked out. Amazing when you think about it, the power of plain text encoding. Of course I agree that it's often not the best choice but somehow it remains incredibly useful and can even be hard to make arguments against it on a team project because it just kind or works well enough in a lot of cases and is simply a lowest common denominator when people can't agree on what format to use.
- benglish11 3y ago
  Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.
  - aldanor 3y ago
    The sole fact that pandas, while trying to catch up to polars, now sort-of supports arrow as it's in-memory backend doesn't make it a "pandas dataframe binary format" in any possible way.
    - ayhanfuat 3y ago
      The whole point of Arrow is that it is language agnostic; so of course it is not a "pandas binary format". However, pandas didn't start supporting Arrow "to try to catch up to polars". The creator of pandas, Wes McKinney, came up with the Arrow idea because of the interoperability limitations he came across while working with pandas and the big data ecosystem. Supporting Arrow has been in the works for a long time: https://wesmckinney.com/blog/pandas-and-apache-arrow/ https://wesmckinney.com/blog/pandas-and-apache-arrow/ Pandas is an older and larger library with less maneuver capability and millions of users relying on things not being broken.
constantcrying 3y ago
Python is definitely more popular and its formats are naturally more widely spread. Python is also a very bad language for many of the things it is used for (anything to do with numerics, e.g. machine learning) and having a competing format from a language which is, at least in that regard, far superior seems like a good thing.
- jstx1 3y ago
  It's not competing if it's limited to Julia. And saving a dataframe isn't on anyone's list of technical concerns anyway. It's a big stretch to say that a tiny Julia library which does a thing we can already do well in Julia and Python and just about any other language has any impact on the language ecosystems and how they compare to each other.
  - constantcrying 3y ago
    >It's not competing if it's limited to Julia. Sure, but it should.
- keithalewis 3y ago
  It is impossible to have an intelligent discussion with someone who only knows Python, FSVO know. Dunning-Kruger in action.
  - tomrod 3y ago
    I have extensive work in Matlab, C, Fortran, R, Python, Julia, SAS (IML), and am learning Go and Rust. I have played with Java and Kotlin on non-serious projects. I typically use Python for data-related tasks. Julia is just okay. It lost its steam due to long beta, slow time to first plot after 1.0 release, and generally not being a strategically adopted language by ML innovation groups. A great example of what is versus what was hoped is John Myles White's work[0], where the headline is "worked on large community Python projects" but is one of the originators of Julia's fundamental data packages. I've had lots of intelligent people who use code as a tool instead of as a craft -- for data folks, their craft is working with data and not the tooling around it. For folks like this, Python is simply a better option in today's environment. [0] https://www.johnmyleswhite.com/about/ https://www.johnmyleswhite.com/about/
juujian 3y ago
Apache Arrow sits in a similar niche, but it has support for multiple languages.
crabbone 3y ago
> Python data frame format That is not a thing. The closest would be the NumPy binary format, which is "nothing to write home about". It's not good in any particular way. Some sort of a stop-gap solution for when you really want to save the contents of a NumPy array very much, and you don't want to deal with writing some commonly supported format that's (hopefully) optimized for some particular task.

Isn’t the Python data frame format becoming an established format? I mean, there’s already commercial products that talk it natively.

by moomin 3y ago

Isn’t the Python data frame format becoming an established format? I mean, there’s already commercial products that talk it natively.

jzwinck 3y ago

There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files no other library understands in an intuitive way. They mostly recommend "to_csv" with all the usual problems of CSV. NumPy does have an official documented format which is straightforward to read and write from other languages if you write some code yourself, but still doesn't have very wide support outside of Python.

dijksterhuis 3y ago
Parquet is a binary file format and is very widely supported. I think there’s even an Excel plug-in for parquet these days.
- radarsat1 3y ago
  > I think there’s even an Excel plug-in for parquet these days. While CSV comes with plenty of known problems, its one advantage, which apparently remains a significant one, is that it doesn't require any plugins or special software support at all, just a text editor. It seems that's a big enough feature that no attempt to fully replace it has really worked out. Amazing when you think about it, the power of plain text encoding. Of course I agree that it's often not the best choice but somehow it remains incredibly useful and can even be hard to make arguments against it on a team project because it just kind or works well enough in a lot of cases and is simply a lowest common denominator when people can't agree on what format to use.
benglish11 3y ago
Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.
- aldanor 3y ago
  The sole fact that pandas, while trying to catch up to polars, now sort-of supports arrow as it's in-memory backend doesn't make it a "pandas dataframe binary format" in any possible way.
  - ayhanfuat 3y ago
    The whole point of Arrow is that it is language agnostic; so of course it is not a "pandas binary format". However, pandas didn't start supporting Arrow "to try to catch up to polars". The creator of pandas, Wes McKinney, came up with the Arrow idea because of the interoperability limitations he came across while working with pandas and the big data ecosystem. Supporting Arrow has been in the works for a long time: https://wesmckinney.com/blog/pandas-and-apache-arrow/ https://wesmckinney.com/blog/pandas-and-apache-arrow/ Pandas is an older and larger library with less maneuver capability and millions of users relying on things not being broken.

constantcrying 3y ago

Python is definitely more popular and its formats are naturally more widely spread. Python is also a very bad language for many of the things it is used for (anything to do with numerics, e.g. machine learning) and having a competing format from a language which is, at least in that regard, far superior seems like a good thing.

jstx1 3y ago
It's not competing if it's limited to Julia. And saving a dataframe isn't on anyone's list of technical concerns anyway. It's a big stretch to say that a tiny Julia library which does a thing we can already do well in Julia and Python and just about any other language has any impact on the language ecosystems and how they compare to each other.
- constantcrying 3y ago
  >It's not competing if it's limited to Julia. Sure, but it should.
keithalewis 3y ago
It is impossible to have an intelligent discussion with someone who only knows Python, FSVO know. Dunning-Kruger in action.
- tomrod 3y ago
  I have extensive work in Matlab, C, Fortran, R, Python, Julia, SAS (IML), and am learning Go and Rust. I have played with Java and Kotlin on non-serious projects. I typically use Python for data-related tasks. Julia is just okay. It lost its steam due to long beta, slow time to first plot after 1.0 release, and generally not being a strategically adopted language by ML innovation groups. A great example of what is versus what was hoped is John Myles White's work[0], where the headline is "worked on large community Python projects" but is one of the originators of Julia's fundamental data packages. I've had lots of intelligent people who use code as a tool instead of as a craft -- for data folks, their craft is working with data and not the tooling around it. For folks like this, Python is simply a better option in today's environment. [0] https://www.johnmyleswhite.com/about/ https://www.johnmyleswhite.com/about/

juujian 3y ago

Apache Arrow sits in a similar niche, but it has support for multiple languages.

crabbone 3y ago

> Python data frame format That is not a thing. The closest would be the NumPy binary format, which is "nothing to write home about". It's not good in any particular way. Some sort of a stop-gap solution for when you really want to save the contents of a NumPy array very much, and you don't want to deal with writing some commonly supported format that's (hopefully) optimized for some particular task.