5 ms·

There is also Polars [0], which is backed by arrow and a great alternative to pandas. [0] https://www.pola.rs/ https://www.pola.rs/

ZeroCool2u 4y ago
Yes, Marc talks extensively about the new interop options between Pandas and Polars enabled by the pyarrow backend.
nerdponx 4y ago
Polars is less ergonomic than Pandas in my experience for basic data analysis, and has less functionality for time series work. But it's better than Pandas for working with larger datasets and it's now my preferred choice for batch data processing in Python eg in ETL jobs. They both have their own places in the world.
- pletnes 4y ago
  Yes, and with both supporting arrow, you can zero-copy share data from polars into pandas once you’ve completed your initial data load+filtering.
  - nerdponx 4y ago
    Does zero-copy work currently, or is that still theoretical/todo? I would also be curious about Numpy, since I know you can transparently map data to a Numpy array, but that's just "raw" fixed-width binary data and not something more structured like Arrow.
    - ec109685 4y ago
      Once this is fixed: https://github.com/pola-rs/polars/pull/7084 https://github.com/pola-rs/polars/pull/7084
      - nerdponx 4y ago
        I meant to ask about the other way, Polars to Pandas, but this is also useful to know.
      - rihegher 4y ago
        It looks like it has already been merged, or did I miss something?
- 323 4y ago
  Polars is more verbose, but simpler. It's core operations "fit the brain". I've used pandas for years, but I still have to regularly google basic indexing stuff.
  - brahbrah 4y ago
    I don’t think it’s so black and white. Depends on what you’re doing. If you’re doing lots of cross dataset operations like econometric and physical system modeling polars can get a bit too verbose. Eg how would you do the following two pandas operations in polars? capacity_df - outage_df prices.loc['2023-01'] *= 1 There’s many workflows and models that do thousands of these types of operations.
    - 323 4y ago
      prices.loc['2023-01'] *= 1 You can always do df.to_pandas() ... prices.loc['2023-01'] *= 1 ... from_pandas() :) More seriously, you are right, this is a tough one to do in polars. Polars seems to want to work with whole columns at a time, it doesn't give you write access to row sets.
eternalban 4y ago
Do you have an opinion on N5? My second hand (somewhat dated) info is that Pandas are not a good fit for unstructured (CV) data or ML.
berkle4455 4y ago
There's also SQL, when paired with optimized column-store databases that can run embedded or local (duckdb or clickhouse), you can avoid the extra hop into python libraries entirely.

ZeroCool2u 4y ago

Yes, Marc talks extensively about the new interop options between Pandas and Polars enabled by the pyarrow backend.

nerdponx 4y ago

Polars is less ergonomic than Pandas in my experience for basic data analysis, and has less functionality for time series work. But it's better than Pandas for working with larger datasets and it's now my preferred choice for batch data processing in Python eg in ETL jobs. They both have their own places in the world.

pletnes 4y ago
Yes, and with both supporting arrow, you can zero-copy share data from polars into pandas once you’ve completed your initial data load+filtering.
- nerdponx 4y ago
  Does zero-copy work currently, or is that still theoretical/todo? I would also be curious about Numpy, since I know you can transparently map data to a Numpy array, but that's just "raw" fixed-width binary data and not something more structured like Arrow.
  - ec109685 4y ago
    Once this is fixed: https://github.com/pola-rs/polars/pull/7084 https://github.com/pola-rs/polars/pull/7084
    - nerdponx 4y ago
      I meant to ask about the other way, Polars to Pandas, but this is also useful to know.
    - rihegher 4y ago
      It looks like it has already been merged, or did I miss something?
323 4y ago
Polars is more verbose, but simpler. It's core operations "fit the brain". I've used pandas for years, but I still have to regularly google basic indexing stuff.
- brahbrah 4y ago
  I don’t think it’s so black and white. Depends on what you’re doing. If you’re doing lots of cross dataset operations like econometric and physical system modeling polars can get a bit too verbose. Eg how would you do the following two pandas operations in polars? capacity_df - outage_df prices.loc['2023-01'] *= 1 There’s many workflows and models that do thousands of these types of operations.
  - 323 4y ago
    prices.loc['2023-01'] *= 1 You can always do df.to_pandas() ... prices.loc['2023-01'] *= 1 ... from_pandas() :) More seriously, you are right, this is a tough one to do in polars. Polars seems to want to work with whole columns at a time, it doesn't give you write access to row sets.

eternalban 4y ago

Do you have an opinion on N5? My second hand (somewhat dated) info is that Pandas are not a good fit for unstructured (CV) data or ML.

berkle4455 4y ago

There's also SQL, when paired with optimized column-store databases that can run embedded or local (duckdb or clickhouse), you can avoid the extra hop into python libraries entirely.