4 ms·

I'm about publish Effective Pandas 2 (waiting for pandas 2.2 release) and an getting reviewed for my next book, Effective Polars. Happy to assist and answer an

I'm about publish Effective Pandas 2 (waiting for pandas 2.2 release) and an getting reviewed for my next book, Effective Polars. Happy to assist and answer any questions I can.

dr_kiszonka 3y ago
Before I ask my questions, here is an idea for your pandas book, if you haven't covered it already. The support of basic operations, like `round`, depends on the underlying data type (regular float and numpy's float16, float32, float64). Some np floats get rounded while others simply get ignored (they will not get rounded). It took me many hours to figure it out and fix resulting bugs. Maybe others would appreciate some information about this and similar gotchas. Regarding polars, would you have time to answer these questions? 1) How are polars supported by popular data science packages, e.g., for plotting? 2) I know it is a bit silly: is there a way to get around typing `pl.col`, etc. all the time? 3) Besides `tidypolars`, are there any reasonable packages that add support for dplyr-style pipes or operation chaining?
- __mharrison__ 3y ago
  1 - Polars has support for plotting in the most recent release. Using Matplotlib w/ Polars data tends to work too. Otherwise, you can drop into Pandas to get support where it may be missing. 2 - If you just want the columns, there is no need to make an expression; just pass the string. There are shortcuts like pl.sum(col) instead of pl.col(col).sum(). I guess you could shortcut, c = pl.col, if you really hate typing it... 3 - Polars supports chaining and encourages it out of the box. Anything specific you are looking for?
- rodonn 3y ago
  For (2) I like to do: 1. `from polars import col` which at least shortens each use by a few characters 2. For columns that I use very frequently, define a "constant" for them e.g. ID=col("id"). This lets you do df.groupby(ID) instead of df.groupby(pl.col("ID")). Another advantage of defining these column "constants" is that it makes it much easier to refactor to rename all usages of the column (without needing to check whether each string "id" is being used as a column name vs. something else)
- rodonn 3y ago
  I haven't run into any friction for (1), since worst case you just call `.to_pandas()` at the end of your pipeline before you start plotting. For any plotting apis that rely on the direct column vectors, no conversion to pandas is required.

I'm about publish Effective Pandas 2 (waiting for pandas 2.2 release) and an getting reviewed for my next book, Effective Polars. Happy to assist and answer any questions I can.

dr_kiszonka 3y ago

Before I ask my questions, here is an idea for your pandas book, if you haven't covered it already. The support of basic operations, like `round`, depends on the underlying data type (regular float and numpy's float16, float32, float64). Some np floats get rounded while others simply get ignored (they will not get rounded). It took me many hours to figure it out and fix resulting bugs. Maybe others would appreciate some information about this and similar gotchas. Regarding polars, would you have time to answer these questions? 1) How are polars supported by popular data science packages, e.g., for plotting? 2) I know it is a bit silly: is there a way to get around typing `pl.col`, etc. all the time? 3) Besides `tidypolars`, are there any reasonable packages that add support for dplyr-style pipes or operation chaining?

__mharrison__ 3y ago
1 - Polars has support for plotting in the most recent release. Using Matplotlib w/ Polars data tends to work too. Otherwise, you can drop into Pandas to get support where it may be missing. 2 - If you just want the columns, there is no need to make an expression; just pass the string. There are shortcuts like pl.sum(col) instead of pl.col(col).sum(). I guess you could shortcut, c = pl.col, if you really hate typing it... 3 - Polars supports chaining and encourages it out of the box. Anything specific you are looking for?
rodonn 3y ago
For (2) I like to do: 1. `from polars import col` which at least shortens each use by a few characters 2. For columns that I use very frequently, define a "constant" for them e.g. ID=col("id"). This lets you do df.groupby(ID) instead of df.groupby(pl.col("ID")). Another advantage of defining these column "constants" is that it makes it much easier to refactor to rename all usages of the column (without needing to check whether each string "id" is being used as a column name vs. something else)
rodonn 3y ago
I haven't run into any friction for (1), since worst case you just call `.to_pandas()` at the end of your pipeline before you start plotting. For any plotting apis that rely on the direct column vectors, no conversion to pandas is required.