3 ms·

I'm interested in possibly contributing / optimizing in polars, I would be curious to hear from a user first-hand which things are painful and/or slow still.

by orlp 4y ago

bobbylarrybobby 4y ago
Painful (polars-lazy): - Very scant documentation. Many things missing docs, many documented items missing examples. - Using a custom function in a map. You have to do this `Ok(series.type()?.into_iter().map(...).collect::<ChunkedArray<_>>()).into_series()` dance every time, and it's annoying. And you have to specify the output type even though it should be inferable from the function body. - There is no good way to "get-or-null" a column, i.e., treat an absent column as all null. You have to try to select the column, check if that errored, and if so insert a column of nulls (casted to the correct type) before proceeding. - Perhaps because it uses Arrow under the hood, there is no such thing as a non-nullable column. You just always have to handle nulls. - A bit verbose due to the lack of Into<Expr> implementations for most basic types. E.G., `lit(false)` instead of `false` because `bool` is not `Into<Expr>`. - It's a bit confusing at times which features you need for which abilities, and things will panic at runtime if you don't have the necessary features enabled (e.g., the strptime function has a tz_aware:bool parameter that panics if it's true but you don't have the time zone feature enabled). Ultimately it's a young library and so I guess it comes with the territory. It's still fantastic and having used it I'd be hard pressed to go back to pandas even despite all the flaws.
grepLeigh 4y ago
To echo the other comment, I've found digging around the test suites to be more reliable and comprehensive than the official examples/documentation. If you're running into issues with the Lazy API, read through these tests and see if that clears anything up: https://github.com/pola-rs/polars/tree/master/polars/polars-lazy/src/tests https://github.com/pola-rs/polars/tree/master/polars/polars-... My usage is very narrow (aggregating a dataframe of bounding box observations over a sliding time window) and I only use streaming Arrow IPC, which both work great!

I'm interested in possibly contributing / optimizing in polars, I would be curious to hear from a user first-hand which things are painful and/or slow still.

by orlp 4y ago

I'm interested in possibly contributing / optimizing in polars, I would be curious to hear from a user first-hand which things are painful and/or slow still.

bobbylarrybobby 4y ago

Painful (polars-lazy): - Very scant documentation. Many things missing docs, many documented items missing examples. - Using a custom function in a map. You have to do this `Ok(series.type()?.into_iter().map(...).collect::<ChunkedArray<_>>()).into_series()` dance every time, and it's annoying. And you have to specify the output type even though it should be inferable from the function body. - There is no good way to "get-or-null" a column, i.e., treat an absent column as all null. You have to try to select the column, check if that errored, and if so insert a column of nulls (casted to the correct type) before proceeding. - Perhaps because it uses Arrow under the hood, there is no such thing as a non-nullable column. You just always have to handle nulls. - A bit verbose due to the lack of Into<Expr> implementations for most basic types. E.G., `lit(false)` instead of `false` because `bool` is not `Into<Expr>`. - It's a bit confusing at times which features you need for which abilities, and things will panic at runtime if you don't have the necessary features enabled (e.g., the strptime function has a tz_aware:bool parameter that panics if it's true but you don't have the time zone feature enabled). Ultimately it's a young library and so I guess it comes with the territory. It's still fantastic and having used it I'd be hard pressed to go back to pandas even despite all the flaws.

grepLeigh 4y ago

To echo the other comment, I've found digging around the test suites to be more reliable and comprehensive than the official examples/documentation. If you're running into issues with the Lazy API, read through these tests and see if that clears anything up: https://github.com/pola-rs/polars/tree/master/polars/polars-lazy/src/tests https://github.com/pola-rs/polars/tree/master/polars/polars-... My usage is very narrow (aggregating a dataframe of bounding box observations over a sliding time window) and I only use streaming Arrow IPC, which both work great!