3 ms·

Between Rust kernels for Jupyter notebooks, PolarRS, and Gstreamer bindings, Rust has become my tool of choice for computer vision pipelines. I used to prototy

Between Rust kernels for Jupyter notebooks, PolarRS, and Gstreamer bindings, Rust has become my tool of choice for computer vision pipelines. I used to prototype in Python, then re-implement the final algorithm in Rust for performance on an embedded device. Now I just start prototyping in Rust.

orlp 4y ago
I'm interested in possibly contributing / optimizing in polars, I would be curious to hear from a user first-hand which things are painful and/or slow still.
- bobbylarrybobby 4y ago
  Painful (polars-lazy): - Very scant documentation. Many things missing docs, many documented items missing examples. - Using a custom function in a map. You have to do this `Ok(series.type()?.into_iter().map(...).collect::<ChunkedArray<_>>()).into_series()` dance every time, and it's annoying. And you have to specify the output type even though it should be inferable from the function body. - There is no good way to "get-or-null" a column, i.e., treat an absent column as all null. You have to try to select the column, check if that errored, and if so insert a column of nulls (casted to the correct type) before proceeding. - Perhaps because it uses Arrow under the hood, there is no such thing as a non-nullable column. You just always have to handle nulls. - A bit verbose due to the lack of Into<Expr> implementations for most basic types. E.G., `lit(false)` instead of `false` because `bool` is not `Into<Expr>`. - It's a bit confusing at times which features you need for which abilities, and things will panic at runtime if you don't have the necessary features enabled (e.g., the strptime function has a tz_aware:bool parameter that panics if it's true but you don't have the time zone feature enabled). Ultimately it's a young library and so I guess it comes with the territory. It's still fantastic and having used it I'd be hard pressed to go back to pandas even despite all the flaws.
- grepLeigh 4y ago
  To echo the other comment, I've found digging around the test suites to be more reliable and comprehensive than the official examples/documentation. If you're running into issues with the Lazy API, read through these tests and see if that clears anything up: https://github.com/pola-rs/polars/tree/master/polars/polars-lazy/src/tests https://github.com/pola-rs/polars/tree/master/polars/polars-... My usage is very narrow (aggregating a dataframe of bounding box observations over a sliding time window) and I only use streaming Arrow IPC, which both work great!
serguzest 4y ago
unrelated to the topic, have you ever used google's mediapipe? Just would like to hear your opinion about it, you seem like deep with cv and ai. I am trying to make a video meme creator with object tracking as a hobby project during my hiatus.
deleted 4y ago
[deleted]
Dowwie 4y ago
Rust is a slow compile time language that isn't suited for notebook driven development
- aldanor 4y ago
  Depends on what you're doing, you could similarly say "Python is a slow runtime language that isn't suited". When you dataset sizes are measured in 10-100gbs, some Python/numpy quirks may become pretty painful and waiting for the cell to recompile may end up being preferable.

orlp 4y ago

I'm interested in possibly contributing / optimizing in polars, I would be curious to hear from a user first-hand which things are painful and/or slow still.

bobbylarrybobby 4y ago
Painful (polars-lazy): - Very scant documentation. Many things missing docs, many documented items missing examples. - Using a custom function in a map. You have to do this `Ok(series.type()?.into_iter().map(...).collect::<ChunkedArray<_>>()).into_series()` dance every time, and it's annoying. And you have to specify the output type even though it should be inferable from the function body. - There is no good way to "get-or-null" a column, i.e., treat an absent column as all null. You have to try to select the column, check if that errored, and if so insert a column of nulls (casted to the correct type) before proceeding. - Perhaps because it uses Arrow under the hood, there is no such thing as a non-nullable column. You just always have to handle nulls. - A bit verbose due to the lack of Into<Expr> implementations for most basic types. E.G., `lit(false)` instead of `false` because `bool` is not `Into<Expr>`. - It's a bit confusing at times which features you need for which abilities, and things will panic at runtime if you don't have the necessary features enabled (e.g., the strptime function has a tz_aware:bool parameter that panics if it's true but you don't have the time zone feature enabled). Ultimately it's a young library and so I guess it comes with the territory. It's still fantastic and having used it I'd be hard pressed to go back to pandas even despite all the flaws.
grepLeigh 4y ago
To echo the other comment, I've found digging around the test suites to be more reliable and comprehensive than the official examples/documentation. If you're running into issues with the Lazy API, read through these tests and see if that clears anything up: https://github.com/pola-rs/polars/tree/master/polars/polars-lazy/src/tests https://github.com/pola-rs/polars/tree/master/polars/polars-... My usage is very narrow (aggregating a dataframe of bounding box observations over a sliding time window) and I only use streaming Arrow IPC, which both work great!

serguzest 4y ago

unrelated to the topic, have you ever used google's mediapipe? Just would like to hear your opinion about it, you seem like deep with cv and ai. I am trying to make a video meme creator with object tracking as a hobby project during my hiatus.

deleted 4y ago

[deleted]

Dowwie 4y ago

Rust is a slow compile time language that isn't suited for notebook driven development

aldanor 4y ago
Depends on what you're doing, you could similarly say "Python is a slow runtime language that isn't suited". When you dataset sizes are measured in 10-100gbs, some Python/numpy quirks may become pretty painful and waiting for the cell to recompile may end up being preferable.