3 ms·

I routinely use cython to compile python for heavy workloads. A big part of the ~10x speedup i usually see comes from strategically assigning true static types

I routinely use cython to compile python for heavy workloads. A big part of the ~10x speedup i usually see comes from strategically assigning true static types to certain variables that get frequently iterated or compared. Most variables are left as standard python as it isn't necessary to change them for performance. My experience has been that cython "just works" even with lots of external libraries etc, and the code ends up performing as well as any c++ code i could realistically write while being much more understandable.

nextaccountic 4y ago
> I routinely use cython to compile python for heavy workloads. an alternative is rust + pyo3 https://pyo3.rs https://pyo3.rs here's a web framework written with it, https://robyn.tech https://robyn.tech the other poster child for pyo3 is polars, https://www.pola.rs https://www.pola.rs it's simply, amazing.
- samstave 4y ago
  which of these links should I start with with zero xp?
  - nextaccountic 4y ago
    If you don't know Rust, but know Python, you can install Python libraries written in Rust with pip. Like, pip install polars or pip install robyn. In this case you follow the two bottom links. But then you don't write your own libraries and stuff so.. I guess that's not what you want. If creating your own Python libraries in Rust is what you want, you would check out the first link I sent only, the one for pyo3. But, if you want to learn Rust, you probably wouldn't start out with pyo3. You first install Rust with https://rustup.rs/ https://rustup.rs/ and then check out the official book, and the book rust by example, that you can find here https://www.rust-lang.org/learn https://www.rust-lang.org/learn - and maybe write some code on the Rust playground https://play.rust-lang.org/ https://play.rust-lang.org/ - then, you use pyo3 to build Python libraries in Rust, and then use maturin https://www.maturin.rs/ https://www.maturin.rs/ to build and publish them to Pypi. But if you still prefer to begin with Rust by writing Python libraries (it's a valid strategy if you are very comfortable with working with multiple stacks), the Maturin link has a tutorial that setups a program that is half written in python, half written in Rust, https://www.maturin.rs/tutorial.html https://www.maturin.rs/tutorial.html (well the pyo3 link I sent also has one too. You should refer to the documentation of both, because you will use the two together) After learning Rust and building some stuff with pyo3, the next step is looking for libraries that you could leverage to make Python programs ultra fast. Here https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon is an obvious choice, see some examples from the Rust cookbook https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html https://rust-lang-nursery.github.io/rust-cookbook/concurrenc... - when you create a parallel iterator, it will distribute the processing to many threads (by default, one per core). The rust cookbook, by the way, is a nice reference to see some of the most used crates (Rust libraries) in the Rust ecosystem. If you are doing async stuff in Python, you probably need to look up https://github.com/awestlake87/pyo3-asyncio https://github.com/awestlake87/pyo3-asyncio and https://tokio.rs/ https://tokio.rs/ - that's two libraries that Robyn uses (see Robyn dependencies here https://github.com/sansyrox/robyn/blob/main/Cargo.toml https://github.com/sansyrox/robyn/blob/main/Cargo.toml) Anyway there are some posts about pyo3 on the web, like this blog post https://boring-guy.sh/posts/river-rust/ https://boring-guy.sh/posts/river-rust/ (note: it uses an outdated version of pyo3, and doesn't seem to use maturin which is a newer tool). This post was written by the developers of https://github.com/online-ml/river https://github.com/online-ml/river - another Python library written in Rust
- dereg 4y ago
  I'm so glad I found Polars. I replaced the most time-consuming parts of my Pandas code with Polars and it has reduced data manipulation times by literally 90-95%. Amazing is not an overstatement.

nextaccountic 4y ago

> I routinely use cython to compile python for heavy workloads. an alternative is rust + pyo3 https://pyo3.rs https://pyo3.rs here's a web framework written with it, https://robyn.tech https://robyn.tech the other poster child for pyo3 is polars, https://www.pola.rs https://www.pola.rs it's simply, amazing.

samstave 4y ago
which of these links should I start with with zero xp?
- nextaccountic 4y ago
  If you don't know Rust, but know Python, you can install Python libraries written in Rust with pip. Like, pip install polars or pip install robyn. In this case you follow the two bottom links. But then you don't write your own libraries and stuff so.. I guess that's not what you want. If creating your own Python libraries in Rust is what you want, you would check out the first link I sent only, the one for pyo3. But, if you want to learn Rust, you probably wouldn't start out with pyo3. You first install Rust with https://rustup.rs/ https://rustup.rs/ and then check out the official book, and the book rust by example, that you can find here https://www.rust-lang.org/learn https://www.rust-lang.org/learn - and maybe write some code on the Rust playground https://play.rust-lang.org/ https://play.rust-lang.org/ - then, you use pyo3 to build Python libraries in Rust, and then use maturin https://www.maturin.rs/ https://www.maturin.rs/ to build and publish them to Pypi. But if you still prefer to begin with Rust by writing Python libraries (it's a valid strategy if you are very comfortable with working with multiple stacks), the Maturin link has a tutorial that setups a program that is half written in python, half written in Rust, https://www.maturin.rs/tutorial.html https://www.maturin.rs/tutorial.html (well the pyo3 link I sent also has one too. You should refer to the documentation of both, because you will use the two together) After learning Rust and building some stuff with pyo3, the next step is looking for libraries that you could leverage to make Python programs ultra fast. Here https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon is an obvious choice, see some examples from the Rust cookbook https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html https://rust-lang-nursery.github.io/rust-cookbook/concurrenc... - when you create a parallel iterator, it will distribute the processing to many threads (by default, one per core). The rust cookbook, by the way, is a nice reference to see some of the most used crates (Rust libraries) in the Rust ecosystem. If you are doing async stuff in Python, you probably need to look up https://github.com/awestlake87/pyo3-asyncio https://github.com/awestlake87/pyo3-asyncio and https://tokio.rs/ https://tokio.rs/ - that's two libraries that Robyn uses (see Robyn dependencies here https://github.com/sansyrox/robyn/blob/main/Cargo.toml https://github.com/sansyrox/robyn/blob/main/Cargo.toml) Anyway there are some posts about pyo3 on the web, like this blog post https://boring-guy.sh/posts/river-rust/ https://boring-guy.sh/posts/river-rust/ (note: it uses an outdated version of pyo3, and doesn't seem to use maturin which is a newer tool). This post was written by the developers of https://github.com/online-ml/river https://github.com/online-ml/river - another Python library written in Rust
dereg 4y ago
I'm so glad I found Polars. I replaced the most time-consuming parts of my Pandas code with Polars and it has reduced data manipulation times by literally 90-95%. Amazing is not an overstatement.