3 ms·

How to write Python extensions in Rust with PyO3

45 points by jeremychone 4y ago

russnes 4y ago
PyO3 works great. I've written an example repository for a hello world app for this if anyone's interested: https://github.com/russnes/rust_python_package_example https://github.com/russnes/rust_python_package_example
- masklinn 4y ago
  For a somewhat more "productive" hello world (as in a very simple application which does useful things), there's nh3 (https://github.com/messense/nh3 https://github.com/messense/nh3) which is a relatively small wrapper around ammonia (https://github.com/rust-ammonia/ammonia https://github.com/rust-ammonia/ammonia), an HTML sanitisation library. The lib.rs is all of 125 SLOC (and about half of that is owing to the handling of `attribute_filter` which is pretty demanding on the glue layer: https://github.com/messense/nh3/commit/72be3e6728b7ceb9185e7431bfea39add15b5bdb https://github.com/messense/nh3/commit/72be3e6728b7ceb9185e7...).
Galanwe 4y ago
I found PyO3 to be a bit weird to use. There's plenty of documentation, but it's kind of vague on the general basic concepts, which makes the library hard to understand (even though I have a background in writing CPython extensions in C). Also, once you play with it more than 30m, you quickly realize that struct attributes are copied all over the place... which is not only slow, but more sadly any mutable function on an attribute will just silently modify a temporary copy...
rob74 4y ago
I assume PyO3 is a reference to the chemical formula of iron oxide (a.k.a. rust), which is Fe2O3?
- Mxrtxn 4y ago
  I think it's a reference to the -o3 optimalization level in gcc.
MrPowers 4y ago
PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs https://github.com/delta-io/delta-rs It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust. This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.
- jeremychone 4y ago
  Pola.rs is another good example of how a great Rust library can have an excellent Python façade. Polars ingenuously uses Python's operator overloading scheme to maximize data frames operation efficiencies. I think Python & Rust can become great pairs, especially in the data processing field.
stefanka 4y ago
Together with rust-numpy, it is a great way of writing numeric code (using the ndarray library) https://github.com/PyO3/rust-numpy https://github.com/PyO3/rust-numpy
UncleEntity 4y ago
Isn’t python “memory safe” too? From my experience the trickiest bit of getting python extensions to work (i.e. not segfault) is dealing with the lifetimes of the objects in the wrapped library. If python owns the object you’re golden, if it borrows a reference then things get real complicated real fast. It usually takes a lot longer to get the ownership model correct than doing the grunt work of wrapping the library — unless you own the lib and make it python friendly it’s usually a case of “do what you can with what you were given”. Blender’s python API is full of cases where the python object can’t be trusted to be valid if you do anything with the underlying container and it is super easy to crash the whole program doing something that should Just Work™. Wondering how rust would deal with the cross-domain model where nothing can be guaranteed once python gets its dirty little paws on the pointer?

45 points by jeremychone 4y ago

PyO3 works great. I've written an example repository for a hello world app for this if anyone's interested: https://github.com/russnes/rust_python_package_example https://github.com/russnes/rust_python_package_example

masklinn 4y ago
For a somewhat more "productive" hello world (as in a very simple application which does useful things), there's nh3 (https://github.com/messense/nh3 https://github.com/messense/nh3) which is a relatively small wrapper around ammonia (https://github.com/rust-ammonia/ammonia https://github.com/rust-ammonia/ammonia), an HTML sanitisation library. The lib.rs is all of 125 SLOC (and about half of that is owing to the handling of `attribute_filter` which is pretty demanding on the glue layer: https://github.com/messense/nh3/commit/72be3e6728b7ceb9185e7431bfea39add15b5bdb https://github.com/messense/nh3/commit/72be3e6728b7ceb9185e7...).

Galanwe 4y ago

I found PyO3 to be a bit weird to use. There's plenty of documentation, but it's kind of vague on the general basic concepts, which makes the library hard to understand (even though I have a background in writing CPython extensions in C). Also, once you play with it more than 30m, you quickly realize that struct attributes are copied all over the place... which is not only slow, but more sadly any mutable function on an attribute will just silently modify a temporary copy...

rob74 4y ago

I assume PyO3 is a reference to the chemical formula of iron oxide (a.k.a. rust), which is Fe2O3?

Mxrtxn 4y ago
I think it's a reference to the -o3 optimalization level in gcc.

MrPowers 4y ago

PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs https://github.com/delta-io/delta-rs It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust. This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.

jeremychone 4y ago
Pola.rs is another good example of how a great Rust library can have an excellent Python façade. Polars ingenuously uses Python's operator overloading scheme to maximize data frames operation efficiencies. I think Python & Rust can become great pairs, especially in the data processing field.

stefanka 4y ago

Together with rust-numpy, it is a great way of writing numeric code (using the ndarray library) https://github.com/PyO3/rust-numpy https://github.com/PyO3/rust-numpy

UncleEntity 4y ago

Isn’t python “memory safe” too? From my experience the trickiest bit of getting python extensions to work (i.e. not segfault) is dealing with the lifetimes of the objects in the wrapped library. If python owns the object you’re golden, if it borrows a reference then things get real complicated real fast. It usually takes a lot longer to get the ownership model correct than doing the grunt work of wrapping the library — unless you own the lib and make it python friendly it’s usually a case of “do what you can with what you were given”. Blender’s python API is full of cases where the python object can’t be trusted to be valid if you do anything with the underlying container and it is super easy to crash the whole program doing something that should Just Work™. Wondering how rust would deal with the cross-domain model where nothing can be guaranteed once python gets its dirty little paws on the pointer?