3 ms·

Libraries Run Rust Inside Python (With PyO3)

71 points by lumpa 19d ago

the__alchemist 19d ago
This is a nice party trick! I use it to provide easy installation of software that is written in rust, but is primarily used by Python users. (e.g.: Biology tools) They can do `pip install <name>`, since some people prefer this over downloading executables/installers. Bonus: Maturin/PyO3 can build "manylinux" binaries automatically, which helps with the Linux ABI diaspora.
- benji-york 19d ago
  `pip install zig` works today.
simonw 19d ago
I was a bit nervous about this trend when it started picking up because I care about Pyodide (Python running via WebAssembly) and libraries that use PyO3 might not work in Pyodide. Thankfully that's now been mostly solved - you can compile and publish WASM builds of Rust or C extensions on PyPI now and a Pyodide can then use them. Here's the WASM build of the Rust-including Pydantic-core package for example: https://pypi.org/project/pydantic_core/#pydantic_core-2.49.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl https://pypi.org/project/pydantic_core/#pydantic_core-2.49.0...
- saghm 19d ago
  I'm kind of surprised to hear that was a concern because my perception is that PyO3 is more analogous to wrapping native C code in Python libraries (which is pretty well established since long before either Rust or WASM), and Pyodide seems more like something for use in browsers rather on the server-side. When I looked into this earlier this year, trying to run Python via WASM on the server side with Pyodide had a bunch of hoops I needed to jump through, and it ended up being a lot simpler to use a build of Python from one of the core Python contributors that directly targeted WASM.
  - simonw 19d ago
    Yes, Pyodide is almost entirely used in web browsers. I want to be able to run existing code that uses libraries like Pyodide-core in a browser.
- hoodchatham 19d ago
  It wasn't all that easy to make it work: https://blog.pyodide.org/posts/rust-pyo3-support-in-pyodide/ https://blog.pyodide.org/posts/rust-pyo3-support-in-pyodide/
roywiggins 19d ago
> reach for, a Rust extension does the work ai; dr, sorry
- dathinab 19d ago
  it probably is AI written but it's a bit strange It's too "clunky" for AI, like you stumble too much over the words when reading If we take the first sentence that could have been from a German person, where such language structures aren't that uncommon. I myself write a lot like that, and always have written like that, often in way worse ways, like sentences which never, not even after docent of words, end, and have levels of sub-sentence nesting no normal thinking human would want to read. (<- yes that was intentional. I tend to write more concise today.) Through there are indicators of AI usage in pretty much every single sentence. But at the same time the writing is too bad, for most current AI agents. Like reading it is hard, in ways AI tends to not mess up. There are missing filler and glue words. Including some cases where I think they are probably grammatically require (through school was too long ago, so idk.). Sometimes it looks like a different language construct was used (like bracket or -- hyphen inlays) which then where search+replaced with commas. So I really wonder what that is? - a lower quality LLM Model? - AI written text modified by hand leading to it having the worst of both worlds? - or a human who has self learned English as a second language using AI and believes this is how normal English speakers write ... and the last option is kinda frightening, because sooner or later we will have exactly that: People who write like AI "but worse" and likely do not even realize what they are doing. --- EDIT/PS: Side note; Rust in Python is a decent match. PyO3 is far from perfect but decently usable. Spreading more awareness about it is nice. This comment is only about the writing structure not the content of the blog post.
skeledrew 19d ago
But does it work everywhere Python works? My main issue with this Rust move has always been compatibility. Python can be embedded and ran in a heck of a lot of places. What's the story when libraries that I may want to depend on are actually implemented in Rust and my target doesn't/can't handle the toolchain and there's no build target?
- the__alchemist 19d ago
  It won't work in those cases. Do you have explicit examples? I suspect this won't come up much as Rust has, IME, much broader target support. (Including platforms without a GPOS)
  - galangalalgol 19d ago
    I can't come up with any cases where cpython would compile that rustc can't target. There are some more niche interpreters that can run on embedded stuff with no rustc support, but they are also subsets of python so it isn't like you can just pull a wheel file and expect it to work
    - vlovich123 19d ago
      For what it’s worth the GCC backend for rustc is making progress and rustc more generally supports a lot of embedded stuff already
      - kstrauser 19d ago
        How’s that GCC backend coming along? I could look it up, but I bet others would like to hear about it too.
        vlovich123 18d ago
        https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-43 https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-43 They’re plugging away at it. A major milestone was getting unwinding to work. I’d say they’re getting quite close and it’s getting better and better. It’s usable for Linux kernel compilation I believe which unlocks Rust to be used for the non-driver components in the kernel.
hk1337 19d ago
Why not just run C inside of Python?
- saghm 19d ago
  For the same reasons someone might prefer Rust to C in general; safety, ergonomics, etc. If you want to write C in Python, nothing is stopping you.
- the__alchemist 19d ago
  You can do that too. I generally prefer rust as it's a nice lang to work with.
  - m00dy 19d ago
    yeah, Rust is eating C/C++...
    - pjmlp 18d ago
      While using a compiler framework written in C++, yeah.
      - zbentley 18d ago
        I'm with you on the broader sentiment that Rust isn't making nearly as many inroads many people think. But this example is silly. Do we say "FORTRAN is eating R" because R uses BLAS/LAPACK/whatever via gfortran? Nah. A zillion languages use LLVM.
        wolvesechoes 18d ago
        But this is not a good analogy. Rust is being talked about explicitly as "better C++", with a huge and vocal group of online people going around and saying that the world will end if we don't rewrite everything in Rust, or that no new C++ project should be started today with Rust being available. Very few people, if any, put R against Fortran in this way, so the fact that Fortran libraries weren't rewritten in R doesn't contradict anyone's expectations.
throwaway63467 19d ago
They’re not running Rust they’re running machine code compiled with the Rust compiler. Like many other Python libraries such as numpy are running machine code compiled with C++, C or Fortran. That was always the case for Python and is its main selling point, it’s a slow but easy to write glue language that can make faster native code scriptable. Also the article reeks of AI slop, it’s just trying to sell you a Rust course.
- vlovich123 19d ago
  One main difference is that pyo3 is safer and easier to use to build said modules and have them work correctly without issues. Thus modules which hadn’t been worth it before to build in this manner suddenly become so
  - throwaway63467 19d ago
    There’s a whole legacy of wrapper interface generators like SWIG that are way more powerful than PyO3, QT is e.g. fully mapped to Python using a very powerful wrapper generator (SIP). PyO3 is just a really simple interface generator in comparison, look at what SWIG can do, or SIP, what PyO3 does is trivial in comparison.
WD-42 19d ago
It’s time to start figuring out how to block domains on hacker news. This slop is getting out of control. Who can read this? The first few sentences already induced a migraine.
- hn993302 18d ago
  Wait til you see the first diagram
- pjmlp 18d ago
  We can't cheer about using Claude for coding, and then complain they use to write blog posts as well. This attitude is what gets me a migraine.
  - roywiggins 18d ago
    For me, it's simply that I have to read Claudese instrumentally for work enough that I am not going to spend my neurons reading it idly. I am not mad about it or anything, it just sucks to open up a blog post and realize that it's just Claude again under a different name.
startup_zombie_ 19d ago
How much of the existing PyPI ecosystem do you think could realistically work this way without package authors doing anything specifically for WASM?
- hoodchatham 19d ago
  In practice it is easier to support rust packages on web assembly than C or C++ packages since Rust cross compilation is in better shape. Of course packages that do IO can be a challenge but problematic IO is also more common in C libraries than Rust libraries. Tokio doesn't work yet but epoll-based support for it is in progress.
AlienRobot 19d ago
This is for WASM, but you can also do it to use a Qt frontend via Python's PySide6 with a Rust backend, avoiding the need of Rust bindings for Qt.
lr1970 18d ago
High performance polars dataFrame processing library [0] is written in Rust with Python bindings via PyO3/maturin. It exposes huge dataframes to python and successfully solved the interface boundayr problems described in the article. [0] https://github.com/pola-rs/polars https://github.com/pola-rs/polars

71 points by lumpa 19d ago

This is a nice party trick! I use it to provide easy installation of software that is written in rust, but is primarily used by Python users. (e.g.: Biology tools) They can do `pip install <name>`, since some people prefer this over downloading executables/installers. Bonus: Maturin/PyO3 can build "manylinux" binaries automatically, which helps with the Linux ABI diaspora.

benji-york 19d ago
`pip install zig` works today.

simonw 19d ago

I was a bit nervous about this trend when it started picking up because I care about Pyodide (Python running via WebAssembly) and libraries that use PyO3 might not work in Pyodide. Thankfully that's now been mostly solved - you can compile and publish WASM builds of Rust or C extensions on PyPI now and a Pyodide can then use them. Here's the WASM build of the Rust-including Pydantic-core package for example: https://pypi.org/project/pydantic_core/#pydantic_core-2.49.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl https://pypi.org/project/pydantic_core/#pydantic_core-2.49.0...

saghm 19d ago
I'm kind of surprised to hear that was a concern because my perception is that PyO3 is more analogous to wrapping native C code in Python libraries (which is pretty well established since long before either Rust or WASM), and Pyodide seems more like something for use in browsers rather on the server-side. When I looked into this earlier this year, trying to run Python via WASM on the server side with Pyodide had a bunch of hoops I needed to jump through, and it ended up being a lot simpler to use a build of Python from one of the core Python contributors that directly targeted WASM.
- simonw 19d ago
  Yes, Pyodide is almost entirely used in web browsers. I want to be able to run existing code that uses libraries like Pyodide-core in a browser.
hoodchatham 19d ago
It wasn't all that easy to make it work: https://blog.pyodide.org/posts/rust-pyo3-support-in-pyodide/ https://blog.pyodide.org/posts/rust-pyo3-support-in-pyodide/

roywiggins 19d ago

> reach for, a Rust extension does the work ai; dr, sorry

dathinab 19d ago
it probably is AI written but it's a bit strange It's too "clunky" for AI, like you stumble too much over the words when reading If we take the first sentence that could have been from a German person, where such language structures aren't that uncommon. I myself write a lot like that, and always have written like that, often in way worse ways, like sentences which never, not even after docent of words, end, and have levels of sub-sentence nesting no normal thinking human would want to read. (<- yes that was intentional. I tend to write more concise today.) Through there are indicators of AI usage in pretty much every single sentence. But at the same time the writing is too bad, for most current AI agents. Like reading it is hard, in ways AI tends to not mess up. There are missing filler and glue words. Including some cases where I think they are probably grammatically require (through school was too long ago, so idk.). Sometimes it looks like a different language construct was used (like bracket or -- hyphen inlays) which then where search+replaced with commas. So I really wonder what that is? - a lower quality LLM Model? - AI written text modified by hand leading to it having the worst of both worlds? - or a human who has self learned English as a second language using AI and believes this is how normal English speakers write ... and the last option is kinda frightening, because sooner or later we will have exactly that: People who write like AI "but worse" and likely do not even realize what they are doing. --- EDIT/PS: Side note; Rust in Python is a decent match. PyO3 is far from perfect but decently usable. Spreading more awareness about it is nice. This comment is only about the writing structure not the content of the blog post.

skeledrew 19d ago

But does it work everywhere Python works? My main issue with this Rust move has always been compatibility. Python can be embedded and ran in a heck of a lot of places. What's the story when libraries that I may want to depend on are actually implemented in Rust and my target doesn't/can't handle the toolchain and there's no build target?

the__alchemist 19d ago
It won't work in those cases. Do you have explicit examples? I suspect this won't come up much as Rust has, IME, much broader target support. (Including platforms without a GPOS)
- galangalalgol 19d ago
  I can't come up with any cases where cpython would compile that rustc can't target. There are some more niche interpreters that can run on embedded stuff with no rustc support, but they are also subsets of python so it isn't like you can just pull a wheel file and expect it to work
  - vlovich123 19d ago
    For what it’s worth the GCC backend for rustc is making progress and rustc more generally supports a lot of embedded stuff already
    - kstrauser 19d ago
      How’s that GCC backend coming along? I could look it up, but I bet others would like to hear about it too.
      - vlovich123 18d ago
        https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-43 https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-43 They’re plugging away at it. A major milestone was getting unwinding to work. I’d say they’re getting quite close and it’s getting better and better. It’s usable for Linux kernel compilation I believe which unlocks Rust to be used for the non-driver components in the kernel.

hk1337 19d ago

Why not just run C inside of Python?

saghm 19d ago
For the same reasons someone might prefer Rust to C in general; safety, ergonomics, etc. If you want to write C in Python, nothing is stopping you.
the__alchemist 19d ago
You can do that too. I generally prefer rust as it's a nice lang to work with.
- m00dy 19d ago
  yeah, Rust is eating C/C++...
  - pjmlp 18d ago
    While using a compiler framework written in C++, yeah.
    - zbentley 18d ago
      I'm with you on the broader sentiment that Rust isn't making nearly as many inroads many people think. But this example is silly. Do we say "FORTRAN is eating R" because R uses BLAS/LAPACK/whatever via gfortran? Nah. A zillion languages use LLVM.
      - wolvesechoes 18d ago
        But this is not a good analogy. Rust is being talked about explicitly as "better C++", with a huge and vocal group of online people going around and saying that the world will end if we don't rewrite everything in Rust, or that no new C++ project should be started today with Rust being available. Very few people, if any, put R against Fortran in this way, so the fact that Fortran libraries weren't rewritten in R doesn't contradict anyone's expectations.

throwaway63467 19d ago

They’re not running Rust they’re running machine code compiled with the Rust compiler. Like many other Python libraries such as numpy are running machine code compiled with C++, C or Fortran. That was always the case for Python and is its main selling point, it’s a slow but easy to write glue language that can make faster native code scriptable. Also the article reeks of AI slop, it’s just trying to sell you a Rust course.

vlovich123 19d ago
One main difference is that pyo3 is safer and easier to use to build said modules and have them work correctly without issues. Thus modules which hadn’t been worth it before to build in this manner suddenly become so
- throwaway63467 19d ago
  There’s a whole legacy of wrapper interface generators like SWIG that are way more powerful than PyO3, QT is e.g. fully mapped to Python using a very powerful wrapper generator (SIP). PyO3 is just a really simple interface generator in comparison, look at what SWIG can do, or SIP, what PyO3 does is trivial in comparison.

WD-42 19d ago

It’s time to start figuring out how to block domains on hacker news. This slop is getting out of control. Who can read this? The first few sentences already induced a migraine.

hn993302 18d ago
Wait til you see the first diagram
pjmlp 18d ago
We can't cheer about using Claude for coding, and then complain they use to write blog posts as well. This attitude is what gets me a migraine.
- roywiggins 18d ago
  For me, it's simply that I have to read Claudese instrumentally for work enough that I am not going to spend my neurons reading it idly. I am not mad about it or anything, it just sucks to open up a blog post and realize that it's just Claude again under a different name.

startup_zombie_ 19d ago

How much of the existing PyPI ecosystem do you think could realistically work this way without package authors doing anything specifically for WASM?

hoodchatham 19d ago
In practice it is easier to support rust packages on web assembly than C or C++ packages since Rust cross compilation is in better shape. Of course packages that do IO can be a challenge but problematic IO is also more common in C libraries than Rust libraries. Tokio doesn't work yet but epoll-based support for it is in progress.

AlienRobot 19d ago

This is for WASM, but you can also do it to use a Qt frontend via Python's PySide6 with a Rust backend, avoiding the need of Rust bindings for Qt.

lr1970 18d ago

High performance polars dataFrame processing library [0] is written in Rust with Python bindings via PyO3/maturin. It exposes huge dataframes to python and successfully solved the interface boundayr problems described in the article. [0] https://github.com/pola-rs/polars https://github.com/pola-rs/polars