9 ms·
PyO3: Rust Bindings for the Python Interpreter
- LockAndLol 6y agoIf this works well, I'd rather use this over being forced to use type hints and mypy. Has anybody used this in conjunction with a python framework? Django, fastapi or something?
- uranusjr 6y agoUh, how do you plan to use FastAPI while avoiding type hints?
- pansa2 6y agoHow would PyO3 help you avoid type hints and mypy?
- brundolf 6y agoI think the idea is that they move their business logic to the Rust code, since Rust's type system is more powerful and more sound, instead of trying to make do with MyPy
- zerkten 6y agoWouldn't it be more of a priority to move it for lower memory use and higher request speed? A better type system is good, but often these are a struggle with scaling interpreted languages compared to other lower level languages.
- brundolf 6y agoFor many people the primary appeal of Rust is its type system and related features (declaring deep immutability, pattern-matching, etc) > often these are a struggle with scaling interpreted languages compared to other lower level languages Not sure what's meant by this
- LockAndLol 6y agoIt would minimize the python surface required to be covered with type-hints and mypy. If possible, one could simply point django to the modules generated from rust. I'll give it a shot tonight and see how it goes. Now I'm curious.
- edenhyacinth 6y agoI have! Used FastAPI as a frontend to do some minor data modification, and passed the data for model inference in Rust. Works really nicely, although given how little work I'm doing in the Python side I honestly prefer using Rocket instead of FastAPI and then using pyo3 to call the Python library in Rust, rather than the other way around.
- LockAndLol 6y agoThanks for the response. That does sound pretty much like what I would like to do. Have you by any chance open-sourced your project? I'm new to rust, but I'll check out Rocket. Cheers
- itamarst 6y agoI've been playing with PyO3 for prototyping, and wrapped some Rust code to see if it's faster than Python. The experience was very much like using Boost Python (whcih these days has alternative with https://github.com/pybind/pybind11 https://github.com/pybind/pybind11). It's _really_ easy to wrap code for Python, and it has nice APIs to ensure GIL is held. Being Rust, I'm much more confident I won't suffer from memory unsafety issues which my C++ at the time did. Now I'm starting to use it as part of the Python memory profiler I'm working on (https://pythonspeed.com/fil https://pythonspeed.com/fil), in this case to call in to the low-level Python C API which PyO3 includes bindings for in addition to its high-level API. This kind of usage is more like writing C, except with the benefit of having high-level APIs (for GIL holding, but also object conversion) available when I need it. So basically you get safe, high-level, easy-to-use APIs, with fallback to low-level unsafe APIs if you need them. Highly recommend trying it out.
- shirakawasuna 6y agoSounds great! Would so much rather drop into Rust than C or C++.
- brundolf 6y agoWhat's the data-conversion overhead look like at the boundary? Which data structures can be passed back and forth without a full clone, etc?
- itamarst 6y agoThere's definitely a conversion cost. For strings, Python apparently caches the UTF-8 encoded string, so if you _repeatedly_ transfer it to Rust I suspect (but haven't checked) that the cost is much lower. In general I suspect it's the usual "NumPy arrays are fast, everything else you better be getting a sufficiently large boost from the low-level code to justify conversion". For the thing I prototyped in Rust, it was wrapping the `ahocorasick` crate which was in fact faster than `pyahocorasick` which is written in C or Cython or something. Both have similar conversion costs, probably, so it came down to "for lots of data the Rust version was faster".
- edeion 6y agoThat's a really great name you came up with! Embodies both parts of your focus, stays pronounceable. Does the 3 relate to the Python version or are you mimicking some specific molecule that I can't think of?
- smlckz 6y agoPy (iv) O O = Py < | O or Py (vi) O || O = Py || O or Py (ii) O Py < > O O heh!
- auscompgeek 6y agoI think you might be missing an oxygen atom there.
- Swenrekcah 6y agoI would guess it is derived from: https://en.wikipedia.org/wiki/Iron(III)_oxide https://en.wikipedia.org/wiki/Iron(III)_oxide
- pansa2 6y agoRelated: RustPython - A Python interpreter written in Rust. https://github.com/RustPython/RustPython https://github.com/RustPython/RustPython
- bluedays 6y agoWithout looking at it I wonder if it's using the Python language underneath, or the python vm. Either way this is pretty cool.
- Nvorzula 6y agoPrecisely, this is Rust that compiles to a C FFI that plugs into CPython.
- benecollyridam 6y agoAnother related project: Wasmtime and Rust+Python Compile your Rust code to wasm to circumvent having to compile for different architectures. https://docs.wasmtime.dev/wasm-rust.html https://docs.wasmtime.dev/wasm-rust.html
- dbrgn 6y agoIf you're interested in publishing Rust libraries as Python packages (or integrating Rust code into an existing Python package), check out https://github.com/PyO3/maturin https://github.com/PyO3/maturin and https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust.
- edenhyacinth 6y agoBeen using Maturin for a little while professionally, and it's surprisingly good. There's a few bugbears here and there - I haven't found a way to have Cargo Test & a pyo3 library working at the same time - but overall it's a lot more pleasant than working with Rust and R was.
- adkadskhj 6y agoI needed Blender integration a while back and wasn't sure what i could write it in. Py03 worked great with Blender with no configuration. I was quite concerned that something about the Python-embedded-Blender behavior would limit Py03.. but nope, so far it's worked flawlessly. Thanks Py03 team :)
- fulafel 6y agoPreviously (2017): https://news.ycombinator.com/item?id=14859844 https://news.ycombinator.com/item?id=14859844
- gukoff 6y agoWith PyO3, I built the library to parse datetimes 10x faster than `datetime.strptime` in just a few lines of code: https://github.com/gukoff/dtparse https://github.com/gukoff/dtparse It just calls the Rust's chrono library that does the parsing and wraps the result in a Python object. You can do it for any Rust library, it's very, very easy! The only slightly complicated part is the distribution. You need to use https://github.com/PyO3/maturin https://github.com/PyO3/maturin or https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, and of course, you need to have Rust installed on the wheel-building machine. Feel free to use this repo as a reference if you want to build a similar thing. The code is commented, and there's a working GitHub action that builds the wheels for all platforms and uploads them to PyPi: https://github.com/gukoff/dtparse/tree/master/.github/workflows https://github.com/gukoff/dtparse/tree/master/.github/workfl...
- throwaway894345 6y agoI'm very curious to hear the use case for which date time parsing was the bottleneck! Also, I'm surprised that the overhead of calling across the language boundary didn't dwarf the gains from parsing...
- oblvious-earth 6y agoI've had this situation a few times. Most recently transforming large (1-50 GB) CSV files in to a format that can be digested by a proprietary bulk DB loader. Because our problem was just about reformatting we ended up reading the CSVs in binary mode and using struct to extract the relevant values from the date time fields. But if we needed to do actual date logic something like this would perhaps be useful (but there other fast date time libraries out there, I've been a fan of pendulum for some tasks).
- throwaway894345 6y agoThat makes sense, but I have a hard time believing the approach of calling into a date time parser O(n) times is going to yield a significant performance gain no matter how much faster the parser is. However, I'm being downvoted, so perhaps I'm mistaken?
- mynameisash 6y agoAt work, I'm using PyO3 for a project that churns through a lot of data (step 1) and does some pattern mining (step 2). This is the second generation of the project and is on-demand compared with the large, batch project in Spark that it is replacing. The Rust+Python project has really good performance, and using Rust for the core logic is such a joy compared with Scala or Python that a lot of other pieces are written in. Learning PyO3, I cobbled together a sample project[0] to demonstrate how some functionality works. It's a little outdated (uses PyO3 0.11.0 compared with the current 0.13.1) and doesn't show everything, but I think it's reasonably clear. One thing I noticed is that passing very large data from Rust and into Python's memory space is a bit of a challenge. I haven't quite grokked who owns what when and how memory gets correctly dropped, but I think the issues I've had are with the amount of RAM used at any moment and not with any memory leaks. [0] https://github.com/aeshirey/CheeseShop https://github.com/aeshirey/CheeseShop
- ksm1717 6y agoBetween pyodide, pyo3, rust-cpython, and rustpython, I think Pyo3 is the best way to drop in rust in a python project for a speed up, if that is your goal. Some of the demos show using python from rust, but to me the biggest feature is without a doubt compiling rust code to native python modules. I'm using it to speed up image manipulation backed by numpy arrays. There’s a setuptools rust [0] extension package that can be used to hook the compilation of the rust into the wheel building or install from source. Maturin [1] seems to be regarded as the new and improved solution for this, but I found that it’s angled toward the using python from rust. There’s also the rust numpy [2] package by the same org which is fantastic in that it lets you pass a numpy matrix to a native method written in rust and convert it to the rust equivalent data structure, perform whatever transformation you want (in parallel using rayon [3]), and return the array. When building for release, I was seeing speed ups of 100x over numpy on the most matrix mathable function imaginable, and numpy is no joke. I think there is a lot of potential for these two ecosystems together. If there’s not a python package for something, there’s probably a rust crate. If anyone is interested the python package that I'm building with some rust backend, its called pyrogis [4] for making custom image manipulations through numpy arrays. [0] https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust [1] https://github.com/PyO3/maturin https://github.com/PyO3/maturin [2] https://github.com/PyO3/rust-numpy https://github.com/PyO3/rust-numpy [3] https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon [4] https://github.com/pierogis/pierogis https://github.com/pierogis/pierogis
- cycomanic 6y ago> Between pyodide, pyo3, rust-cpython, and rustpython, I think Pyo3 is the best way to drop in rust in a python project for a speed up, if that is your goal. Some of the demos show using python from rust, but to me the biggest feature is without a doubt compiling rust code to native python modules. I'm using it to speed up image manipulation backed by numpy arrays. > There’s a setuptools rust [0] extension package that can be used to hook the compilation of the rust into the wheel building or install from source. Maturin [1] seems to be regarded as the new and improved solution for this, but I found that it’s angled toward the using python from rust. > There’s also the rust numpy [2] package by the same org which is fantastic in that it lets you pass a numpy matrix to a native method written in rust and convert it to the rust equivalent data structure, perform whatever transformation you want (in parallel using rayon [3]), and return the array. When building for release, I was seeing speed ups of 100x over numpy on the most matrix mathable function imaginable, and numpy is no joke. What sort of algorithm was that? Generally getting 100x speedup on vectorized code is highly unusual even using handcoded c++. So I suspect it was quite loop heavy? In those cases I have also seen very significant speed ups. I have been using pythran [1] for speeding up my python code. It generally achieves extremely good performance. I have blogged about it here [2] and recently a member used pythran to speed up some nbody benchmarks [3] which was used in an article to argue for using compiled languages. That said I find pyO3 quite exciting and have been contemplating to try it with some of my projects. [1] https://github.com/serge-sans-paille/pythran https://github.com/serge-sans-paille/pythran [2] https://jochenschroeder.com/blog/articles/DSP_with_Python2/ https://jochenschroeder.com/blog/articles/DSP_with_Python2/ [3] https://github.com/paugier/nbabel https://github.com/paugier/nbabel
- minimaxir 6y agoHuggingface Tokenizers (https://github.com/huggingface/tokenizers https://github.com/huggingface/tokenizers), which are now used by default in their Transformers Python library, use pyO3 and became popular due to the pitch that it encoded text an order of magnitude faster with zero config changes. It lives up to that claim. (I had issues with return object typing when going between Python/Rust at first but those are more consistent now)
- adsharma 6y agoThere is another way to speed up python: Write code in python and transpile to another language (could be rust) and then import it back into python https://github.com/adsharma/py2many/tree/main/tests/expected https://github.com/adsharma/py2many/tree/main/tests/expected Figuring out a mapping between a subset of a compiled language and a subset of statically typed python should be possible. The hard part is mapping standard library. I suspect something like nim might have an advantage there.
- mleonhard 6y agoI'm interested in running Python inside wasmtime. I think PyO3 would be useful. We could build a small Rust wasm binary that exports an "execute_python_script" function. This would finally be a way to run Python in a strong sandbox with memory [0] and CPU [1] restrictions. (In 1999, I asked Guido for sandboxing support in Python, but he refused.) [0] https://github.com/bytecodealliance/wasmtime/issues/2273 https://github.com/bytecodealliance/wasmtime/issues/2273 [1] https://github.com/bytecodealliance/wasmtime/issues/2274 https://github.com/bytecodealliance/wasmtime/issues/2274