8 ms·

Extending Python with Rust

214 points by behnamoh 4y ago

shoobs 4y ago
I implemented a well known curve simplification algorithm in Rust, and was pleasantly surprised how easy the interaction with Python was. For packaging, setuptools_rust was great, and I too used PyO3 for the bindings. I haven't tried Rumpy yet, but it looks interesting.
- masklinn 4y ago
  > I haven't tried Rumpy yet, but it looks interesting. Do you mean maturin? Rumpy is TFA's demo project.
  - shoobs 4y ago
    Oops, yep, my mistake.
college_physics 4y ago
> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python should really look whether numpy helps before complicating their codebase more. But there are more benefits to those bindings besides performance so its really nice to see the expanding options
- dj_mc_merlin 4y ago
  The diagram shows Numpy (orange line) below "Rumpy" (blue line). Since the y axis is time, less is more, so Numpy is faster indeed.
- dagmx 4y ago
  Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which means you need to be somewhat careful if you ever relocate an install of it to a new machine
  - orlp 4y ago
    And note that for example on Apple M1 it's essentially impossible to beat an implementation that uses Apple's Accelerate library for things like matrix multiplication, because Apple uses undocumented instructions unavailable to the public in that library.
    - orangepurple 4y ago
      Deprecated since version 1.20: The native libraries on macOS, provided by Accelerate, are not fit for use in NumPy since they have bugs that cause wrong output under easily reproducible conditions. If the vendor fixes those bugs, the library could be reinstated, but until then users compiling for themselves should use another linear algebra library or use the built-in (but slower) default, see the next section. Source: https://numpy.org/doc/stable/user/building.html https://numpy.org/doc/stable/user/building.html
      - dagmx 4y ago
        Added back the very next release https://numpy.org/doc/stable/release/1.21.0-notes.html https://numpy.org/doc/stable/release/1.21.0-notes.html > With the release of macOS 11.3, several different issues that numpy was encountering when using Accelerate Framework’s implementation of BLAS and LAPACK should be resolved.
  - remram 4y ago
    Interesting! Using what compiler?
- orangepurple 4y ago
  I had to do a double take. The Rust implementation is slower and harder to maintain. I recommend adding a cythonized function and a numba jitted function to the benchmark for completeness.
ryantgtg 4y ago
Meta comment: “<doing something>… with Rust” feels like a HN meme to me. I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.
- timeon 4y ago
  > followed this format And comments like yours are following one as well.
  - ryantgtg 4y ago
    As is yours!
- smcl 4y ago
  I'm not a Rust developer, but it seems to me that there are very real benefits to using the language. So I think the reason there are so many "I did $task in Rust!" - where $task is something usually accomplished in C, the current lingua franca of lower-level systems programming - is that people are genuinely excited to share that it really is looking possible that Rust can be used in place of C. So it's a bit of a meme, sure, but there is a good reason for it :)
  - ryantgtg 4y ago
    Yeah, without knowing much about Rust, I’ve decided from these types of posts that I should gravitate toward “app I use but now it’s in Rust.” I use plenty of Rust programs simply because people seem excited about it.
    - smcl 4y ago
      Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort to do so with any level of quality (and then maintain it) would be pretty massive. However another genre is more interesting, to me at least - utilising existing interfaces to extend software people already use without having to make them throw out what currently works and take the risk of replacing it with a recently implemented version. In this case someone implemented a Python module in Rust, in others entire Linux device drivers have been implemented in Rust. Lower-level programming in embedded systems seems like a good application too, but I don't know how many architectures Rust can target and is officially supported on. So to summarise, the two genres[0] I've identified are, roughly: - I did a RIIR[1] of 50% of grep's functionality for fun - Here is how you can accomplish a common task in C using Rust instead As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written. [0] - both are valid, and more exist but these are two common ones that came up [1] - RIIR = "Rewrite It In Rust", which was a bit of a meme for a while as a bunch of devs got excited about Rust and launched projects of varying completeness reimplementing things.
remram 4y ago
This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it. Thankfully there are GitHub Actions and similar tools that help with this.
- dz0ny 4y ago
  Yes, this. If you run something that's not in the binary cache you are an in a lot of pain because of Cargo. I have no idea how it works but it's painfully slow (is there some sort of network speed limit implemented? or is it downloading entire git history? )
- kodablah 4y ago
  Maturin wasn't powerful enough for our use cases, so we used raw setuptools-rust and cibuildwheel. Even with abi3 compat, yes it's not always easy to get the wheels exactly as you want. And with a lack of ARM runners by default with GH actions, you'll most likely be paying for your own CI instances (or wait forever for cibuildwheel cross compilation/qemu). Also for others doing this, the rust-cache GH action saves a lot of rebuild time too.
- rpep 4y ago
  Generally for scientific libraries where this is common, the expectation is from many users that you'll make it available via Conda Forge/Spack/EasyBuild for easier distribution, and that anyone wanting anything else will either get a manylinux wheel targeting a lowest common denominator machine, or will have to build from source.
- e3bc54b2 4y ago
  Yeah, Python's dependency management is already abysmal, but when lower level language modules get involved its suddenly whole different level of hell. This actually put me off Python entirely. Sadly no real alternative in ML.
wcrossbow 4y ago
I would have never thought my humble blog would end up in the front page of HN! Please be merciful :)
rpep 4y ago
I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementation, but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently. For e.g. doing things like add two arrays then multiply them by another array and subtract a constant. If I put that in a C or Rust function I’d expect that to be auto-vectorized using fewer operations than what NumPy will do, and you of course also remove the overhead of the foreign function interface and control returning to the interpreter. On top of that, it’s usually trivial to drop in thread level parallelism in a lower level language, and NumPy doesn’t operate with core level parallelism out of the box for most operations, you have to use something like numexpr to achieve it. I've personally used OpenMP + SIMD intrinsics or auto-vectorisation to great effect in scientific software and got performance >100x that of NumPy, but it's certainly not a free lunch. The question you have to ask yourself is normally “is it worth it”, and that’s something that can only be answered by profiling, looking at the overhead of maintaining the architecture, and understanding whether the increased friction with debugging and building is worth the hassle. Additionally, if you're doing linear algebra then generally if you're dropping to a lower level language you still will want to be using the same libraries that NumPy etc. uses unless you're exploiting some property of the matrix structure that can make operations more efficient in memory or compute. I worked on one codebase for e.g. where there was a block matrix structure and a hand written implementation of a solver was written because much more efficient operations could be performed with knowledge of that structure.
- jacquesm 4y ago
  NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scripting language can perform so well for compute intensive tasks. Of course, technically you are just stringing together highly optimized low level functions together using the Python language but the advantages of doing it that way have more often than not surprised me.
Ultimatt 4y ago
This blog article was missing the first step of using Numba on numpy https://numba.pydata.org/ https://numba.pydata.org/
waynehn 4y ago
This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency. docs: https://docs.greptime.com/user-guide/coprocessor-and-scripting https://docs.greptime.com/user-guide/coprocessor-and-scripti... code: https://github.com/GreptimeTeam/greptimedb/tree/develop/src/script https://github.com/GreptimeTeam/greptimedb/tree/develop/src/...
- orangepurple 4y ago
  Python 3 can be executed server side by Postgres as well CREATE FUNCTION pymax (a integer, b integer) RETURNS integer AS $$ if a > b: return a return b $$ LANGUAGE plpython3u; https://www.postgresql.org/docs/current/plpython.html https://www.postgresql.org/docs/current/plpython.html
  - waynehn 4y ago
    Actually Postgres does it better. Greptime only supports writing Python logic as a post cooprocessor for now, but not as an in-query function. We are trying to evolve towards the Postgres' functionality you mention.
  - VWWHFSfQ 4y ago
    Python running in Postgres sounds like it would be unusably slow. What's the performance like?
    - orangepurple 4y ago
      I have no idea who uses it.
amelius 4y ago
To be honest, that Rust syntax looks pretty horrible for something as simple as multiplying a vector by a scalar.
- jeroenhd 4y ago
  Definitely. However, the equivalent for the array multiplication in numpy looks like this: https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c58d1def6743fcf83/numpy/core/src/multiarray/number.c#L37 https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c... + https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c58d1def6743fcf83/numpy/core/src/multiarray/number.c#L688 https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c... To know what they do, you need the source for INPLACE_GIVE_UP_IF_NEEDED (https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f020ed8303614a87f/numpy/core/src/common/binop_override.h#L190 https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f...) and PyArray_GenericInplaceBinaryFunction (I don't even know where that's coming from, it's not defined in Numpy, maybe it's part of the Python interface?). In the end, both are unreadable in their own way. I personally prefer the Rust version above to the macrofied C version that's in Numpy but that's a matter of taste. I'd also trust the safe Rust implementations more than the C implementations because of the memory management guarantees Rust provides, though I suppose for simple operations like multiplication it'll be easier to make the program safe enough in C.
- Matumio 4y ago
  Part of that is the ndarray crate, which IMO is too generic often, making syntax complicated. The nalgebra crate is a bit nicer IMO, example: https://github.com/martinxyz/progenitor/blob/85260/crates/progenitor/src/builders/nn.rs#L33-L43 https://github.com/martinxyz/progenitor/blob/85260/crates/pr...
the__alchemist 4y ago
I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot. The surface plot brings my computer to a crawl. And, I'm not sure how to make it interactive. I gave up on it a few days ago. Could have tried a Rust module, but ended up throwing together a custom Rust plotter in a few hours, with translated Python code. (I already have a basic WGPU-based rendering engine, with EGUI for UI). It runs much faster. They're both imperitive langs, so you can copy and paste, replacing `**` with `pow()`, `np.exp(x)` with `x.exp()` etc. And indexing with x[i][j][k] instead of vectorized numpy ops. Ie the program compiles and runs in release mode faster than Python/numpy could perform the calcs. And the 3D graphics are smooth instead of a slideshow. And, I can make it real-time interactive since I'm using a low-level lang that integrates with GPU APIs directly. I'm not sure how feasible that would be in Python.
- europeanguy 4y ago
  Have a look at Julia. Julia is fast, and in addition its differential equations lib is possibly the best in the world.
  - the__alchemist 4y ago
    I've used it in the past, but not recently. I remember that Diffeq lib, and it being best-in-class! Does it work for 3D (PDEs?) I remember having trouble applying it to that domain earlier, although it was outstanding for ODEs. I think the big issue with Julia here is (other than the improving(?) JIT slowness (IIRC it was faster to compile and run a Rust program than JIT a Julia script), is I'm not sure how I'd interface with the GPU for graphics, compute shaders, and a GUI. Julia's mathematical syntax is best-in-class; I wish more langs used something like that.
    - ChrisRackauckas 4y ago
      There are some nice tools for 3D PDEs which connect to DiffEq like GridAP (https://docs.sciml.ai/Gridap/stable/ https://docs.sciml.ai/Gridap/stable/) and Ferrite (https://docs.sciml.ai/Ferrite/stable/ https://docs.sciml.ai/Ferrite/stable/). PDE tooling is where focus has been moving to as things evolve. As for JIT, just today there was a PR that was merged that makes Julia cache and reuse binaries of packages (https://github.com/JuliaLang/julia/pull/47184 https://github.com/JuliaLang/julia/pull/47184). It won't be out until the next release of Julia, but it's a pretty major improvement to not JIT fully inferred package calls.
ok123456 4y ago
Now compare it to f2py.
super256 4y ago
I wish the other way round was as easy as supporting Lua.
warangal 4y ago
Compiling extensions to python from another language also provide opportunities to fuse two or more operations, resulting in speed up even for cases where fast but independent implementation of operations are already available in python. And may be really worth it if that functionality is needed frequently. I have been writing python extensions[0] using Nim language for a while now and it has been a very smooth experience every time. https://ramanlabs.in/static/blog/Generate_Python_extensions_using_Nim_language.html https://ramanlabs.in/static/blog/Generate_Python_extensions_...
itamarst 4y ago
If you're writing a compiled extension, you can use Rust. Or Cython. Or C. Or C++. Which should you use? TL;DR: * If you're wrapping existing C library, I'd use Cython. * If you're wrapping existing C++ library, I'd use PyBind11 (no personal experience, but it's based on Boost::Python, which I have happily used). Cython in theory does C++ but it's a frustrating, limited experience. * If you're writing a tiny library and you don't know Rust, and you're not worried about memory safety, Cython is nice. * For anything involving writing extensive new low-level code, Rust with PyO3. Memory safety _will_ bite you in the ass. Concurrency is vastly easier with Rust. You get a package manager for dependencies. You're not writing a pile of code in a language without good tooling (Cython). Long form, with more alternatives and use cases: https://pythonspeed.com/articles/rust-cython-python-extensions/ https://pythonspeed.com/articles/rust-cython-python-extensio...
vmchale 4y ago
Thank you to the author, this is a fun post!
LarsDu88 4y ago
How does this compare with numba?

214 points by behnamoh 4y ago

I implemented a well known curve simplification algorithm in Rust, and was pleasantly surprised how easy the interaction with Python was. For packaging, setuptools_rust was great, and I too used PyO3 for the bindings. I haven't tried Rumpy yet, but it looks interesting.

masklinn 4y ago
> I haven't tried Rumpy yet, but it looks interesting. Do you mean maturin? Rumpy is TFA's demo project.
- shoobs 4y ago
  Oops, yep, my mistake.

college_physics 4y ago

> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python should really look whether numpy helps before complicating their codebase more. But there are more benefits to those bindings besides performance so its really nice to see the expanding options

dj_mc_merlin 4y ago
The diagram shows Numpy (orange line) below "Rumpy" (blue line). Since the y axis is time, less is more, so Numpy is faster indeed.
dagmx 4y ago
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which means you need to be somewhat careful if you ever relocate an install of it to a new machine
- orlp 4y ago
  And note that for example on Apple M1 it's essentially impossible to beat an implementation that uses Apple's Accelerate library for things like matrix multiplication, because Apple uses undocumented instructions unavailable to the public in that library.
  - orangepurple 4y ago
    Deprecated since version 1.20: The native libraries on macOS, provided by Accelerate, are not fit for use in NumPy since they have bugs that cause wrong output under easily reproducible conditions. If the vendor fixes those bugs, the library could be reinstated, but until then users compiling for themselves should use another linear algebra library or use the built-in (but slower) default, see the next section. Source: https://numpy.org/doc/stable/user/building.html https://numpy.org/doc/stable/user/building.html
    - dagmx 4y ago
      Added back the very next release https://numpy.org/doc/stable/release/1.21.0-notes.html https://numpy.org/doc/stable/release/1.21.0-notes.html > With the release of macOS 11.3, several different issues that numpy was encountering when using Accelerate Framework’s implementation of BLAS and LAPACK should be resolved.
- remram 4y ago
  Interesting! Using what compiler?
orangepurple 4y ago
I had to do a double take. The Rust implementation is slower and harder to maintain. I recommend adding a cythonized function and a numba jitted function to the benchmark for completeness.

ryantgtg 4y ago

Meta comment: “<doing something>… with Rust” feels like a HN meme to me. I know nothing about Rust, but I’ve noticed that there’s a top post on HN every day that followed this format.

timeon 4y ago
> followed this format And comments like yours are following one as well.
- ryantgtg 4y ago
  As is yours!
smcl 4y ago
I'm not a Rust developer, but it seems to me that there are very real benefits to using the language. So I think the reason there are so many "I did $task in Rust!" - where $task is something usually accomplished in C, the current lingua franca of lower-level systems programming - is that people are genuinely excited to share that it really is looking possible that Rust can be used in place of C. So it's a bit of a meme, sure, but there is a good reason for it :)
- ryantgtg 4y ago
  Yeah, without knowing much about Rust, I’ve decided from these types of posts that I should gravitate toward “app I use but now it’s in Rust.” I use plenty of Rust programs simply because people seem excited about it.
  - smcl 4y ago
    Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort to do so with any level of quality (and then maintain it) would be pretty massive. However another genre is more interesting, to me at least - utilising existing interfaces to extend software people already use without having to make them throw out what currently works and take the risk of replacing it with a recently implemented version. In this case someone implemented a Python module in Rust, in others entire Linux device drivers have been implemented in Rust. Lower-level programming in embedded systems seems like a good application too, but I don't know how many architectures Rust can target and is officially supported on. So to summarise, the two genres[0] I've identified are, roughly: - I did a RIIR[1] of 50% of grep's functionality for fun - Here is how you can accomplish a common task in C using Rust instead As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written. [0] - both are valid, and more exist but these are two common ones that came up [1] - RIIR = "Rewrite It In Rust", which was a bit of a meme for a while as a bunch of devs got excited about Rust and launched projects of varying completeness reimplementing things.

remram 4y ago

This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it. Thankfully there are GitHub Actions and similar tools that help with this.

dz0ny 4y ago
Yes, this. If you run something that's not in the binary cache you are an in a lot of pain because of Cargo. I have no idea how it works but it's painfully slow (is there some sort of network speed limit implemented? or is it downloading entire git history? )
kodablah 4y ago
Maturin wasn't powerful enough for our use cases, so we used raw setuptools-rust and cibuildwheel. Even with abi3 compat, yes it's not always easy to get the wheels exactly as you want. And with a lack of ARM runners by default with GH actions, you'll most likely be paying for your own CI instances (or wait forever for cibuildwheel cross compilation/qemu). Also for others doing this, the rust-cache GH action saves a lot of rebuild time too.
rpep 4y ago
Generally for scientific libraries where this is common, the expectation is from many users that you'll make it available via Conda Forge/Spack/EasyBuild for easier distribution, and that anyone wanting anything else will either get a manylinux wheel targeting a lowest common denominator machine, or will have to build from source.
e3bc54b2 4y ago
Yeah, Python's dependency management is already abysmal, but when lower level language modules get involved its suddenly whole different level of hell. This actually put me off Python entirely. Sadly no real alternative in ML.

wcrossbow 4y ago

I would have never thought my humble blog would end up in the front page of HN! Please be merciful :)

rpep 4y ago

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementation, but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently. For e.g. doing things like add two arrays then multiply them by another array and subtract a constant. If I put that in a C or Rust function I’d expect that to be auto-vectorized using fewer operations than what NumPy will do, and you of course also remove the overhead of the foreign function interface and control returning to the interpreter. On top of that, it’s usually trivial to drop in thread level parallelism in a lower level language, and NumPy doesn’t operate with core level parallelism out of the box for most operations, you have to use something like numexpr to achieve it. I've personally used OpenMP + SIMD intrinsics or auto-vectorisation to great effect in scientific software and got performance >100x that of NumPy, but it's certainly not a free lunch. The question you have to ask yourself is normally “is it worth it”, and that’s something that can only be answered by profiling, looking at the overhead of maintaining the architecture, and understanding whether the increased friction with debugging and building is worth the hassle. Additionally, if you're doing linear algebra then generally if you're dropping to a lower level language you still will want to be using the same libraries that NumPy etc. uses unless you're exploiting some property of the matrix structure that can make operations more efficient in memory or compute. I worked on one codebase for e.g. where there was a block matrix structure and a hand written implementation of a solver was written because much more efficient operations could be performed with knowledge of that structure.

jacquesm 4y ago
NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scripting language can perform so well for compute intensive tasks. Of course, technically you are just stringing together highly optimized low level functions together using the Python language but the advantages of doing it that way have more often than not surprised me.

Ultimatt 4y ago

This blog article was missing the first step of using Numba on numpy https://numba.pydata.org/ https://numba.pydata.org/

waynehn 4y ago

This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency. docs: https://docs.greptime.com/user-guide/coprocessor-and-scripting https://docs.greptime.com/user-guide/coprocessor-and-scripti... code: https://github.com/GreptimeTeam/greptimedb/tree/develop/src/script https://github.com/GreptimeTeam/greptimedb/tree/develop/src/...

orangepurple 4y ago
Python 3 can be executed server side by Postgres as well CREATE FUNCTION pymax (a integer, b integer) RETURNS integer AS $$ if a > b: return a return b $$ LANGUAGE plpython3u; https://www.postgresql.org/docs/current/plpython.html https://www.postgresql.org/docs/current/plpython.html
- waynehn 4y ago
  Actually Postgres does it better. Greptime only supports writing Python logic as a post cooprocessor for now, but not as an in-query function. We are trying to evolve towards the Postgres' functionality you mention.
- VWWHFSfQ 4y ago
  Python running in Postgres sounds like it would be unusably slow. What's the performance like?
  - orangepurple 4y ago
    I have no idea who uses it.

amelius 4y ago

To be honest, that Rust syntax looks pretty horrible for something as simple as multiplying a vector by a scalar.

jeroenhd 4y ago
Definitely. However, the equivalent for the array multiplication in numpy looks like this: https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c58d1def6743fcf83/numpy/core/src/multiarray/number.c#L37 https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c... + https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c58d1def6743fcf83/numpy/core/src/multiarray/number.c#L688 https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c... To know what they do, you need the source for INPLACE_GIVE_UP_IF_NEEDED (https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f020ed8303614a87f/numpy/core/src/common/binop_override.h#L190 https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f...) and PyArray_GenericInplaceBinaryFunction (I don't even know where that's coming from, it's not defined in Numpy, maybe it's part of the Python interface?). In the end, both are unreadable in their own way. I personally prefer the Rust version above to the macrofied C version that's in Numpy but that's a matter of taste. I'd also trust the safe Rust implementations more than the C implementations because of the memory management guarantees Rust provides, though I suppose for simple operations like multiplication it'll be easier to make the program safe enough in C.
Matumio 4y ago
Part of that is the ndarray crate, which IMO is too generic often, making syntax complicated. The nalgebra crate is a bit nicer IMO, example: https://github.com/martinxyz/progenitor/blob/85260/crates/progenitor/src/builders/nn.rs#L33-L43 https://github.com/martinxyz/progenitor/blob/85260/crates/pr...

the__alchemist 4y ago

I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot. The surface plot brings my computer to a crawl. And, I'm not sure how to make it interactive. I gave up on it a few days ago. Could have tried a Rust module, but ended up throwing together a custom Rust plotter in a few hours, with translated Python code. (I already have a basic WGPU-based rendering engine, with EGUI for UI). It runs much faster. They're both imperitive langs, so you can copy and paste, replacing `**` with `pow()`, `np.exp(x)` with `x.exp()` etc. And indexing with x[i][j][k] instead of vectorized numpy ops. Ie the program compiles and runs in release mode faster than Python/numpy could perform the calcs. And the 3D graphics are smooth instead of a slideshow. And, I can make it real-time interactive since I'm using a low-level lang that integrates with GPU APIs directly. I'm not sure how feasible that would be in Python.

europeanguy 4y ago
Have a look at Julia. Julia is fast, and in addition its differential equations lib is possibly the best in the world.
- the__alchemist 4y ago
  I've used it in the past, but not recently. I remember that Diffeq lib, and it being best-in-class! Does it work for 3D (PDEs?) I remember having trouble applying it to that domain earlier, although it was outstanding for ODEs. I think the big issue with Julia here is (other than the improving(?) JIT slowness (IIRC it was faster to compile and run a Rust program than JIT a Julia script), is I'm not sure how I'd interface with the GPU for graphics, compute shaders, and a GUI. Julia's mathematical syntax is best-in-class; I wish more langs used something like that.
  - ChrisRackauckas 4y ago
    There are some nice tools for 3D PDEs which connect to DiffEq like GridAP (https://docs.sciml.ai/Gridap/stable/ https://docs.sciml.ai/Gridap/stable/) and Ferrite (https://docs.sciml.ai/Ferrite/stable/ https://docs.sciml.ai/Ferrite/stable/). PDE tooling is where focus has been moving to as things evolve. As for JIT, just today there was a PR that was merged that makes Julia cache and reuse binaries of packages (https://github.com/JuliaLang/julia/pull/47184 https://github.com/JuliaLang/julia/pull/47184). It won't be out until the next release of Julia, but it's a pretty major improvement to not JIT fully inferred package calls.

ok123456 4y ago

Now compare it to f2py.

super256 4y ago

I wish the other way round was as easy as supporting Lua.

warangal 4y ago

Compiling extensions to python from another language also provide opportunities to fuse two or more operations, resulting in speed up even for cases where fast but independent implementation of operations are already available in python. And may be really worth it if that functionality is needed frequently. I have been writing python extensions[0] using Nim language for a while now and it has been a very smooth experience every time. https://ramanlabs.in/static/blog/Generate_Python_extensions_using_Nim_language.html https://ramanlabs.in/static/blog/Generate_Python_extensions_...

itamarst 4y ago

If you're writing a compiled extension, you can use Rust. Or Cython. Or C. Or C++. Which should you use? TL;DR: * If you're wrapping existing C library, I'd use Cython. * If you're wrapping existing C++ library, I'd use PyBind11 (no personal experience, but it's based on Boost::Python, which I have happily used). Cython in theory does C++ but it's a frustrating, limited experience. * If you're writing a tiny library and you don't know Rust, and you're not worried about memory safety, Cython is nice. * For anything involving writing extensive new low-level code, Rust with PyO3. Memory safety _will_ bite you in the ass. Concurrency is vastly easier with Rust. You get a package manager for dependencies. You're not writing a pile of code in a language without good tooling (Cython). Long form, with more alternatives and use cases: https://pythonspeed.com/articles/rust-cython-python-extensions/ https://pythonspeed.com/articles/rust-cython-python-extensio...

vmchale 4y ago

Thank you to the author, this is a fun post!

LarsDu88 4y ago

How does this compare with numba?