16 ms·

JNumPy: Writing high-performance C extensions for Python in minutes

125 points by xgdgsc 4y ago

metadat 4y ago
(Writing Python extensions in Julia) ..which could be a heckuva lot friendlier and often safer than C! The requisite python boilerplate is pretty ugly, though: init_jl() include_src('example.jl', __file__) exec_julia('example.init()') I also don't understand how jl_mat_mul gets mapped to mat_mul. jl_mat_mul = Pyfunc(jl_mat_mul) Huh?
- kortex 4y ago
  Yeah, the boilerplate is rough, but python's import system is magic. There's no reason why you couldn't just do from example.jl import mat_mul and have it just work. (With the right libraries and import functionality of course.)
- majoe 4y ago
  I had a look at the demo folder, that's just a typo. It should be: jl_mat_mul = Pyfunc(mat_mul) which makes a lot more sense.
cycomanic 4y ago
Unless I'm misunderstanding, this seems to be writing python extensions in Julia not C. So quite a misleading title. Also why should I pick this over cython, pythran or numba? with those I don't need to learn how to optimise another language (and no just writing Julia code does not necessarily get you large speed ups and certainly not the speed of C in many cases). I really wish the Julia community would stop overselling the language.
- adgjlsfhk1 4y ago
  Do you have any source on well written julia being any slower than well written C?
  - bigdict 4y ago
    https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia-gcc.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
    - webshit2 4y ago
      It says right there: "Always look at the source code" fannkuch-redux, n-body, spectral-norm, reverse-complement: the fastest C wins by a large margin because it's an unreadable mess of vector intrinsics. If you really wanted to, you can do that in Julia too (and it might read better). Julia looks pretty even with C in the ones that were faithfully translated.
  - jakobnissen 4y ago
    Don't be unreasonable. Numerical Julia code is just as fast, precisely because it's easy to design efficient high-level APIs and let the compiler optimise it. Try timing Julia's IO, or it's string processing, or it's hash tables, or its array allocation or it's GC - you know, nearly everything other that mathematical operations on floats or ints. Nearly everything leaves performance on the table. Sure Julia is fast, but not on a C level in these areas. There is still lots of room for optimisation, though.
    - jkrubin 4y ago
      What’s your point?
      - anewhnaccount2 4y ago
        The point is that these things (string processing/hash tables) are typically done using custom/tuned code in C and so would be faster than the generic, and not particularly carefully tuned Julia implementations. However, given this library seems to be about passing NumPy arrays to/from Julia, the application is almost certainly to expose numerical stuff from Julia to Python. Basically, the use case is similar to Numba, but it works by producing a C-extension from a Julia system image (if I understand correctly). The main advantage over something like Numba would be access to the Julia ecosystem.
- pizza 4y ago
  This is why it's a C extension: https://github.com/Suzhou-Tongyuan/jnumpy/blob/main/TyPython/src/C.jl https://github.com/Suzhou-Tongyuan/jnumpy/blob/main/TyPython... The title is correct because it is a Julia numpy interface. Inside it seems is another project, TyPython, that provides an efficient Julia-Python-Numpy bridge, via C extensions.
  - anewhnaccount2 4y ago
    Agreed that the title is technically accurate. Julia should also be in the title, but this is possibly an oversight. It seems a bit unnecessary for the GP to assume that this is a bad faith attempt to mislead and insert Julia where it is not welcome rather than a potentially useful extra tool for writing high performance code. Note that the author/poster is probably not a native English speaker.
  - cycomanic 4y ago
    I would argue that it might technically be correct (and admittedly that's the best type of correct) in the strict sense, it is also misleading. If I read that I'm writing C extensions in minutes I at least expect that the extension compiles to C (which the Julia code doesn't).
    - crabbygrabby 4y ago
      Definitely misleading given all the bloat and baggage adding Julia as a dependency to a project can have. It's the heaviest glue ever.
- anewhnaccount2 4y ago
  > Also why should I pick this over cython, pythran or numba? One scenario would be that there is code already written in Julia, which you would like to expose to a broader audience. Although overall Julia's ecosystem is smaller than Python's, there are some niches within which it is highly developed. Library code accessible from the alternatives you listed is mainly limited to existing Python and C code. See the demo with `ParallelKMeans.jl`, which wraps a high performance Julia library performing K-means clustering, which could be a useful addition to the Python ecosystem. Another scenario is that you want to write some custom code which makes use of the Julia ecosystem and then expose it to Python.
- oefrha 4y ago
  Title says “C extension” which is accurate, not “extension written in C”. You can write C extensions for CPython in non-C languages with a bridge, e.g. PyO3 for Rust, but the extension ultimately has to use the C API and ABI.
- crabbygrabby 4y ago
  I wish they would stop overselling too. There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Pretty toxic community too.
  - dunefox 4y ago
    > Pretty toxic community too. Quite the opposite according to my own experience.
  - Sukera 4y ago
    > There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Can you expand on that, so that it's possible to improve on them?
    - JZerf 4y ago
      It seems to use a lot of memory which can be unacceptable for many people and can become impractical if you plan on running several programs simultaneously. For several of the problems at https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... on the Computer Language Benchmarks Games, it shows the Julia programs as using several times more memory than similar Java programs. It's even worse when comparing to C, C++, Rust, etc...
      - ChrisRackauckas 4y ago
        Remember that for those benchmarks, C, C++, and Rust compile the binaries and then run the benchmark, while Julia compiles as part of the benchmark. The memory usage is dramatically decreased if you do a workflow similar to the other languages using something like PackageCompiler.jl to build a binary that is benchmarked. If you treat the other languages as "JIT" and include those factors in the benchmarking process, it's a lot closer.
        4y ago
pizza 4y ago
Very cool and nice work.
dbcooper 4y ago
Great to see contributions from a fellow Suzhou Julia enthusiast.
TekMol 4y ago
Python itself has to become faster. For the love of god. Make it as fast as PHP. My biggest pain point in computer science is that for every project I have to decide to either cope with PHP's butt-ugly namespace system or with Python's masochistic slowness. Please, Python developers. Take a closer look at how PHP achieves its speed, hold all other development and copy whatever they do! The result would be heaven.
- anewhnaccount2 4y ago
  It is happening: https://github.com/faster-cpython/ideas https://github.com/faster-cpython/ideas
  - TekMol 4y ago
    It seems to indicate a 30% performance boost? That would be a step into the right direction. 6 more 30% improvements and it is on par with PHP.
  - lraxl 4y ago
    It will yield a small speedup (< 50%) at best, probably for selected benchmarks that will be widely promoted. Microsoft is misallocating its resources. The scientific ecosystem should be ported to .NET, with first class support for F#.
    - machinekob 4y ago
      They already try it like 2/3 times and it didnt take off so why should it be different today?
      - melony 4y ago
        Everytime they did it, it was half baked at best. Scientific libraries need driver support, Microsoft always takes the happy path and ignores all the other hardware permutations. If you are not running Intel/Nvidia in a Win32 environment you are out of luck. WSL2, ARM, AMD, etc. don't have much synergy in terms of product support.
majoe 4y ago
Coincidentally I used PyJulia last week to call some numerical code I translated from Python/Numpy to Julia. The actual translation was straightforward and done in under one hour, but I needed an additional day for fine tuning and profiling to actually make it fast. Does anyone know, how JNumPy compares to PyJulia? Calling my Julia code via PyJulia seems to increase the startup time of Julia even more and it would be great, if JNumpy would do better here.
- sundarurfriend 4y ago
  Are you aware of the juliacall package? [1] The announcement post [2] says "Calling Julia from Python, juliacall seems to have smaller overhead (in time) than julia". The benchmarks below that show the overhead on calling `identity` with `juliacall` being less than half of what it is with PyJulia. [1] https://docs.juliahub.com/PythonCall/WdXsa/0.9.4/juliacall/ https://docs.juliahub.com/PythonCall/WdXsa/0.9.4/juliacall/ [2] https://discourse.julialang.org/t/ann-pythoncall-and-juliacall/76778 https://discourse.julialang.org/t/ann-pythoncall-and-juliaca...
ngcc_hk 4y ago
Should change title to at least hint this is a c-interface to Julia. The issue is not that you cannot use c. The key is if you have problem you have to know Julia.
Leo_Germond 4y ago
So, the example is about using Julia code bindings instead of numpy's BLAS bindings for "C-level performance"... that's a misguided optimization attempt by the book.
immmmmm 4y ago
for C++ folks i can only recommend pybind11 which provides a numpy interface. it's a breeze to use, header only.
tpoacher 4y ago
> import julia_numpy as np I don't get it. Does this override python's numpy library with a julia native? If so does it implement all of numpy functionality? Or does this piggyback on normal numpy, redirecting accordingly, but also allows plugging extensions written in julia? The documentation doesn't shed any light at all. Neither do the demos. Also, is this simply a python julia interface and the "high-performance" part of the description is just part of the usual juliaspeak? Or is there an actual "high-performance" component there that makes this more than a simple python julia interface?

125 points by xgdgsc 4y ago

(Writing Python extensions in Julia) ..which could be a heckuva lot friendlier and often safer than C! The requisite python boilerplate is pretty ugly, though: init_jl() include_src('example.jl', __file__) exec_julia('example.init()') I also don't understand how jl_mat_mul gets mapped to mat_mul. jl_mat_mul = Pyfunc(jl_mat_mul) Huh?

kortex 4y ago
Yeah, the boilerplate is rough, but python's import system is magic. There's no reason why you couldn't just do from example.jl import mat_mul and have it just work. (With the right libraries and import functionality of course.)
majoe 4y ago
I had a look at the demo folder, that's just a typo. It should be: jl_mat_mul = Pyfunc(mat_mul) which makes a lot more sense.

cycomanic 4y ago

Unless I'm misunderstanding, this seems to be writing python extensions in Julia not C. So quite a misleading title. Also why should I pick this over cython, pythran or numba? with those I don't need to learn how to optimise another language (and no just writing Julia code does not necessarily get you large speed ups and certainly not the speed of C in many cases). I really wish the Julia community would stop overselling the language.

adgjlsfhk1 4y ago
Do you have any source on well written julia being any slower than well written C?
- bigdict 4y ago
  https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia-gcc.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - webshit2 4y ago
    It says right there: "Always look at the source code" fannkuch-redux, n-body, spectral-norm, reverse-complement: the fastest C wins by a large margin because it's an unreadable mess of vector intrinsics. If you really wanted to, you can do that in Julia too (and it might read better). Julia looks pretty even with C in the ones that were faithfully translated.
- jakobnissen 4y ago
  Don't be unreasonable. Numerical Julia code is just as fast, precisely because it's easy to design efficient high-level APIs and let the compiler optimise it. Try timing Julia's IO, or it's string processing, or it's hash tables, or its array allocation or it's GC - you know, nearly everything other that mathematical operations on floats or ints. Nearly everything leaves performance on the table. Sure Julia is fast, but not on a C level in these areas. There is still lots of room for optimisation, though.
  - jkrubin 4y ago
    What’s your point?
    - anewhnaccount2 4y ago
      The point is that these things (string processing/hash tables) are typically done using custom/tuned code in C and so would be faster than the generic, and not particularly carefully tuned Julia implementations. However, given this library seems to be about passing NumPy arrays to/from Julia, the application is almost certainly to expose numerical stuff from Julia to Python. Basically, the use case is similar to Numba, but it works by producing a C-extension from a Julia system image (if I understand correctly). The main advantage over something like Numba would be access to the Julia ecosystem.
pizza 4y ago
This is why it's a C extension: https://github.com/Suzhou-Tongyuan/jnumpy/blob/main/TyPython/src/C.jl https://github.com/Suzhou-Tongyuan/jnumpy/blob/main/TyPython... The title is correct because it is a Julia numpy interface. Inside it seems is another project, TyPython, that provides an efficient Julia-Python-Numpy bridge, via C extensions.
- anewhnaccount2 4y ago
  Agreed that the title is technically accurate. Julia should also be in the title, but this is possibly an oversight. It seems a bit unnecessary for the GP to assume that this is a bad faith attempt to mislead and insert Julia where it is not welcome rather than a potentially useful extra tool for writing high performance code. Note that the author/poster is probably not a native English speaker.
- cycomanic 4y ago
  I would argue that it might technically be correct (and admittedly that's the best type of correct) in the strict sense, it is also misleading. If I read that I'm writing C extensions in minutes I at least expect that the extension compiles to C (which the Julia code doesn't).
  - crabbygrabby 4y ago
    Definitely misleading given all the bloat and baggage adding Julia as a dependency to a project can have. It's the heaviest glue ever.
anewhnaccount2 4y ago
> Also why should I pick this over cython, pythran or numba? One scenario would be that there is code already written in Julia, which you would like to expose to a broader audience. Although overall Julia's ecosystem is smaller than Python's, there are some niches within which it is highly developed. Library code accessible from the alternatives you listed is mainly limited to existing Python and C code. See the demo with `ParallelKMeans.jl`, which wraps a high performance Julia library performing K-means clustering, which could be a useful addition to the Python ecosystem. Another scenario is that you want to write some custom code which makes use of the Julia ecosystem and then expose it to Python.
oefrha 4y ago
Title says “C extension” which is accurate, not “extension written in C”. You can write C extensions for CPython in non-C languages with a bridge, e.g. PyO3 for Rust, but the extension ultimately has to use the C API and ABI.
crabbygrabby 4y ago
I wish they would stop overselling too. There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Pretty toxic community too.
- dunefox 4y ago
  > Pretty toxic community too. Quite the opposite according to my own experience.
- Sukera 4y ago
  > There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Can you expand on that, so that it's possible to improve on them?
  - JZerf 4y ago
    It seems to use a lot of memory which can be unacceptable for many people and can become impractical if you plan on running several programs simultaneously. For several of the problems at https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/julia.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... on the Computer Language Benchmarks Games, it shows the Julia programs as using several times more memory than similar Java programs. It's even worse when comparing to C, C++, Rust, etc...
    - ChrisRackauckas 4y ago
      Remember that for those benchmarks, C, C++, and Rust compile the binaries and then run the benchmark, while Julia compiles as part of the benchmark. The memory usage is dramatically decreased if you do a workflow similar to the other languages using something like PackageCompiler.jl to build a binary that is benchmarked. If you treat the other languages as "JIT" and include those factors in the benchmarking process, it's a lot closer.
      - 4y ago

pizza 4y ago

Very cool and nice work.

dbcooper 4y ago

Great to see contributions from a fellow Suzhou Julia enthusiast.

TekMol 4y ago

Python itself has to become faster. For the love of god. Make it as fast as PHP. My biggest pain point in computer science is that for every project I have to decide to either cope with PHP's butt-ugly namespace system or with Python's masochistic slowness. Please, Python developers. Take a closer look at how PHP achieves its speed, hold all other development and copy whatever they do! The result would be heaven.

anewhnaccount2 4y ago
It is happening: https://github.com/faster-cpython/ideas https://github.com/faster-cpython/ideas
- TekMol 4y ago
  It seems to indicate a 30% performance boost? That would be a step into the right direction. 6 more 30% improvements and it is on par with PHP.
- lraxl 4y ago
  It will yield a small speedup (< 50%) at best, probably for selected benchmarks that will be widely promoted. Microsoft is misallocating its resources. The scientific ecosystem should be ported to .NET, with first class support for F#.
  - machinekob 4y ago
    They already try it like 2/3 times and it didnt take off so why should it be different today?
    - melony 4y ago
      Everytime they did it, it was half baked at best. Scientific libraries need driver support, Microsoft always takes the happy path and ignores all the other hardware permutations. If you are not running Intel/Nvidia in a Win32 environment you are out of luck. WSL2, ARM, AMD, etc. don't have much synergy in terms of product support.

majoe 4y ago

Coincidentally I used PyJulia last week to call some numerical code I translated from Python/Numpy to Julia. The actual translation was straightforward and done in under one hour, but I needed an additional day for fine tuning and profiling to actually make it fast. Does anyone know, how JNumPy compares to PyJulia? Calling my Julia code via PyJulia seems to increase the startup time of Julia even more and it would be great, if JNumpy would do better here.

sundarurfriend 4y ago
Are you aware of the juliacall package? [1] The announcement post [2] says "Calling Julia from Python, juliacall seems to have smaller overhead (in time) than julia". The benchmarks below that show the overhead on calling `identity` with `juliacall` being less than half of what it is with PyJulia. [1] https://docs.juliahub.com/PythonCall/WdXsa/0.9.4/juliacall/ https://docs.juliahub.com/PythonCall/WdXsa/0.9.4/juliacall/ [2] https://discourse.julialang.org/t/ann-pythoncall-and-juliacall/76778 https://discourse.julialang.org/t/ann-pythoncall-and-juliaca...

ngcc_hk 4y ago

Should change title to at least hint this is a c-interface to Julia. The issue is not that you cannot use c. The key is if you have problem you have to know Julia.

Leo_Germond 4y ago

So, the example is about using Julia code bindings instead of numpy's BLAS bindings for "C-level performance"... that's a misguided optimization attempt by the book.

immmmmm 4y ago

for C++ folks i can only recommend pybind11 which provides a numpy interface. it's a breeze to use, header only.

tpoacher 4y ago

> import julia_numpy as np I don't get it. Does this override python's numpy library with a julia native? If so does it implement all of numpy functionality? Or does this piggyback on normal numpy, redirecting accordingly, but also allows plugging extensions written in julia? The documentation doesn't shed any light at all. Neither do the demos. Also, is this simply a python julia interface and the "high-performance" part of the description is just part of the usual juliaspeak? Or is there an actual "high-performance" component there that makes this more than a simple python julia interface?