8 ms·

Can you call Julia routines from other languages? Otherwise you sort of have to go all in...

by cozzyd 3y ago

eigenspace 3y ago
Yes, julia can be called from other languages rather easily, Julia functions can be exposed and called with a C-like ABI [1], and then there's also various dedicated packages for languages like Python[2], R[3], C++[4], Rust[5] and more to call Julia code. With PackageCompiler.jl [6] you can even make AOT compiled standalone binaries, though these are rather large. They've shrunk a fair amount in recent releases, but they're still a lot of low hanging fruit to make the compiled binaries smaller, and some manual work you can do like removing LLVM and filtering stdlibs when they're not needed. Work is also happening on a more stable / mature system that acts like StaticCompiler.jl [7] except provided by the base language and people who are more experienced in the compiler (i.e. not a janky prototype) [1] https://docs.julialang.org/en/v1/manual/embedding/ https://docs.julialang.org/en/v1/manual/embedding/ [2] https://pypi.org/project/juliacall/ https://pypi.org/project/juliacall/ [3] https://www.rdocumentation.org/packages/JuliaCall/ https://www.rdocumentation.org/packages/JuliaCall/ [4] https://github.com/Clemapfel/jluna https://github.com/Clemapfel/jluna [5] https://github.com/Taaitaaiger/jlrs https://github.com/Taaitaaiger/jlrs [6] https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl [7] https://github.com/tshort/StaticCompiler.jl https://github.com/tshort/StaticCompiler.jl
- huijzer 3y ago
  > Yes, julia can be called from other languages rather easily This seems false to me. StaticCompiler.jl [1] puts in their limitations that "GC-tracked allocations and global variables do not work with compile_executable or compile_shlib. This has some interesting consequences, including that all functions within the function you want to compile must either be inlined or return only native types (otherwise Julia would have to allocate a place to put the results, which will fail)." In practice, this means that you can therefore not use the base library in your own programs. PackageCompiler.jl [2] has the same limitations if I'm not mistaken. So then you have to fall back to distributing the Julia "binary" with a full Julia runtime, which is pretty heavy. There are some packages, such as PySR [3], which do this. It seems pretty usable for research I'd say, but difficult to put in production. There is some word going around though that there is an even better static compiler in the making, but as long as that one is not publicly available I'd say that Julia cannot easily be called from other languages. [1]: https://github.com/tshort/StaticCompiler.jl https://github.com/tshort/StaticCompiler.jl [2]: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl [3]: https://github.com/MilesCranmer/PySR https://github.com/MilesCranmer/PySR
  - eigenspace 3y ago
    I think you misunderstood me. Nothing about calling julia from other languages requires StaticCompiler.jl, that's just an experimental proof of concept for small, standalone binary generation with some heavy limitations. Calling julia from another language does not in general require AOT compilation, though it does help to make things more portable and self contained. If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language. > PackageCompiler.jl [2] has the same limitations if I'm not mistaken You are mistaken. PackageCompiler works by basically bundling an entire julia system image with your program and the whole compiler, runtime, and stdlib compiled into one bundle together. If a piece of code works in the REPL it'll work from PackageCompiler
    - huijzer 3y ago
      > If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language. Okay but how do you distribute the program that depends on Julia to clients? You then also need to ensure that they have the right Julia available on their system. That's much heavier than dynamic binaries (which rely only on some shared libraries). My point was that it's not always "easy" to call Julia. That it's easy in HPC is a reasonable statement since you probably have full control over the system and it has loads of RAM available, but I wouldn't call it easy in general. > You are mistaken. I guess we're both right. I was talking about the "libraries" functionality [1]. [1]: https://julialang.github.io/PackageCompiler.jl/dev/libs.html https://julialang.github.io/PackageCompiler.jl/dev/libs.html
      - eigenspace 3y ago
        > Okay but how do you distribute the program that depends on Julia to clients? That's not the question that was asked though. The question the person asked as "Can you call Julia routines from other languages?" and the answer to that is "yes, you can easily do that". __________________________________________________ > I guess we're both right. I was talking about the "libraries" functionality I'm confused, what functionality exactly are you saying is missing from the libraries functionality? That very example you link to shows one using the julia runtime via printing. And yes, dynamic dispatch and dynamic code generation do work from libraries created by PackageCompiler.jl as well...
sertbdfgbnfgsd 3y ago
Going all in is precisely the point. The biggest selling point of Julia is that it's as easy to write as python if you don't care about performance, and if you do care about performance it's as fast as C++ if you can just follow some simple principles. This actually works in practice. For my day job I write the slow path exactly as I would python, and that's a really fast way to write code. If you want a slow language that can call fast code, you already have python.
- jampekka 3y ago
  Biggest selling point of Julia is that you're locked in to Julia if you use it? For me this is the biggest point for giving up on Julia. Julia has a lot going for it in the basic design and I was really excited about it in the beginning. But even after all these years the environment is sadly almost unusable if you want to do something saner than REPL/notebook. The TTFP is too damn high and I don't think it will ever be fixed.
  - sertbdfgbnfgsd 3y ago
    > environment is sadly almost unusable if you want to do something saner than REPL/notebook I work in "one of those industries" where failure can easily cost millions in the space of minutes, and we use Julia almost exclusively. Unfortunately I can't give you details because of reasons, other than tell you you're very poorly informed. > you're locked in to Julia You're not "locked in" to Julia any more you're locked in to Python. Arguably less, as Julia can call C natively, Python can't. My point isn't that you can't interop, and it seems you're intentionally (?) equivocating on this point. Surely you didn't misinterpret what I wrote as "you can't iterop"? My point is that you can more benefits if you don't. Like I said, if you want a slow language that can call fast routines, why not just use Python.
    - jampekka 3y ago
      Python can call C natively. Also Python programs can be used as CLI tools interacting with practically any language. Julia programs in practice can't. But I know from experience that this debate will not go anywhere. It's is the same old refusal to see the problem: "you're using it wrong". Julia is dying and this is why it will die.

Can you call Julia routines from other languages? Otherwise you sort of have to go all in...

by cozzyd 3y ago

Can you call Julia routines from other languages? Otherwise you sort of have to go all in...

Yes, julia can be called from other languages rather easily, Julia functions can be exposed and called with a C-like ABI [1], and then there's also various dedicated packages for languages like Python[2], R[3], C++[4], Rust[5] and more to call Julia code. With PackageCompiler.jl [6] you can even make AOT compiled standalone binaries, though these are rather large. They've shrunk a fair amount in recent releases, but they're still a lot of low hanging fruit to make the compiled binaries smaller, and some manual work you can do like removing LLVM and filtering stdlibs when they're not needed. Work is also happening on a more stable / mature system that acts like StaticCompiler.jl [7] except provided by the base language and people who are more experienced in the compiler (i.e. not a janky prototype) [1] https://docs.julialang.org/en/v1/manual/embedding/ https://docs.julialang.org/en/v1/manual/embedding/ [2] https://pypi.org/project/juliacall/ https://pypi.org/project/juliacall/ [3] https://www.rdocumentation.org/packages/JuliaCall/ https://www.rdocumentation.org/packages/JuliaCall/ [4] https://github.com/Clemapfel/jluna https://github.com/Clemapfel/jluna [5] https://github.com/Taaitaaiger/jlrs https://github.com/Taaitaaiger/jlrs [6] https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl [7] https://github.com/tshort/StaticCompiler.jl https://github.com/tshort/StaticCompiler.jl

huijzer 3y ago
> Yes, julia can be called from other languages rather easily This seems false to me. StaticCompiler.jl [1] puts in their limitations that "GC-tracked allocations and global variables do not work with compile_executable or compile_shlib. This has some interesting consequences, including that all functions within the function you want to compile must either be inlined or return only native types (otherwise Julia would have to allocate a place to put the results, which will fail)." In practice, this means that you can therefore not use the base library in your own programs. PackageCompiler.jl [2] has the same limitations if I'm not mistaken. So then you have to fall back to distributing the Julia "binary" with a full Julia runtime, which is pretty heavy. There are some packages, such as PySR [3], which do this. It seems pretty usable for research I'd say, but difficult to put in production. There is some word going around though that there is an even better static compiler in the making, but as long as that one is not publicly available I'd say that Julia cannot easily be called from other languages. [1]: https://github.com/tshort/StaticCompiler.jl https://github.com/tshort/StaticCompiler.jl [2]: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl [3]: https://github.com/MilesCranmer/PySR https://github.com/MilesCranmer/PySR
- eigenspace 3y ago
  I think you misunderstood me. Nothing about calling julia from other languages requires StaticCompiler.jl, that's just an experimental proof of concept for small, standalone binary generation with some heavy limitations. Calling julia from another language does not in general require AOT compilation, though it does help to make things more portable and self contained. If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language. > PackageCompiler.jl [2] has the same limitations if I'm not mistaken You are mistaken. PackageCompiler works by basically bundling an entire julia system image with your program and the whole compiler, runtime, and stdlib compiled into one bundle together. If a piece of code works in the REPL it'll work from PackageCompiler
  - huijzer 3y ago
    > If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language. Okay but how do you distribute the program that depends on Julia to clients? You then also need to ensure that they have the right Julia available on their system. That's much heavier than dynamic binaries (which rely only on some shared libraries). My point was that it's not always "easy" to call Julia. That it's easy in HPC is a reasonable statement since you probably have full control over the system and it has loads of RAM available, but I wouldn't call it easy in general. > You are mistaken. I guess we're both right. I was talking about the "libraries" functionality [1]. [1]: https://julialang.github.io/PackageCompiler.jl/dev/libs.html https://julialang.github.io/PackageCompiler.jl/dev/libs.html
    - eigenspace 3y ago
      > Okay but how do you distribute the program that depends on Julia to clients? That's not the question that was asked though. The question the person asked as "Can you call Julia routines from other languages?" and the answer to that is "yes, you can easily do that". __________________________________________________ > I guess we're both right. I was talking about the "libraries" functionality I'm confused, what functionality exactly are you saying is missing from the libraries functionality? That very example you link to shows one using the julia runtime via printing. And yes, dynamic dispatch and dynamic code generation do work from libraries created by PackageCompiler.jl as well...

sertbdfgbnfgsd 3y ago

Going all in is precisely the point. The biggest selling point of Julia is that it's as easy to write as python if you don't care about performance, and if you do care about performance it's as fast as C++ if you can just follow some simple principles. This actually works in practice. For my day job I write the slow path exactly as I would python, and that's a really fast way to write code. If you want a slow language that can call fast code, you already have python.

jampekka 3y ago
Biggest selling point of Julia is that you're locked in to Julia if you use it? For me this is the biggest point for giving up on Julia. Julia has a lot going for it in the basic design and I was really excited about it in the beginning. But even after all these years the environment is sadly almost unusable if you want to do something saner than REPL/notebook. The TTFP is too damn high and I don't think it will ever be fixed.
- sertbdfgbnfgsd 3y ago
  > environment is sadly almost unusable if you want to do something saner than REPL/notebook I work in "one of those industries" where failure can easily cost millions in the space of minutes, and we use Julia almost exclusively. Unfortunately I can't give you details because of reasons, other than tell you you're very poorly informed. > you're locked in to Julia You're not "locked in" to Julia any more you're locked in to Python. Arguably less, as Julia can call C natively, Python can't. My point isn't that you can't interop, and it seems you're intentionally (?) equivocating on this point. Surely you didn't misinterpret what I wrote as "you can't iterop"? My point is that you can more benefits if you don't. Like I said, if you want a slow language that can call fast routines, why not just use Python.
  - jampekka 3y ago
    Python can call C natively. Also Python programs can be used as CLI tools interacting with practically any language. Julia programs in practice can't. But I know from experience that this debate will not go anywhere. It's is the same old refusal to see the problem: "you're using it wrong". Julia is dying and this is why it will die.