5 ms·

This is absolutely correct. The benchmarks they claim are misleading, and under fair conditions, Julia simply isn't the speed/efficiency solution they market t

This is absolutely correct. The benchmarks they claim are misleading, and under fair conditions, Julia simply isn't the speed/efficiency solution they market themselves to be. I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting.

ninjin 8y ago
> I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting. Isn’t Google’s “Swift for TensorFlow” move in direct opposition to this statement? I think you are right that it gets you 95% of the way, but that the final 5% of performance and portability simply will not be there. https://github.com/tensorflow/swift/blob/master/docs/DesignOverview.md https://github.com/tensorflow/swift/blob/master/docs/DesignO...
- ChrisRackauckas 8y ago
  It's not just 5%. If you have any API which takes a higher order function, for example a differential equation or optimization package, then even if your code is compiled and the user's input is compiled, you still have to hit Python in the middle, and that context switch can be the most costly part of an optimized code, making Numba+SciPy about 10x slower than it should be. So yeah, Python + compiled code is not a viable solution in all cases, or in fact what seems to be most of scientific computing (but not data science where things tend to not include function input). http://juliadiffeq.org/2018/04/30/Jupyter.html http://juliadiffeq.org/2018/04/30/Jupyter.html
  - stochastic_monk 8y ago
    There can be a significant hit in performance due to using Python. I generally prefer developing pure C++, but being able to test/prototype in Python is great. pybind11 makes interoperability easy and efficient.
    - ScottPJones 8y ago
      What about using Cxx.jl? You can have a nice little REPL for your C++ code. I haven't seen pybind11, does it give you that ability?
      - stochastic_monk 8y ago
        No, but cling [1] does. I might be interested in trying cxx.jl, thank you for the pointer. [1]: https://github.com/vgvassilev/cling https://github.com/vgvassilev/cling
j605 8y ago
Hello world program measures startup time which is not what julia is optimized for right now. I''ve heard that they are looking into it but it is not as important as startup time in python where we have a lot of python scripts as binaries which are executed repeatedly or interactively(like "hg status").
- ChrisRackauckas 8y ago
  Julia v0.7/v1.0 comes with an interpreter which doesn't have any JIT startup time (because it's an actual interpreter). It's slow because it cannot use precompiled code yet, which is why it's not documented, but it's on the "more to come" list.
  - sivakon 8y ago
    Do you have a link to share? Sounds interesting.
    - ChrisRackauckas 8y ago
      It's one of the things being developed on the down low right now, but you can find PRs like: https://github.com/JuliaLang/julia/pull/24473 https://github.com/JuliaLang/julia/pull/24473 which are making this: https://github.com/JuliaLang/julia/blob/master/src/interpreter.c https://github.com/JuliaLang/julia/blob/master/src/interpret... a compilation-free version of Julia. At the same people there is discussion about how to cache native compilation results after precompilation, so that way packages can store all of their compiled code and users can directly use it without compilation through the interpreter. This second part isn't done yet, but it'll be interesting to see what happens when it has a truly dynamic + precompiled form.
eigenspace 8y ago
That is absolutley false. Look closely at the 'benchmarks' shown in the above article and tell me if they are at all compelling. They measure Julia's startup time and compilation time. IF you wanted to compare Julia to C there, you'd want to include the time it takes to compile your hello_world script. Julia's performance optimizations have to far been mostly focused on intensive numerical computations where the startup time and compilation time are merely a constant overhead that are irrelevant to high performance numerics.
- stochastic_monk 8y ago
  No, it's not. My comment is unrelated to startup and the issue in this article but more generally about Julia as a language of failed promises. The biggest issue with the standard Julia benchmarks is that they're not using what would be optimal C code to compare against. In addition, if you look at the Julia issues on GitHub, you'll find hundreds of performance regressions where code performs more than 10 times as slow as what they expected/claimed at one point. It's not reliably fast, even when written by Julia experts/developers.
  - eigenspace 8y ago
    > The biggest issue with the standard Julia benchmarks is that they're not using what would be optimal C code to compare against. Yes, the C code benchmarks are not optimal. Neither are the julia benchmarks or any of the other languages for the matter. The julia devs made a hard decision with those benchmarks and decided that if they allowed arbitrary optimization, the benchmarks would become more a measure of who spent the most time and knowhow writing the benchmarks for ______ language. Instead, they tried to keep the code for all the languages at a reasonable level and avoided super specialized magic. That may make some uncomfortable, but keep in mind that the julia code used in the benchmarks also has a lot of room for improvement. Some julia devs are absolute wizards and getting performance out of julia code if you let them go crazy. > In addition, if you look at the Julia issues on GitHub, you'll find hundreds of performance regressions where code performs more than 10 times as slow as what they expected/claimed at one point. It's not reliably fast, even when written by Julia experts/developers. Julia 0.7 (which is still in its alpha build by the way) included a ground-up replacement of julias iteration protocol and a reworking of a ton of code optimization routines. If you think you or anyone else in the world could take a codebase as large as julia's and replace fundamental parts of it without seeing performance regressions anywhere you're delusional. There are a number of performance regressions, some mysterious and some not mysterious and they will all be worked on because the julia devs take regressions very seriously. It won't be instantaneous, but I do not doubt that the bulk of them will be eliminated promptly. There are also orders of magnitude more performance improvements in 0.7-alpha than there are regressions, they just aren't filed as issues. Some of these performance improvements, especially with broadcasting, are state of the art and not seen in other languages. Calling julia a "language of failed promises" because the alpha build of a pre 1.0 version has some performance regressions is sensationalist and disingenuous.
Certhas 8y ago
That's a sweeping statement without any backing argument. Others have debunked the blog post enough (even if you seem to want to cling on to it). I know several efforts that are transitioning Python + C Libraries to Julia. It's simply much much nicer and simpler to write fast code in Julia than in the Python/C paradigm.

ninjin 8y ago

> I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting. Isn’t Google’s “Swift for TensorFlow” move in direct opposition to this statement? I think you are right that it gets you 95% of the way, but that the final 5% of performance and portability simply will not be there. https://github.com/tensorflow/swift/blob/master/docs/DesignOverview.md https://github.com/tensorflow/swift/blob/master/docs/DesignO...

ChrisRackauckas 8y ago
It's not just 5%. If you have any API which takes a higher order function, for example a differential equation or optimization package, then even if your code is compiled and the user's input is compiled, you still have to hit Python in the middle, and that context switch can be the most costly part of an optimized code, making Numba+SciPy about 10x slower than it should be. So yeah, Python + compiled code is not a viable solution in all cases, or in fact what seems to be most of scientific computing (but not data science where things tend to not include function input). http://juliadiffeq.org/2018/04/30/Jupyter.html http://juliadiffeq.org/2018/04/30/Jupyter.html
- stochastic_monk 8y ago
  There can be a significant hit in performance due to using Python. I generally prefer developing pure C++, but being able to test/prototype in Python is great. pybind11 makes interoperability easy and efficient.
  - ScottPJones 8y ago
    What about using Cxx.jl? You can have a nice little REPL for your C++ code. I haven't seen pybind11, does it give you that ability?
    - stochastic_monk 8y ago
      No, but cling [1] does. I might be interested in trying cxx.jl, thank you for the pointer. [1]: https://github.com/vgvassilev/cling https://github.com/vgvassilev/cling

j605 8y ago

Hello world program measures startup time which is not what julia is optimized for right now. I''ve heard that they are looking into it but it is not as important as startup time in python where we have a lot of python scripts as binaries which are executed repeatedly or interactively(like "hg status").

ChrisRackauckas 8y ago
Julia v0.7/v1.0 comes with an interpreter which doesn't have any JIT startup time (because it's an actual interpreter). It's slow because it cannot use precompiled code yet, which is why it's not documented, but it's on the "more to come" list.
- sivakon 8y ago
  Do you have a link to share? Sounds interesting.
  - ChrisRackauckas 8y ago
    It's one of the things being developed on the down low right now, but you can find PRs like: https://github.com/JuliaLang/julia/pull/24473 https://github.com/JuliaLang/julia/pull/24473 which are making this: https://github.com/JuliaLang/julia/blob/master/src/interpreter.c https://github.com/JuliaLang/julia/blob/master/src/interpret... a compilation-free version of Julia. At the same people there is discussion about how to cache native compilation results after precompilation, so that way packages can store all of their compiled code and users can directly use it without compilation through the interpreter. This second part isn't done yet, but it'll be interesting to see what happens when it has a truly dynamic + precompiled form.

eigenspace 8y ago

That is absolutley false. Look closely at the 'benchmarks' shown in the above article and tell me if they are at all compelling. They measure Julia's startup time and compilation time. IF you wanted to compare Julia to C there, you'd want to include the time it takes to compile your hello_world script. Julia's performance optimizations have to far been mostly focused on intensive numerical computations where the startup time and compilation time are merely a constant overhead that are irrelevant to high performance numerics.

stochastic_monk 8y ago
No, it's not. My comment is unrelated to startup and the issue in this article but more generally about Julia as a language of failed promises. The biggest issue with the standard Julia benchmarks is that they're not using what would be optimal C code to compare against. In addition, if you look at the Julia issues on GitHub, you'll find hundreds of performance regressions where code performs more than 10 times as slow as what they expected/claimed at one point. It's not reliably fast, even when written by Julia experts/developers.
- eigenspace 8y ago
  > The biggest issue with the standard Julia benchmarks is that they're not using what would be optimal C code to compare against. Yes, the C code benchmarks are not optimal. Neither are the julia benchmarks or any of the other languages for the matter. The julia devs made a hard decision with those benchmarks and decided that if they allowed arbitrary optimization, the benchmarks would become more a measure of who spent the most time and knowhow writing the benchmarks for ______ language. Instead, they tried to keep the code for all the languages at a reasonable level and avoided super specialized magic. That may make some uncomfortable, but keep in mind that the julia code used in the benchmarks also has a lot of room for improvement. Some julia devs are absolute wizards and getting performance out of julia code if you let them go crazy. > In addition, if you look at the Julia issues on GitHub, you'll find hundreds of performance regressions where code performs more than 10 times as slow as what they expected/claimed at one point. It's not reliably fast, even when written by Julia experts/developers. Julia 0.7 (which is still in its alpha build by the way) included a ground-up replacement of julias iteration protocol and a reworking of a ton of code optimization routines. If you think you or anyone else in the world could take a codebase as large as julia's and replace fundamental parts of it without seeing performance regressions anywhere you're delusional. There are a number of performance regressions, some mysterious and some not mysterious and they will all be worked on because the julia devs take regressions very seriously. It won't be instantaneous, but I do not doubt that the bulk of them will be eliminated promptly. There are also orders of magnitude more performance improvements in 0.7-alpha than there are regressions, they just aren't filed as issues. Some of these performance improvements, especially with broadcasting, are state of the art and not seen in other languages. Calling julia a "language of failed promises" because the alpha build of a pre 1.0 version has some performance regressions is sensationalist and disingenuous.

Certhas 8y ago

That's a sweeping statement without any backing argument. Others have debunked the blog post enough (even if you seem to want to cling on to it). I know several efforts that are transitioning Python + C Libraries to Julia. It's simply much much nicer and simpler to write fast code in Julia than in the Python/C paradigm.