6 ms·

Honest question, just how fast is Julia anyway? Because performance is typically the main reason anyone would want to use it over alternatives. A lot of people

by jb1991 8y ago

Honest question, just how fast is Julia anyway? Because performance is typically the main reason anyone would want to use it over alternatives. A lot of people justify Julia because, as noted in other comments, it is both high-level yet claims to be nearly as fast as C: > "Julia is the fastest modern open-source language for data science, machine learning and scientific computing...with the speed, capacity and performance of C, C++..." [4] That's a bold claim! And there is much evidence to the contrary. I often find that the language falls into a similar trap of many other languages where the creators are evangelists not quite sharing the whole picture. Writing fast Julia code is not always a pleasant experience as you often need to fight the easier idioms. It is marketed as fast, but really, how fast is it? [0] http://www.zverovich.net/2016/05/13/giving-up-on-julia.html http://www.zverovich.net/2016/05/13/giving-up-on-julia.html "What’s disappointing is the striking difference between the claimed performance and the observed one. For example, a trivial hello world program in Julia runs ~27x slower than Python’s version and ~187x slower than the one in C." [1] https://www.ibm.com/developerworks/community/blogs/jfp/entry/A_Comparison_Of_C_Julia_Python_Numba_Cython_Scipy_and_BLAS_on_LU_Factorization?lang=en https://www.ibm.com/developerworks/community/blogs/jfp/entry... "We can code in Julia in a way similar to Python's code. However, that code is slower than it should. One way to speed Julia is to take into account the Fortran ordering it uses by looping on j before looping on i in the second loop. We also add decorators to speed up the code." [2] https://www.codementor.io/zhuojiadai/julia-vs-r-vs-python-string-sort-performance-an-unfinished-journey-to-optimizing-julia-s-performance-f57tf9f8s https://www.codementor.io/zhuojiadai/julia-vs-r-vs-python-st... "...comparing R's sorting speeds to Julia's is not the complete story, even though on the surface R appears faster, and from a users' perspective, (once the data is loaded) R is still the king of speed." Stackoverflow has quite a few user posts [3] from folks trying to get their code to perform at the advertised speed vs popular alternative languages. It's not so easy. [3] https://stackoverflow.com/questions/20613817/julia-julia-lang-performance-compared-to-fortran-and-python https://stackoverflow.com/questions/20613817/julia-julia-lan... [4] https://juliacomputing.com/ https://juliacomputing.com/

montalbano 8y ago
Regarding [0], startup time is a commonly acknowledged issue which affects the 'hello world' benchmark. (Can you even call it a benchmark? Maybe if startup time is what you're comparing then it makes sense, other wise I don't agree.) I used to call Fortran code from Python for a particularly complex function (Mittag-Leffler function) from Python, I then rewrote everything in Julia and it runs approx 10-20x faster. So there you go, I'm a primary source. Edit: just for clarity: The Mittag-Leffler function itself runs just ever so slightly faster in Julia than Fortran-called-from-Python (due to the probably inefficient way I was sending and receiving the data to/from a Python subprocess, which itself is an example of the difficulty of combining two languages). It is the program as a whole which runs 10-20x faster.
stochastic_monk 8y ago
This is absolutely correct. The benchmarks they claim are misleading, and under fair conditions, Julia simply isn't the speed/efficiency solution they market themselves to be. I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting.
- ninjin 8y ago
  > I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting. Isn’t Google’s “Swift for TensorFlow” move in direct opposition to this statement? I think you are right that it gets you 95% of the way, but that the final 5% of performance and portability simply will not be there. https://github.com/tensorflow/swift/blob/master/docs/DesignOverview.md https://github.com/tensorflow/swift/blob/master/docs/DesignO...
  - ChrisRackauckas 8y ago
    It's not just 5%. If you have any API which takes a higher order function, for example a differential equation or optimization package, then even if your code is compiled and the user's input is compiled, you still have to hit Python in the middle, and that context switch can be the most costly part of an optimized code, making Numba+SciPy about 10x slower than it should be. So yeah, Python + compiled code is not a viable solution in all cases, or in fact what seems to be most of scientific computing (but not data science where things tend to not include function input). http://juliadiffeq.org/2018/04/30/Jupyter.html http://juliadiffeq.org/2018/04/30/Jupyter.html
    - stochastic_monk 8y ago
      There can be a significant hit in performance due to using Python. I generally prefer developing pure C++, but being able to test/prototype in Python is great. pybind11 makes interoperability easy and efficient.
stabbles 8y ago
Concerning the "Giving up on Julia" article, it's very dated and dubious. Please read [1]. About the second quote: this is not about Julia being slower than Python; it's about avoiding cache misses. Obviously you should exploit contiguity of the data in memory. Secondly, the 'decorators' like "@inbounds" just ensure that Julia does not redundantly checks bounds, otherwise the inner loop has branches and cannot exploit SIMD. [1] https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-for-the-giving-up-on-julia-blog-post/ https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...
- hellofunk 8y ago
  That's a fair rebuttal. But if the original author struggled to get fast Julia working, maybe it's a similar experience to the very many users of Stackoverflow who ask how to get their code to run at marketed speed. It appears to not be obvious or idiomatic in Julia to really get the speed it advertises.
  - stabbles 8y ago
    Fortunately there is documentation about performance [1], which is about everything from type stability to SIMD. [1] https://docs.julialang.org/en/latest/manual/performance-tips/ https://docs.julialang.org/en/latest/manual/performance-tips...
  - improbable22 8y ago
    Last week's discussion on matrix multiply: https://news.ycombinator.com/item?id=17164737 https://news.ycombinator.com/item?id=17164737 I guess the claim is that you could write all that in Julia, and hope to be competitive, while you could never do this in (pure) Python. It would still be a lot of work, and would require you to think about the cache, there's no way around that. The selling point here (it seems to me) is that you can easily doodle up the most naive version of whatever algorithm you're thinking about, and then start optimising to the degree needed, where needed, without having to start over in C or something.
    - hellofunk 8y ago
oxinabox 8y ago
I actually could hardly care less about julia's speed. It is fast enough. What matters to me is that it is actually really nice to write. The syntax is elegant, the community is excellent, and the core of the language's semantics being multiple dispatch is a game changer.
ziotom78 8y ago
My experience tells that computationally intensive codes are roughly 10% slower that their C++/Fortran counterparts. So fare we are implementing a data analysis pipeline for a cosmological telescope (LSPE/STRIP), and we are migrating a number of codes my team developed for the Planck/LFI instrument. As data formats and a few details of the instrument changed, we decided to rewrite everything from scratch instead of manually fix and recheck every line of code, and we chose Julia 0.6. The 10% figure comes from a comparison of the performance of the codes whose logic has been left virtually unchanged, of course. What is awesome is the ability to run codes within Jupyter (like Python), and the ability to run "for" loops on large arrays fast and with little memory consumption (unlike Python). The 10% increase in computational time is fully absorbed by the reduced development/validation time!
- Certhas 8y ago
  This is mirrored in my experience with dynamical simulations and mathematical optimization work.
- ChrisRackauckas 8y ago
  This 10% is due to some aliasing optimizations that haven't been added to the compiler yet, at least for the differential equations code we've looked at. But now that the compiler guys have some more free time...
Derbasti 8y ago
This has been my experience as well: Python-style Julia code often does not run faster than in Python. Only of you rewrite it in C-style do you get C-like performance. This is not necessarily a bad thing. Just an observation from me translating my Python code to Julia.

by jb1991 8y ago

montalbano 8y ago

Regarding [0], startup time is a commonly acknowledged issue which affects the 'hello world' benchmark. (Can you even call it a benchmark? Maybe if startup time is what you're comparing then it makes sense, other wise I don't agree.) I used to call Fortran code from Python for a particularly complex function (Mittag-Leffler function) from Python, I then rewrote everything in Julia and it runs approx 10-20x faster. So there you go, I'm a primary source. Edit: just for clarity: The Mittag-Leffler function itself runs just ever so slightly faster in Julia than Fortran-called-from-Python (due to the probably inefficient way I was sending and receiving the data to/from a Python subprocess, which itself is an example of the difficulty of combining two languages). It is the program as a whole which runs 10-20x faster.

stochastic_monk 8y ago

This is absolutely correct. The benchmarks they claim are misleading, and under fair conditions, Julia simply isn't the speed/efficiency solution they market themselves to be. I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting.

ninjin 8y ago
> I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting. Isn’t Google’s “Swift for TensorFlow” move in direct opposition to this statement? I think you are right that it gets you 95% of the way, but that the final 5% of performance and portability simply will not be there. https://github.com/tensorflow/swift/blob/master/docs/DesignOverview.md https://github.com/tensorflow/swift/blob/master/docs/DesignO...
- ChrisRackauckas 8y ago
  It's not just 5%. If you have any API which takes a higher order function, for example a differential equation or optimization package, then even if your code is compiled and the user's input is compiled, you still have to hit Python in the middle, and that context switch can be the most costly part of an optimized code, making Numba+SciPy about 10x slower than it should be. So yeah, Python + compiled code is not a viable solution in all cases, or in fact what seems to be most of scientific computing (but not data science where things tend to not include function input). http://juliadiffeq.org/2018/04/30/Jupyter.html http://juliadiffeq.org/2018/04/30/Jupyter.html
  - stochastic_monk 8y ago
    There can be a significant hit in performance due to using Python. I generally prefer developing pure C++, but being able to test/prototype in Python is great. pybind11 makes interoperability easy and efficient.

stabbles 8y ago

Concerning the "Giving up on Julia" article, it's very dated and dubious. Please read [1]. About the second quote: this is not about Julia being slower than Python; it's about avoiding cache misses. Obviously you should exploit contiguity of the data in memory. Secondly, the 'decorators' like "@inbounds" just ensure that Julia does not redundantly checks bounds, otherwise the inner loop has branches and cannot exploit SIMD. [1] https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-for-the-giving-up-on-julia-blog-post/ https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...

hellofunk 8y ago
That's a fair rebuttal. But if the original author struggled to get fast Julia working, maybe it's a similar experience to the very many users of Stackoverflow who ask how to get their code to run at marketed speed. It appears to not be obvious or idiomatic in Julia to really get the speed it advertises.
- stabbles 8y ago
  Fortunately there is documentation about performance [1], which is about everything from type stability to SIMD. [1] https://docs.julialang.org/en/latest/manual/performance-tips/ https://docs.julialang.org/en/latest/manual/performance-tips...
- improbable22 8y ago
  Last week's discussion on matrix multiply: https://news.ycombinator.com/item?id=17164737 https://news.ycombinator.com/item?id=17164737 I guess the claim is that you could write all that in Julia, and hope to be competitive, while you could never do this in (pure) Python. It would still be a lot of work, and would require you to think about the cache, there's no way around that. The selling point here (it seems to me) is that you can easily doodle up the most naive version of whatever algorithm you're thinking about, and then start optimising to the degree needed, where needed, without having to start over in C or something.
  - hellofunk 8y ago

oxinabox 8y ago

I actually could hardly care less about julia's speed. It is fast enough. What matters to me is that it is actually really nice to write. The syntax is elegant, the community is excellent, and the core of the language's semantics being multiple dispatch is a game changer.

ziotom78 8y ago

My experience tells that computationally intensive codes are roughly 10% slower that their C++/Fortran counterparts. So fare we are implementing a data analysis pipeline for a cosmological telescope (LSPE/STRIP), and we are migrating a number of codes my team developed for the Planck/LFI instrument. As data formats and a few details of the instrument changed, we decided to rewrite everything from scratch instead of manually fix and recheck every line of code, and we chose Julia 0.6. The 10% figure comes from a comparison of the performance of the codes whose logic has been left virtually unchanged, of course. What is awesome is the ability to run codes within Jupyter (like Python), and the ability to run "for" loops on large arrays fast and with little memory consumption (unlike Python). The 10% increase in computational time is fully absorbed by the reduced development/validation time!

Certhas 8y ago
This is mirrored in my experience with dynamical simulations and mathematical optimization work.
ChrisRackauckas 8y ago
This 10% is due to some aliasing optimizations that haven't been added to the compiler yet, at least for the differential equations code we've looked at. But now that the compiler guys have some more free time...

Derbasti 8y ago

This has been my experience as well: Python-style Julia code often does not run faster than in Python. Only of you rewrite it in C-style do you get C-like performance. This is not necessarily a bad thing. Just an observation from me translating my Python code to Julia.