13 ms·

Julia: Dynamism and Performance Reconciled by Design (2018)

94 points by bshanks 6y ago

jpeloquin 6y ago
> Specificity is required to disambiguate between two or more methods which are all supertypes of the argument type. … The specificity rules are defined by the implementation and lack a formal semantics. The lack of formal rules governing which method will be selected for a given set of arguments has made me somewhat reluctant to use Julia for large projects. Since everything in the language works off of multiple dispatch, it's a little disconcerting to have it governed by vague rules. However, the manual [0] says that when two methods of equal specificity are present, "Julia raises a MethodError rather than arbitrarily picking a method". The manual doesn't discuss the 5 special cases in the linked PDF, though, as far as I can tell. So did the dispatch / specificity rules get formalized at some point? Or is concern about having formal rules governing dispatch unwarranted? [0] https://docs.julialang.org/en/v1/manual/methods/ https://docs.julialang.org/en/v1/manual/methods/
- KenoFischer 6y ago
  The subtyping system is fairly complicated, but in practice users only need to really think about situations where the LHS is a concrete type, since that's primarily what determines runtime semantics (not to say that there aren't situations where specificity really matters, but it's quite rare). That simplifies the mental model considerably and our users tend to find it relatively intuitive. For a formal discussion, the group at Northeastern has been doing great work in phrasing some of the Julia internals in more traditional PL language, including a formal subtyping model. That paper is at http://janvitek.org/pubs/oopsla18a.pdf http://janvitek.org/pubs/oopsla18a.pdf
  - snicker7 6y ago
    Love JuliaLang. But it definitely comes across as a language defined by its implementation. It is poorly specified relative to most popular languages. Appreciate the work by the paper's authors, though.
    - KenoFischer 6y ago
      I think that's observation is fair, but I would also argue it is neither unusual (e.g. Python doesn't have one and it's probably one of the most popular language on the planet at the moment) nor a bad thing. In programming language design, you need to implement things and try out how they feel to use in practice. Programming language design is mostly about interfacing with humans, not computers. Something that has a beatiful proof on paper, but feels clumsy to use is bad programming language design. Eventually one of the efforts to make Julia an ISO standard will win and we'll be doing language design by committee and have a formal specification, but until then we'll try to build the best language we can ;).
- deleted 6y ago
  [deleted]
aborsy 6y ago
The performance comparison in Fig. 7 is quite limited. Python with Numpy and JIT, as well as Matlab, R and FORTRAN could also be included with suitable tasks. The code should be optimized in each case.
- vmchale 6y ago
  I'm curious about J.
- deleted 6y ago
  [deleted]
- huitzitziltzin 6y ago
  Here's a broader set of benchmarks, including to R, Matlab and Fortran: https://julialang.org/benchmarks/ https://julialang.org/benchmarks/ AFAIK these have been optimized for the other languages, but your opinion may differ if you look at the implementations. Code for these is available here: https://github.com/JuliaLang/Microbenchmarks https://github.com/JuliaLang/Microbenchmarks
  - TheRealKing 6y ago
    The Fortran implementation that I see on GitHub, is in no way the most optimized implementation that one could have, or would do in any serious heavy numerical computation. Yet it is amazing to see how well Fortran performs given all the lack of optimizations in the benchmark code. Also, performance highly depends on the compiler and compiler optimization flags and there is no mention of the optimization flags used for the Fortran compiler. and remember optimizations do not end with "-O3" flag in Fortran (and C). A Julia benchmark written by Julia developers showing Julia outperforms any other language does not seem to be the best way of assessing languages' performance. Did you have experts in each of the other languages to implement the respective codes?
    - eigenspace 6y ago
      From what I understand, the point of these benchmarks was to write relatively straightforward idiomatic code in all the laguages. The Julia devs were well aware that their knowledge of julia would make benchmark hacking straightforward, so they purposefully did not take many optimization opportunities available to them. They did the same with the Fotran and C benchmarks. These things are all pretty subjective though when it comes to 'what is a fair benchmarking methodology'? Another valid approach would be to have experts in each language try to write as optimal an implementation as possible, but this would quickly become a bit of an arms race.
quixoticelixer- 6y ago
One thing that does annoy me about Julia is how much they focus on performance. I do appreciate the speed but the reason why I use Julia isn't because it's fast, it's because its (in my opinion) a nicer language than R or Python/Numpy and I would like to see more effort go into that.
- eigenspace 6y ago
  I do agree, but one thing I’ll say is that the performance is important because it means that you essentially need to call C code for performance. Having ll the code in a project and dependencies be in one language has some really nice advantages! Of course we do call things like C or Python whenever there’s a good implementation of something we want, and because of that we have some excellent FFI capabilities. But yeah, all that said, I’d use Julia even if it was slow. It’s an amazing, powerful and fun language that has taught me a lot and made me enjoy programming.
  - systemvoltage 6y ago
    My experience has been completely the opposite. Julia has been frustrating to work with. Most of the time I’m just waiting for precompilation of the packages. It’s unbelievable that the Julia core devs still haven’t addressed it. It is not fun by any means. Not even tolerable.
    - jey 6y ago
      Try to use the REPL more instead of re-running your program from scratch. I just do `include("blah.jl")` then re-run my testcase.
      - jjoonathan 6y ago
        Ah, the statefulness foot-gun. It'll solve all your problems... if your problem is having feet! Ok, that's a little mean, there's nothing quite as convenient and simple as a REPL or notebook for ad-hoc caching, but sometimes you need the opposite and the language really needs to support that.
badjeans 6y ago
It would be interesting to compare it to Nim, which is also a very convenient and very performant language.
- quixoticelixer- 6y ago
  You could probably do some tests yourself. Also as far as scientific computing goes Nim lacks a lot of libraries.
  - j-james 6y ago
    There's a tracking issue for this: https://github.com/nim-lang/needed-libraries/issues/77 https://github.com/nim-lang/needed-libraries/issues/77
- sk2020 6y ago
  https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks There are a few tests which have Julia and Nim implementations. It would be nice if the benchmarks game maintainer would include some more languages, like Nim, Crystal, and Zig. This doesn’t have Nim, but is another interesting one. https://modelingguru.nasa.gov/docs/DOC-2783 https://modelingguru.nasa.gov/docs/DOC-2783
  - deleted 6y ago
    [deleted]
  - dnautics 6y ago
    the benchmarks game maintainer is incredibly unprofessional, and looks bad for the debian team.
    - igouy 6y ago
      fyi debian salsa provides a project hosting service — that's the extent of debian involvement with the benchmarks game project. (Name-calling isn't a good look.)
cs702 6y ago
I have dabbled only a bit with Julia, but I have used Python extensively, including for deep learning and for more classic machine learning applications, and my conclusion is that Julia is a superior alternative to Python in all ways except one: ecosystem. At present, the Python ecosystem -- packages, frameworks, open-source repositories, tutorials, online discussions, communities, documentations, etc. -- is a significant advantage in favor of Python.
- quixoticelixer- 6y ago
  To be fair that's an extremely high bar, one of the best things about Python is it's ecosystem and few other languages can match it.
- dklend122 6y ago
  There are many areas in which Julia's ecosystem is far ahead of python's, today. Standard ML will catch up eventually
- dunefox 6y ago
  > my conclusion is that Julia is a superior alternative to Python in all ways except one: ecosystem. Fully agree, but with PyCall you can just use any Python package in Julia: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl I do that regularly when I need something from Python and it works perfectly. So, Julias ecosystem is really a superset of Pythons when it comes to libraries.
kruxigt 6y ago
For me it's really interesting to read all the complaints about precompilation times etc. I'm definitely not saying the complaints are false, just that there are also some of us out there where they matter close to 0 (at current level). I use Julia from the REPL, running mostly self constructed algorithms. Only fairly basic modules are used. My main project is a backtesting framework for algorithmic stock trading. A typical run took more than a month of runtime on my computer when the code was in optimized Numpy and now takes less than 1 minute in optimized (but likely far from optimal) Julia code. The only way to come at all close to the speed of Julia with Python would probably be something like Numba, which I figured would be more immature than Julia. Python is only usable for heavy computing on large data when the glue can be in the outermost loop so to speak.
- arthurcolle 6y ago
  What kind of architecture are you using for your backtesting framework? I have been collecting data to attempt to build something similar to that with close to around 2.3TB of tick data stretching around 20 years across 75 major pairs (indices, fx, major commods, etc) and I am always overwhelmed about design of systems to process large datasets like this.
  - Reelin 6y ago
    I realize it's a bit off topic, but I just have to ask - where did you get so much ticker data?
    - arthurcolle 6y ago
      It's been a work in progress for quite a while. Involved writing many different scrapers and other data ingestion systems that systematically downloaded everything I could get my hands on. I'm building an API to eventually sell some of the information, so I'd prefer not to say specific sources.
  - kruxigt 6y ago
    I'm not sure what you mean with architecture? The computers I currently run on are ordinary Macs of the latest generations. (Before Julia came and saved me I ran on Google cloud) That is way more data than I am currently using! I'm currently a hobo using data from Yahoo finance. :) That sounds interesting! Could you contact me at kruxigt at gmail.com and tell me more?

94 points by bshanks 6y ago

> Specificity is required to disambiguate between two or more methods which are all supertypes of the argument type. … The specificity rules are defined by the implementation and lack a formal semantics. The lack of formal rules governing which method will be selected for a given set of arguments has made me somewhat reluctant to use Julia for large projects. Since everything in the language works off of multiple dispatch, it's a little disconcerting to have it governed by vague rules. However, the manual [0] says that when two methods of equal specificity are present, "Julia raises a MethodError rather than arbitrarily picking a method". The manual doesn't discuss the 5 special cases in the linked PDF, though, as far as I can tell. So did the dispatch / specificity rules get formalized at some point? Or is concern about having formal rules governing dispatch unwarranted? [0] https://docs.julialang.org/en/v1/manual/methods/ https://docs.julialang.org/en/v1/manual/methods/

KenoFischer 6y ago
The subtyping system is fairly complicated, but in practice users only need to really think about situations where the LHS is a concrete type, since that's primarily what determines runtime semantics (not to say that there aren't situations where specificity really matters, but it's quite rare). That simplifies the mental model considerably and our users tend to find it relatively intuitive. For a formal discussion, the group at Northeastern has been doing great work in phrasing some of the Julia internals in more traditional PL language, including a formal subtyping model. That paper is at http://janvitek.org/pubs/oopsla18a.pdf http://janvitek.org/pubs/oopsla18a.pdf
- snicker7 6y ago
  Love JuliaLang. But it definitely comes across as a language defined by its implementation. It is poorly specified relative to most popular languages. Appreciate the work by the paper's authors, though.
  - KenoFischer 6y ago
    I think that's observation is fair, but I would also argue it is neither unusual (e.g. Python doesn't have one and it's probably one of the most popular language on the planet at the moment) nor a bad thing. In programming language design, you need to implement things and try out how they feel to use in practice. Programming language design is mostly about interfacing with humans, not computers. Something that has a beatiful proof on paper, but feels clumsy to use is bad programming language design. Eventually one of the efforts to make Julia an ISO standard will win and we'll be doing language design by committee and have a formal specification, but until then we'll try to build the best language we can ;).
deleted 6y ago
[deleted]

aborsy 6y ago

The performance comparison in Fig. 7 is quite limited. Python with Numpy and JIT, as well as Matlab, R and FORTRAN could also be included with suitable tasks. The code should be optimized in each case.

vmchale 6y ago
I'm curious about J.
deleted 6y ago
[deleted]
huitzitziltzin 6y ago
Here's a broader set of benchmarks, including to R, Matlab and Fortran: https://julialang.org/benchmarks/ https://julialang.org/benchmarks/ AFAIK these have been optimized for the other languages, but your opinion may differ if you look at the implementations. Code for these is available here: https://github.com/JuliaLang/Microbenchmarks https://github.com/JuliaLang/Microbenchmarks
- TheRealKing 6y ago
  The Fortran implementation that I see on GitHub, is in no way the most optimized implementation that one could have, or would do in any serious heavy numerical computation. Yet it is amazing to see how well Fortran performs given all the lack of optimizations in the benchmark code. Also, performance highly depends on the compiler and compiler optimization flags and there is no mention of the optimization flags used for the Fortran compiler. and remember optimizations do not end with "-O3" flag in Fortran (and C). A Julia benchmark written by Julia developers showing Julia outperforms any other language does not seem to be the best way of assessing languages' performance. Did you have experts in each of the other languages to implement the respective codes?
  - eigenspace 6y ago
    From what I understand, the point of these benchmarks was to write relatively straightforward idiomatic code in all the laguages. The Julia devs were well aware that their knowledge of julia would make benchmark hacking straightforward, so they purposefully did not take many optimization opportunities available to them. They did the same with the Fotran and C benchmarks. These things are all pretty subjective though when it comes to 'what is a fair benchmarking methodology'? Another valid approach would be to have experts in each language try to write as optimal an implementation as possible, but this would quickly become a bit of an arms race.

quixoticelixer- 6y ago

One thing that does annoy me about Julia is how much they focus on performance. I do appreciate the speed but the reason why I use Julia isn't because it's fast, it's because its (in my opinion) a nicer language than R or Python/Numpy and I would like to see more effort go into that.

eigenspace 6y ago
I do agree, but one thing I’ll say is that the performance is important because it means that you essentially need to call C code for performance. Having ll the code in a project and dependencies be in one language has some really nice advantages! Of course we do call things like C or Python whenever there’s a good implementation of something we want, and because of that we have some excellent FFI capabilities. But yeah, all that said, I’d use Julia even if it was slow. It’s an amazing, powerful and fun language that has taught me a lot and made me enjoy programming.
- systemvoltage 6y ago
  My experience has been completely the opposite. Julia has been frustrating to work with. Most of the time I’m just waiting for precompilation of the packages. It’s unbelievable that the Julia core devs still haven’t addressed it. It is not fun by any means. Not even tolerable.
  - jey 6y ago
    Try to use the REPL more instead of re-running your program from scratch. I just do `include("blah.jl")` then re-run my testcase.
    - jjoonathan 6y ago
      Ah, the statefulness foot-gun. It'll solve all your problems... if your problem is having feet! Ok, that's a little mean, there's nothing quite as convenient and simple as a REPL or notebook for ad-hoc caching, but sometimes you need the opposite and the language really needs to support that.

badjeans 6y ago

It would be interesting to compare it to Nim, which is also a very convenient and very performant language.

quixoticelixer- 6y ago
You could probably do some tests yourself. Also as far as scientific computing goes Nim lacks a lot of libraries.
- j-james 6y ago
  There's a tracking issue for this: https://github.com/nim-lang/needed-libraries/issues/77 https://github.com/nim-lang/needed-libraries/issues/77
sk2020 6y ago
https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks There are a few tests which have Julia and Nim implementations. It would be nice if the benchmarks game maintainer would include some more languages, like Nim, Crystal, and Zig. This doesn’t have Nim, but is another interesting one. https://modelingguru.nasa.gov/docs/DOC-2783 https://modelingguru.nasa.gov/docs/DOC-2783
- deleted 6y ago
  [deleted]
- dnautics 6y ago
  the benchmarks game maintainer is incredibly unprofessional, and looks bad for the debian team.
  - igouy 6y ago
    fyi debian salsa provides a project hosting service — that's the extent of debian involvement with the benchmarks game project. (Name-calling isn't a good look.)

cs702 6y ago

I have dabbled only a bit with Julia, but I have used Python extensively, including for deep learning and for more classic machine learning applications, and my conclusion is that Julia is a superior alternative to Python in all ways except one: ecosystem. At present, the Python ecosystem -- packages, frameworks, open-source repositories, tutorials, online discussions, communities, documentations, etc. -- is a significant advantage in favor of Python.

quixoticelixer- 6y ago
To be fair that's an extremely high bar, one of the best things about Python is it's ecosystem and few other languages can match it.
dklend122 6y ago
There are many areas in which Julia's ecosystem is far ahead of python's, today. Standard ML will catch up eventually
dunefox 6y ago
> my conclusion is that Julia is a superior alternative to Python in all ways except one: ecosystem. Fully agree, but with PyCall you can just use any Python package in Julia: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl I do that regularly when I need something from Python and it works perfectly. So, Julias ecosystem is really a superset of Pythons when it comes to libraries.

kruxigt 6y ago

For me it's really interesting to read all the complaints about precompilation times etc. I'm definitely not saying the complaints are false, just that there are also some of us out there where they matter close to 0 (at current level). I use Julia from the REPL, running mostly self constructed algorithms. Only fairly basic modules are used. My main project is a backtesting framework for algorithmic stock trading. A typical run took more than a month of runtime on my computer when the code was in optimized Numpy and now takes less than 1 minute in optimized (but likely far from optimal) Julia code. The only way to come at all close to the speed of Julia with Python would probably be something like Numba, which I figured would be more immature than Julia. Python is only usable for heavy computing on large data when the glue can be in the outermost loop so to speak.

arthurcolle 6y ago
What kind of architecture are you using for your backtesting framework? I have been collecting data to attempt to build something similar to that with close to around 2.3TB of tick data stretching around 20 years across 75 major pairs (indices, fx, major commods, etc) and I am always overwhelmed about design of systems to process large datasets like this.
- Reelin 6y ago
  I realize it's a bit off topic, but I just have to ask - where did you get so much ticker data?
  - arthurcolle 6y ago
    It's been a work in progress for quite a while. Involved writing many different scrapers and other data ingestion systems that systematically downloaded everything I could get my hands on. I'm building an API to eventually sell some of the information, so I'd prefer not to say specific sources.
- kruxigt 6y ago
  I'm not sure what you mean with architecture? The computers I currently run on are ordinary Macs of the latest generations. (Before Julia came and saved me I ran on Google cloud) That is way more data than I am currently using! I'm currently a hobo using data from Yahoo finance. :) That sounds interesting! Could you contact me at kruxigt at gmail.com and tell me more?