8 ms·

I don't think this is accurately representing Julia's aims. Of course the Julia team wrote the Python code in a way that makes it run slowly. But it looks like

by jordigh 11y ago

I don't think this is accurately representing Julia's aims. Of course the Julia team wrote the Python code in a way that makes it run slowly. But it looks like perfectly natural Python code! It's almost a literal translation of the Julia code. Julia's benchmark is even far worse with Octave, which they almost deliberately wrote in the worst way possible for Octave, with lots of loops and recursion. We have written some documentation for Octave in order to guide people towards writing faster Octave code: https://www.gnu.org/software/octave/doc/interpreter/Vectorization-and-Faster-Code-Execution.html https://www.gnu.org/software/octave/doc/interpreter/Vectoriz... But look at how much we have to explain, and look at all the hoops we have to jump through in Python and Octave in order to write faster code. Matlab used to have similar guides telling people don't write this, write that instead. What Matlab eventually did was look at what people were writing and making that fast. Julia did the same and made it even faster. This is a lesson from C and C++ compilers that seems to be taking a long time to trickle to other programming languages: as long as you're writing reasonable code, speeding up your code is your compiler's job, not yours. Your compiler usually knows better than you how to unroll loops, how to cache results, how to use multiple cores, how to elide unnecessary intermediate results, how to completely remove dead code. You should focus on writing easy to understand, maintainable, high-level code. That's why you're using a programming language and not machine code.

reikonomusha 11y ago
This is a really good comment and covers what a lot of people seem to miss in benchmarks. The fact I can install 2 different compilers (with varying degrees of support) and some hyperoptimized linear algebra libraries don't really tell me about how fast the system is in general when I need performance.
thebooktocome 11y ago
What lessons have you learned from developing Octave that you think Julia could benefit from?
- jordigh 11y ago
  That's a bit of a tough one. Julia is much newer software than Octave, and it also has a lot of knowledge already of what does and doesn't work. They're already well-staffed by programming language experts. I have learned a lot about how the Matlab language works, and also a lot about how Octave is put together. I don't have much to offer to Julia on technical terms. Octave's codebase started as pre-standard C++, and it's been a long and arduous job to very slowly modernise it. Julia has the advantage of starting modern. I think the biggest lesson I've learned is, don't underestimate Matlab. It may be old, it may be an ugly language, but it also is gigantic, very well-funded, and everywhere. Julia and Octave are approaching Matlab from different angles, Julia by providing a better language and Octave by providing a free drop-in replacement, but ultimately we are both trying to replace Matlab. Matlab users greatly value having a complete package with an editor, debugger, visualiser, profiler, and system modeller all in one. They don't generally see Simulink or the symbolic toolbox as being separate components, or indeed, as the completely separate programs that they really are. For the devout Matlab users, these conveniences greatly outweigh a better programming language. Going further, many Matlab users don't even consider their activity "programming" at all, and in recent years Matlab even writes the code for you if you just push a few buttons. We're seeing some academics slowly switching over to Julia. This is great. The Mathworks knows very well that captivating young, malleable minds in university is how you get loyal paying customers, namely, the future bosses of these newly-molded minds. Julia is doing a good job appealing to profs. I am optimistic that within five years or so, new generations of Julia-trained graduates will slowly spread the love of the new language.
  - thebooktocome 11y ago
    > Matlab users greatly value having a complete package with an editor, debugger, visualiser, profiler, and system modeller all in one. That's a very good point. Julia's debugger and profiler are both horrifyingly bad.
    - ViralBShah 11y ago
      The debugger is one of Julia's top priorities, and here's a debugger roadmap that Keno recently posted. https://groups.google.com/forum/#!topic/julia-dev/-LTsBVRv1dU https://groups.google.com/forum/#!topic/julia-dev/-LTsBVRv1d...
jacobolus 11y ago
> What Matlab eventually did was look at what people were writing and making that fast. Except in practice, to write fast Matlab code you need very deep understanding of Matlab’s internals and years of experience. Seemingly trivial patterns end up slowing your code down by multiple orders of magnitude, and there’s more “guess and check” involved when trying to write code that can be effectively JIT compiled by Matlab than careful reasoning. Even worse, the JIT and the profiler don’t get along, so it’s often impossible to get any insight into the reasons for JIT-related performance differences. In general, Matlab is an extremely unpleasant and frustrating environment compared to almost any other language I’ve worked with. The only thing Matlab has on Python/Numpy is a nice quantity of publicly available code for various technical functions. Most of this code is hacky academic prototype stuff, but that’s much better than nothing if you’re trying to follow someone’s algorithm written up in a paper. The Matlab GUI and tooling is a buggy and unpolished Java turd from the 90s which fits in poorly with any modern operating system.
- x0x0 11y ago
  And that's the pitch for julia. Oh, and mostly fast enough to write in julia the stuff that, for python, you'd write as C extensions, crucially enabling lots of optimizations that are terribly hard to do for python. And avoiding the whole pypy / C api plugin problem. It's still pretty immature, but promising. That said, matlab's linear algebra syntax is still better than python's.
- chm 11y ago
  Your first quip also applies to Mathematica. It's very easy to write quite inefficient code in Mathematica. One classic example of this is a newcomer not using N[] and evaluating every expression symbolically. Seems trivial, but it really isn't for someone new. I still could get speedups on linalg code two years into grad school.
  - TheOtherHobbes 11y ago
    It applies to most languages. In practice you can't rely on the compiler, because you don't know what the compiler is doing. You certainly can't assume it's going to make the smartest possible decision for all of your code. There's no standardisation for optimisations, and they're often a context-dependent trade-off anyway. The only way to write fast code is to learn the quirks of the tool chain and profile production binaries to find the bottlenecks.
jfpuget 11y ago
Thank you, great comment, it spurred a great discussion. Let me just clear one thing: I am not trying to represent Julia in any way. I wouldn't be legitimate for that at all. I tried to not write anything negative about the Julia language. Let me know if you think i did, in which case I'll modify my text. To your point about compilers in charge of speeding up code, I see Numba doing more and more. I hope it will cover all of Python soon.
- Lanzaa 11y ago
  I thought the article was good. It started with a basic premise, "Should we ditch Python and other languages in favor of Julia for technical computing?" Then the article shows several methods for speeding up python code to be faster than using julia. I was a little disappointed that pypy was not mentioned, but it is a good introduction to speeding up python code. I think the flaw in the article is when it switches to answering, "did the Julia team [write] Python benchmarks the best way for Python?" Then it rewrites the fib implementation to use a cache, which makes the comparison to the julia version completely ridiculous. It also does all sorts of optimization which clearly deviated from the spirit of the benchmark, naive python vs naive julia. I wish the conclusion had been written in a way that clearly answered the original premise, should we ditch python for julia? The article clearly showed that there are a lot of good ways to speed up python code. Looking for algorithmic complexity wins (like in the fib example), using cpython, using numba, and profiling all can be used to speed up python code to the level of naive julia code. Which leads to the conclusion, if all you want is faster code there is no need to ditch python for julia.
  - jfpuget 11y ago
    I agree that the first question is a very interesting one. I would be interested in the answer myself. I made it clear however that I was going to answer a different question: did the Julia team wrote Python benchmarks the best way for Python? That's all what the post is about: how to run Python code faster. I will need way more Julia experience to be able to even think about answering the first question. The only thing I am 100% sure is that these micro benchmarks do not help answering that question.
andrepd 11y ago
Exactly! The point is to compare standard, idiomatic and straightforward Julia code with standard, idiomatic and straightforward Python code, not with a jumble of workarounds, experimental JITs, and cumbersome syntax. Because if you are writing statically typed python, then you are not really writing python anymore, are you?
- JupiterMoon 11y ago
  Looping over lists/arrays is not idiomatic Python.
  - co_dh 11y ago
    +1
    - andrepd 11y ago
      You can use the up vote for that.
  - tavert 11y ago
    The point of that benchmark is looping over lists or arrays. The benchmark verifies the well-known fact that straight Python is a lousy choice for those situations when you absolutely need to do this. Dropping into C extensions or some-of-the-language Numba would be a different comparison, and show a different result.
    - JupiterMoon 11y ago
      Did they at least attempt list comprehensions rather than loops? EDIT Also in the scientific fields I've encountered not using the various libraries like numpy is non-idomatic. I personally think that python should absorb numpy into the standard library but effectively people I know treat it as if it were already there. You can argue that numpy etc are implemented in C. But is the Julia interpretor not implemented in C?
      - deleted 11y ago
co_dh 11y ago
I don't agree that speeding up your code is your compiler's job. If that is true, there is no point to learn data structure and algorithm.
- the_af 11y ago
  I don't think you and the parent post are talking about the same kind of speedups. Of course choosing the right algorithm and data structure is your job. How you tackle the problem is definitely your job, not the compiler's. But in general, unrolling loops, caching (some) results, and eliding redundant code rightfully is the compiler's job. That's a different kind of speedup.
  - vvanders 11y ago
    I would argue that data layout is even more important the the data structure you use(see radix sort) which is also something the compiler can't control. I pretty strongly disagree that with the meta-parent that it's the "compiler's job to speed up your code". It just doesn't have enough context to be able to do it effectively. I would actually prefer a predictable compiler to one that tries to do all sorts of magic tricks that can be perturbed by seemingly trivial code changes.
- scott_s 11y ago
  Picking the right data structure and algorithms gets your code in the right algorithmic complexity range - did you implement a naive linear algorithm, or did you figure out a way to make it logarithmic? Or maybe even average-case constant? Compilers are generally not good at making optimizations that change complexity class. (Possible, yes, but not commonly.) That's not what jordigh is talking about. Let's say you figured out a clever way to get your problem to O(log n). That's definitely your job. But in practice, constants matter - that is, 10 log n is way, way worse in practice than 2 log n, even though they are both O(log n). And that's the kind of optimization that compilers may be better at than you. Once you have convinced yourself that the performance of some component matters, a good principle is to figure out the best complexity class you can (using the "right" data structures and algorithms), then write the most idiomatic code possible.
jfpuget 11y ago
In light of your comment, it is interesting to see that Swift team just agreed to remove C style for loop. https://lists.swift.org/pipermail/swift-evolution-announce/2015-December/000001.html https://lists.swift.org/pipermail/swift-evolution-announce/2...
Nrpf 11y ago
Right now, numba is a better compiler than Julia has. You can write array expressions and loops in numba and it will do fusion/deforestation to eliminate temporaries giving you faster vectorized in loop code than Julia currently does. Of course, this is currently being worked on in julia as well, but I don't see your point.
JupiterMoon 11y ago
> It's almost a literal translation of the Julia code. Therein lies your problem. If I took some of my python code and literally translated it to Julia would it run like a pig as well? Could I even do a literal translation?
- Veedrac 11y ago
  It'll probably translate, and it'll probably run fine. Julia's like scientific Python, but where the naïve implementation also runs fast. "Fast by default.", perhaps.
  - Nrpf 11y ago
    Actually no. Naive Julia is often slower than python. Vectorized Array expressions are on par with numpy and slower than numba. "type unstable", IO and text processing code is also slower.
    - Veedrac 11y ago
      Although Numba might beat Julia, that's far from saying Julia would "run like a pig".
      - Nrpf 11y ago
        Type unstable Julia is often slower than python.

by jordigh 11y ago

reikonomusha 11y ago

This is a really good comment and covers what a lot of people seem to miss in benchmarks. The fact I can install 2 different compilers (with varying degrees of support) and some hyperoptimized linear algebra libraries don't really tell me about how fast the system is in general when I need performance.

thebooktocome 11y ago

What lessons have you learned from developing Octave that you think Julia could benefit from?

jordigh 11y ago
That's a bit of a tough one. Julia is much newer software than Octave, and it also has a lot of knowledge already of what does and doesn't work. They're already well-staffed by programming language experts. I have learned a lot about how the Matlab language works, and also a lot about how Octave is put together. I don't have much to offer to Julia on technical terms. Octave's codebase started as pre-standard C++, and it's been a long and arduous job to very slowly modernise it. Julia has the advantage of starting modern. I think the biggest lesson I've learned is, don't underestimate Matlab. It may be old, it may be an ugly language, but it also is gigantic, very well-funded, and everywhere. Julia and Octave are approaching Matlab from different angles, Julia by providing a better language and Octave by providing a free drop-in replacement, but ultimately we are both trying to replace Matlab. Matlab users greatly value having a complete package with an editor, debugger, visualiser, profiler, and system modeller all in one. They don't generally see Simulink or the symbolic toolbox as being separate components, or indeed, as the completely separate programs that they really are. For the devout Matlab users, these conveniences greatly outweigh a better programming language. Going further, many Matlab users don't even consider their activity "programming" at all, and in recent years Matlab even writes the code for you if you just push a few buttons. We're seeing some academics slowly switching over to Julia. This is great. The Mathworks knows very well that captivating young, malleable minds in university is how you get loyal paying customers, namely, the future bosses of these newly-molded minds. Julia is doing a good job appealing to profs. I am optimistic that within five years or so, new generations of Julia-trained graduates will slowly spread the love of the new language.
- thebooktocome 11y ago
  > Matlab users greatly value having a complete package with an editor, debugger, visualiser, profiler, and system modeller all in one. That's a very good point. Julia's debugger and profiler are both horrifyingly bad.
  - ViralBShah 11y ago
    The debugger is one of Julia's top priorities, and here's a debugger roadmap that Keno recently posted. https://groups.google.com/forum/#!topic/julia-dev/-LTsBVRv1dU https://groups.google.com/forum/#!topic/julia-dev/-LTsBVRv1d...

jacobolus 11y ago

> What Matlab eventually did was look at what people were writing and making that fast. Except in practice, to write fast Matlab code you need very deep understanding of Matlab’s internals and years of experience. Seemingly trivial patterns end up slowing your code down by multiple orders of magnitude, and there’s more “guess and check” involved when trying to write code that can be effectively JIT compiled by Matlab than careful reasoning. Even worse, the JIT and the profiler don’t get along, so it’s often impossible to get any insight into the reasons for JIT-related performance differences. In general, Matlab is an extremely unpleasant and frustrating environment compared to almost any other language I’ve worked with. The only thing Matlab has on Python/Numpy is a nice quantity of publicly available code for various technical functions. Most of this code is hacky academic prototype stuff, but that’s much better than nothing if you’re trying to follow someone’s algorithm written up in a paper. The Matlab GUI and tooling is a buggy and unpolished Java turd from the 90s which fits in poorly with any modern operating system.

x0x0 11y ago
And that's the pitch for julia. Oh, and mostly fast enough to write in julia the stuff that, for python, you'd write as C extensions, crucially enabling lots of optimizations that are terribly hard to do for python. And avoiding the whole pypy / C api plugin problem. It's still pretty immature, but promising. That said, matlab's linear algebra syntax is still better than python's.
chm 11y ago
Your first quip also applies to Mathematica. It's very easy to write quite inefficient code in Mathematica. One classic example of this is a newcomer not using N[] and evaluating every expression symbolically. Seems trivial, but it really isn't for someone new. I still could get speedups on linalg code two years into grad school.
- TheOtherHobbes 11y ago
  It applies to most languages. In practice you can't rely on the compiler, because you don't know what the compiler is doing. You certainly can't assume it's going to make the smartest possible decision for all of your code. There's no standardisation for optimisations, and they're often a context-dependent trade-off anyway. The only way to write fast code is to learn the quirks of the tool chain and profile production binaries to find the bottlenecks.

jfpuget 11y ago

Thank you, great comment, it spurred a great discussion. Let me just clear one thing: I am not trying to represent Julia in any way. I wouldn't be legitimate for that at all. I tried to not write anything negative about the Julia language. Let me know if you think i did, in which case I'll modify my text. To your point about compilers in charge of speeding up code, I see Numba doing more and more. I hope it will cover all of Python soon.

Lanzaa 11y ago
I thought the article was good. It started with a basic premise, "Should we ditch Python and other languages in favor of Julia for technical computing?" Then the article shows several methods for speeding up python code to be faster than using julia. I was a little disappointed that pypy was not mentioned, but it is a good introduction to speeding up python code. I think the flaw in the article is when it switches to answering, "did the Julia team [write] Python benchmarks the best way for Python?" Then it rewrites the fib implementation to use a cache, which makes the comparison to the julia version completely ridiculous. It also does all sorts of optimization which clearly deviated from the spirit of the benchmark, naive python vs naive julia. I wish the conclusion had been written in a way that clearly answered the original premise, should we ditch python for julia? The article clearly showed that there are a lot of good ways to speed up python code. Looking for algorithmic complexity wins (like in the fib example), using cpython, using numba, and profiling all can be used to speed up python code to the level of naive julia code. Which leads to the conclusion, if all you want is faster code there is no need to ditch python for julia.
- jfpuget 11y ago
  I agree that the first question is a very interesting one. I would be interested in the answer myself. I made it clear however that I was going to answer a different question: did the Julia team wrote Python benchmarks the best way for Python? That's all what the post is about: how to run Python code faster. I will need way more Julia experience to be able to even think about answering the first question. The only thing I am 100% sure is that these micro benchmarks do not help answering that question.

andrepd 11y ago

Exactly! The point is to compare standard, idiomatic and straightforward Julia code with standard, idiomatic and straightforward Python code, not with a jumble of workarounds, experimental JITs, and cumbersome syntax. Because if you are writing statically typed python, then you are not really writing python anymore, are you?

JupiterMoon 11y ago
Looping over lists/arrays is not idiomatic Python.
- co_dh 11y ago
  +1
  - andrepd 11y ago
    You can use the up vote for that.
- tavert 11y ago
  The point of that benchmark is looping over lists or arrays. The benchmark verifies the well-known fact that straight Python is a lousy choice for those situations when you absolutely need to do this. Dropping into C extensions or some-of-the-language Numba would be a different comparison, and show a different result.
  - JupiterMoon 11y ago
    Did they at least attempt list comprehensions rather than loops? EDIT Also in the scientific fields I've encountered not using the various libraries like numpy is non-idomatic. I personally think that python should absorb numpy into the standard library but effectively people I know treat it as if it were already there. You can argue that numpy etc are implemented in C. But is the Julia interpretor not implemented in C?
    - deleted 11y ago

co_dh 11y ago

I don't agree that speeding up your code is your compiler's job. If that is true, there is no point to learn data structure and algorithm.

the_af 11y ago
I don't think you and the parent post are talking about the same kind of speedups. Of course choosing the right algorithm and data structure is your job. How you tackle the problem is definitely your job, not the compiler's. But in general, unrolling loops, caching (some) results, and eliding redundant code rightfully is the compiler's job. That's a different kind of speedup.
- vvanders 11y ago
  I would argue that data layout is even more important the the data structure you use(see radix sort) which is also something the compiler can't control. I pretty strongly disagree that with the meta-parent that it's the "compiler's job to speed up your code". It just doesn't have enough context to be able to do it effectively. I would actually prefer a predictable compiler to one that tries to do all sorts of magic tricks that can be perturbed by seemingly trivial code changes.
scott_s 11y ago
Picking the right data structure and algorithms gets your code in the right algorithmic complexity range - did you implement a naive linear algorithm, or did you figure out a way to make it logarithmic? Or maybe even average-case constant? Compilers are generally not good at making optimizations that change complexity class. (Possible, yes, but not commonly.) That's not what jordigh is talking about. Let's say you figured out a clever way to get your problem to O(log n). That's definitely your job. But in practice, constants matter - that is, 10 log n is way, way worse in practice than 2 log n, even though they are both O(log n). And that's the kind of optimization that compilers may be better at than you. Once you have convinced yourself that the performance of some component matters, a good principle is to figure out the best complexity class you can (using the "right" data structures and algorithms), then write the most idiomatic code possible.

jfpuget 11y ago

In light of your comment, it is interesting to see that Swift team just agreed to remove C style for loop. https://lists.swift.org/pipermail/swift-evolution-announce/2015-December/000001.html https://lists.swift.org/pipermail/swift-evolution-announce/2...

Nrpf 11y ago

Right now, numba is a better compiler than Julia has. You can write array expressions and loops in numba and it will do fusion/deforestation to eliminate temporaries giving you faster vectorized in loop code than Julia currently does. Of course, this is currently being worked on in julia as well, but I don't see your point.

JupiterMoon 11y ago

> It's almost a literal translation of the Julia code. Therein lies your problem. If I took some of my python code and literally translated it to Julia would it run like a pig as well? Could I even do a literal translation?

Veedrac 11y ago
It'll probably translate, and it'll probably run fine. Julia's like scientific Python, but where the naïve implementation also runs fast. "Fast by default.", perhaps.
- Nrpf 11y ago
  Actually no. Naive Julia is often slower than python. Vectorized Array expressions are on par with numpy and slower than numba. "type unstable", IO and text processing code is also slower.
  - Veedrac 11y ago
    Although Numba might beat Julia, that's far from saying Julia would "run like a pig".
    - Nrpf 11y ago
      Type unstable Julia is often slower than python.