11 ms·

Java vs. Scala: Divided We Fail

238 points by mritun 12y ago

integricho 12y ago
I just love the ending sentence in the conclusion.
- jimbishopp 12y ago
  Wish that was the first sentence; would have saved me some time.
  - javadocmd 12y ago
    If that's all you got out of this article then you missed the point. It's clearly targeted as a criticism of "kneel before my benchmark" posts; highlighting the fiddly technical details that make runtime scores nearly meaningless on their own. It's analogous to medical journals having to analyze all the confounders of their study in order to attempt to prove that what their graphs show is causation instead of correlation.
  - octo_t 12y ago
    it is true for basically every programming language, from haskell to C++ to python to ruby.
    - JimmyM 12y ago
      I think it was a joke, but I'm genuinely not sure.
- deleted 12y ago
  [deleted]
mbell 12y ago
I'm left with a feeling something else must be going on here in addition to the 'fast mod' optimization. It looks like the optimization would only be used on the first 2 or 4 calls to isEvenlyDistributed() which doesn't strike me as enough to cut runtime in half, nor does it seem to make sense that you'd see roughly the same performance difference for all values of lim (I'd expect the performance to get closer as lim grows). Am I missing something?
- Perseids 12y ago
  Maybe you underestimate how many numbers are weeded out just by checking divisibility by 3, 4 and 5? The first check regarding 2 will always return true, as `val` is incremented by 2. The check for divisibility by 3 will cause an abort for two out of three calls. The check for divisibility by 4 will cause an abort of half of the remaining calls (as `val` is incremented). And I guess the check for 5 will weed out an additional segment of 4 out 5. Thus just after checking 3, 4 and 5 only 1/3 * 1/2 * 1/5 of the `val`s will continue down the recursion call. So the majority of calls terminate in the highly optimized static division code. Now you can still argue that the remaining calls that go deep into the recursion could account for a lot of time. I don't believe this is the case though, as in each recursion call you should terminate a constant fraction of each calls, so the drop down should be exponential. An other interesting observation of this phenomenon is that running the recursion backwards (beginning with the large divisors) might greatly decrease the runtime as a more significant fraction of the `val`s are falsified by larger divisors in the first calls of the recursion.
  - mbell 12y ago
    I see what your saying and that was the type of thing I was getting at. If i understood the article correctly, the claim was that it was faster because the first 2-4 checks could be optimized by avoiding the idiv. But even if you optimize those check to zero time, that is a small number of checks in comparison the number of checks needed as lim increases (even at lim = 20, only 10%-20% of the checks would be optimized). It doesn't follow that the run time would be cut in half based on the article's conclusions alone.
- dbenhur 12y ago
  With 64bit operands, IMUL+SAR is between 10 and 25 times faster than IDIV on current x86 hardware. (div speed is very dependant on the values being divided. eg: on Sandy Bridge arch, latency is between 40 and 103 cycles.) With these deltas, the first few checks are essentially free. A large number of the numbers to check will be eliminated early and never try the expensive idiv tests.
chton 12y ago
While it felt a bit chaotically written (maybe due to my lack of in-depth JVM knowledge), a very interesting article. Is there a way to detect when these cases happen? If there was, the compiler could choose to ignore the @tailrec directive if it wouldn't give any meaningful speed increase.
- bad_user 12y ago
  The purpose of `@tailrec` is not necessarily for performance, but more for correctness, otherwise you can end up with recursive functions that blow up the stack pretty fast on large enough inputs. Plus this benchmark only does at most 20 iterations per loop - which means nothing if you want to measure the cost of method calls.
  - chton 12y ago
    hmm that's a very good point, I didn't consider that. We'd need additional benchmarking to know what the actual impact is and at what point the @tailrec version becomes more desirable.
hellodevnull 12y ago
>because they both allow me to write stupid programs with performance problems. Which language doesn't?
- michaelochurch 12y ago
  He's being sarcastic (I think).
  - bad_user 12y ago
    I'm pretty sure he's being half-sarcastic.
    - michaelochurch 12y ago
      I wonder what he considers to be good languages. Is he a C++ diehard, or a Haskell purist?
      - javadocmd 12y ago
        Perhaps he's saying all current languages suck in that respect, and that the future of programming is languages that don't let you write unoptimizable code.
        fleitz 12y ago
        I guess programmers won't be allowed to write optimized code anymore, as optimized code is unoptimizable :)
  - hellodevnull 12y ago
    Okay and I have somehow said something terrible because I didn't pick up on this hilarious dubious sarcasm in a post about benchmarks.
- szatkus 12y ago
  Brainfuck.
michaelochurch 12y ago
I'm going to comment on the political aspect and leave the technical JVM-centric bit alone. In this example, did we learn what is faster, Java or Scala? Nope. But we learned a lot digging for explanations why the results are different, which will hopefully result in having better cohesion between languages and the underlying platform. Performance is often FUD when it comes to languages. Clojure, Scala, Haskell, and even Common Lisp are plenty fast enough for most purposes. Hell, Python and Ruby are fast enough for most applications. Besides, runtime performance has more to do with how the code is written than anything else. You can write very fast C++, if you spend a lot tweaking it and hire experience and extremely expensive ($300k/year and up) programmers, but typical C++ isn't any faster than well-written code in other languages. (I've seen C++ projects fail for performance reasons related to maintainability issues that arguably wouldn't exist if Haskell were used.) However, invoking performance is a great (if unreliable, given who it brings on to the field) way to scare decision-making business people (toddlers with guns) into taking your side on an issue they know nothing about. This is a place where it'd be better if programmers were a little more politically savvy. (Bringing The Business into a technical dispute is not politically savvy. It ruins everything, in the long run. Never invite executives, also known as toddlers with guns, to anything. As many a Chinese noble learned about opening The Wall and letting the Mongols in to fight one's battles, it's impossible to get them out after it's done.) Let's say you have a team of 5 programmers who want to write something in Python, which is (for most purposes) fast enough. One of them stands up and says, "oh no, we can't do this in Python because if we end up running this on 100,000 boxes it will be too expensive, so we can only use C++" (premature optimization). If he were more politically savvy, he'd build the thing in Python and then, if the software were to run on 100,000 boxes, rewrite performance-critical pieces in C++, and justify a bonus for himself by pointing to the 20,000 CPUs that were just deprovisioned. And a year later, a month before bonuses are disbursed, he can rewrite another performance-critical component. This is good for him (he actually gets recognized, instead of being that annoying guy who bludgeoned a team into writing C++ and taking 4x as long to deliver an MVP) and for the business (only performance-critical components, with price tags large enough for him to care, get rewritten). Oh, and if you think that part is constructive and boring, and you came here for a language holy-war, I do think both Java and Scala suck as programming languages, because they both allow me to write stupid programs with performance problems. I assume this is sarcasm, but I actually worry about the fact that Those of Us Who Care About Languages are too divided over minutiae like Haskell's syntactic whitespace and Clojure's parentheses, and that may be why The Business comes in, mushroom stamps us and says, "I'm sick of your shit, programmers. Everything has to be in Java." (Actually, if you've watched Orange is the New Black, you know that The Business has taken to calling software engineers "inmate", but that's another discussion.) I like Haskell and I like Clojure for very different reasons, and they are very different languages, but the day-to-day real-world differences between them are small compared to the very real risk of The Business overhearing our flamewar and saying, "fuck you guys, Java all the way, now lick my SCRUM or it's minus-5 story points for you." One of the things that worries me about Clojure is that, while I'd argue that it's the best (for a definition of "best" that includes short-term business viability; this enables me to exclude obscure niche languages that may be better on paper, but that are just too numerous for me to know anything about) dynamically-typed language-- a great language on its own right, but sitting on top of the JVM and having access to those libraries-- there's been a mind-share split between Clojure and Scala. And while Scala/Java and Clojure/Java interop aren't bad, Clojure/Scala interop is a mess. On top of this, while Odersky's brilliant, I think Scala has taken in a little too much of the Java culture for it's own good. Scala's a fine language to write in, but large Scala codebases are generally things that I'd rather not risk my sanity and career by being anywhere near them. If Scala wanted to be "Haskell with Java libraries" it would been a different and harder fight... but then again, it might not have taken off at all without the "slightly better Java" crowd, so maybe the way things happened was the only possibility. The mind-share split between Clojure and Scala scares me, because it generates a very real risk that the intellectual energy that I'd like to see benefitting both languages, or at least consistently benefitting one of them, might fall back down the tree onto Java. The real risk, to me, is "Clojure vs. Scala: Divided We Fall". But I don't know exactly what to do about it.
skybrian 12y ago
The conclusion I take away from this is that good performance isn't necessarily more useful than predictable performance. For many language implementations, the performance characteristics are quite fragile, and you'd better be benchmarking after every change or you won't notice a regression. Even setting up a benchmark is tricky because the same binary can have dramatically different performance depending on the environment. At the machine level, caching effects from different processors can have dramatically different effects on performance. But compilers and runtimes can easily make things even more unpredictable, requiring things like warmup time to even hope to get an idea of what's going on. I think Go starts out a bit closer to achieving predictable performance; any ahead-of-time compiler has an advantage here. Other than the use of garbage collection, the whole language seems to be designed with predictable performance in mind over ease of use, and you can avoid GC where it matters. Also, recently they replaced segmented stacks with copying of contiguous stacks, which both improves performance and makes it more predictable (while a goroutine is running).
- pron 12y ago
  > good performance isn't necessarily more useful than predictable performance That may be true, but on modern machines it's pretty much a pipe dream, as even the hardware nowadays (let alone modern OSs) is unpredictable. So once you get to the compiler it's a little late to get predictability. I think we're better off foregoing predictability altogether -- unless when writing real time applications (and I'm using realtime in its original sense) -- and getting the best performance we can. JITs currently seem to be the best hope for top performance, especially for a given unit of manual programmer effort expended on optimization.
  - _dps 12y ago
    (note: this comes mostly from a background in scientific/numerical programming in C) You are certainly right if one takes "predictable" to mean something close to "deterministic"; on the other hand I think there are numerous almost-deterministic characteristics that can be evaluated without resorting to plain-ol' empiricism (particularly involving likelihood of hits in L1/L2/L3 and likelihood of being able to speculatively execute particular branches of code). So I would agree that my a priori predictions may be a factor of 2-3x off; OTOH I often succeed in predicting that a particular tight loop will never have to leave L1, or that a low-entropy condition in the loop will be made essentially irrelevant by speculative execution. With that proviso, I share your belief that JITs seem to be the best hope for top performance (and I am a heavy user of LuaJIT).
_3u10 12y ago
tl;dr: When trying to make code run faster, use a profiler rather than doing things you think make the code faster.
joereggan190 12y ago
http://learn.hightail.com/t5/Ideas/12-00-Watch-Argentina-vs-Belgium-Live-Stream-2014-WOrld-Cup/idi-p/86046 http://learn.hightail.com/t5/Ideas/12-00-Watch-Argentina-vs-...

238 points by mritun 12y ago

integricho 12y ago

I just love the ending sentence in the conclusion.

jimbishopp 12y ago
Wish that was the first sentence; would have saved me some time.
- javadocmd 12y ago
  If that's all you got out of this article then you missed the point. It's clearly targeted as a criticism of "kneel before my benchmark" posts; highlighting the fiddly technical details that make runtime scores nearly meaningless on their own. It's analogous to medical journals having to analyze all the confounders of their study in order to attempt to prove that what their graphs show is causation instead of correlation.
- octo_t 12y ago
  it is true for basically every programming language, from haskell to C++ to python to ruby.
  - JimmyM 12y ago
    I think it was a joke, but I'm genuinely not sure.
deleted 12y ago
[deleted]

mbell 12y ago

I'm left with a feeling something else must be going on here in addition to the 'fast mod' optimization. It looks like the optimization would only be used on the first 2 or 4 calls to isEvenlyDistributed() which doesn't strike me as enough to cut runtime in half, nor does it seem to make sense that you'd see roughly the same performance difference for all values of lim (I'd expect the performance to get closer as lim grows). Am I missing something?

Perseids 12y ago
Maybe you underestimate how many numbers are weeded out just by checking divisibility by 3, 4 and 5? The first check regarding 2 will always return true, as `val` is incremented by 2. The check for divisibility by 3 will cause an abort for two out of three calls. The check for divisibility by 4 will cause an abort of half of the remaining calls (as `val` is incremented). And I guess the check for 5 will weed out an additional segment of 4 out 5. Thus just after checking 3, 4 and 5 only 1/3 * 1/2 * 1/5 of the `val`s will continue down the recursion call. So the majority of calls terminate in the highly optimized static division code. Now you can still argue that the remaining calls that go deep into the recursion could account for a lot of time. I don't believe this is the case though, as in each recursion call you should terminate a constant fraction of each calls, so the drop down should be exponential. An other interesting observation of this phenomenon is that running the recursion backwards (beginning with the large divisors) might greatly decrease the runtime as a more significant fraction of the `val`s are falsified by larger divisors in the first calls of the recursion.
- mbell 12y ago
  I see what your saying and that was the type of thing I was getting at. If i understood the article correctly, the claim was that it was faster because the first 2-4 checks could be optimized by avoiding the idiv. But even if you optimize those check to zero time, that is a small number of checks in comparison the number of checks needed as lim increases (even at lim = 20, only 10%-20% of the checks would be optimized). It doesn't follow that the run time would be cut in half based on the article's conclusions alone.
dbenhur 12y ago
With 64bit operands, IMUL+SAR is between 10 and 25 times faster than IDIV on current x86 hardware. (div speed is very dependant on the values being divided. eg: on Sandy Bridge arch, latency is between 40 and 103 cycles.) With these deltas, the first few checks are essentially free. A large number of the numbers to check will be eliminated early and never try the expensive idiv tests.

chton 12y ago

While it felt a bit chaotically written (maybe due to my lack of in-depth JVM knowledge), a very interesting article. Is there a way to detect when these cases happen? If there was, the compiler could choose to ignore the @tailrec directive if it wouldn't give any meaningful speed increase.

bad_user 12y ago
The purpose of `@tailrec` is not necessarily for performance, but more for correctness, otherwise you can end up with recursive functions that blow up the stack pretty fast on large enough inputs. Plus this benchmark only does at most 20 iterations per loop - which means nothing if you want to measure the cost of method calls.
- chton 12y ago
  hmm that's a very good point, I didn't consider that. We'd need additional benchmarking to know what the actual impact is and at what point the @tailrec version becomes more desirable.

hellodevnull 12y ago

>because they both allow me to write stupid programs with performance problems. Which language doesn't?

michaelochurch 12y ago
He's being sarcastic (I think).
- bad_user 12y ago
  I'm pretty sure he's being half-sarcastic.
  - michaelochurch 12y ago
    I wonder what he considers to be good languages. Is he a C++ diehard, or a Haskell purist?
    - javadocmd 12y ago
      Perhaps he's saying all current languages suck in that respect, and that the future of programming is languages that don't let you write unoptimizable code.
      - fleitz 12y ago
        I guess programmers won't be allowed to write optimized code anymore, as optimized code is unoptimizable :)
- hellodevnull 12y ago
  Okay and I have somehow said something terrible because I didn't pick up on this hilarious dubious sarcasm in a post about benchmarks.
szatkus 12y ago
Brainfuck.

michaelochurch 12y ago

I'm going to comment on the political aspect and leave the technical JVM-centric bit alone. In this example, did we learn what is faster, Java or Scala? Nope. But we learned a lot digging for explanations why the results are different, which will hopefully result in having better cohesion between languages and the underlying platform. Performance is often FUD when it comes to languages. Clojure, Scala, Haskell, and even Common Lisp are plenty fast enough for most purposes. Hell, Python and Ruby are fast enough for most applications. Besides, runtime performance has more to do with how the code is written than anything else. You can write very fast C++, if you spend a lot tweaking it and hire experience and extremely expensive ($300k/year and up) programmers, but typical C++ isn't any faster than well-written code in other languages. (I've seen C++ projects fail for performance reasons related to maintainability issues that arguably wouldn't exist if Haskell were used.) However, invoking performance is a great (if unreliable, given who it brings on to the field) way to scare decision-making business people (toddlers with guns) into taking your side on an issue they know nothing about. This is a place where it'd be better if programmers were a little more politically savvy. (Bringing The Business into a technical dispute is not politically savvy. It ruins everything, in the long run. Never invite executives, also known as toddlers with guns, to anything. As many a Chinese noble learned about opening The Wall and letting the Mongols in to fight one's battles, it's impossible to get them out after it's done.) Let's say you have a team of 5 programmers who want to write something in Python, which is (for most purposes) fast enough. One of them stands up and says, "oh no, we can't do this in Python because if we end up running this on 100,000 boxes it will be too expensive, so we can only use C++" (premature optimization). If he were more politically savvy, he'd build the thing in Python and then, if the software were to run on 100,000 boxes, rewrite performance-critical pieces in C++, and justify a bonus for himself by pointing to the 20,000 CPUs that were just deprovisioned. And a year later, a month before bonuses are disbursed, he can rewrite another performance-critical component. This is good for him (he actually gets recognized, instead of being that annoying guy who bludgeoned a team into writing C++ and taking 4x as long to deliver an MVP) and for the business (only performance-critical components, with price tags large enough for him to care, get rewritten). Oh, and if you think that part is constructive and boring, and you came here for a language holy-war, I do think both Java and Scala suck as programming languages, because they both allow me to write stupid programs with performance problems. I assume this is sarcasm, but I actually worry about the fact that Those of Us Who Care About Languages are too divided over minutiae like Haskell's syntactic whitespace and Clojure's parentheses, and that may be why The Business comes in, mushroom stamps us and says, "I'm sick of your shit, programmers. Everything has to be in Java." (Actually, if you've watched Orange is the New Black, you know that The Business has taken to calling software engineers "inmate", but that's another discussion.) I like Haskell and I like Clojure for very different reasons, and they are very different languages, but the day-to-day real-world differences between them are small compared to the very real risk of The Business overhearing our flamewar and saying, "fuck you guys, Java all the way, now lick my SCRUM or it's minus-5 story points for you." One of the things that worries me about Clojure is that, while I'd argue that it's the best (for a definition of "best" that includes short-term business viability; this enables me to exclude obscure niche languages that may be better on paper, but that are just too numerous for me to know anything about) dynamically-typed language-- a great language on its own right, but sitting on top of the JVM and having access to those libraries-- there's been a mind-share split between Clojure and Scala. And while Scala/Java and Clojure/Java interop aren't bad, Clojure/Scala interop is a mess. On top of this, while Odersky's brilliant, I think Scala has taken in a little too much of the Java culture for it's own good. Scala's a fine language to write in, but large Scala codebases are generally things that I'd rather not risk my sanity and career by being anywhere near them. If Scala wanted to be "Haskell with Java libraries" it would been a different and harder fight... but then again, it might not have taken off at all without the "slightly better Java" crowd, so maybe the way things happened was the only possibility. The mind-share split between Clojure and Scala scares me, because it generates a very real risk that the intellectual energy that I'd like to see benefitting both languages, or at least consistently benefitting one of them, might fall back down the tree onto Java. The real risk, to me, is "Clojure vs. Scala: Divided We Fall". But I don't know exactly what to do about it.

skybrian 12y ago

The conclusion I take away from this is that good performance isn't necessarily more useful than predictable performance. For many language implementations, the performance characteristics are quite fragile, and you'd better be benchmarking after every change or you won't notice a regression. Even setting up a benchmark is tricky because the same binary can have dramatically different performance depending on the environment. At the machine level, caching effects from different processors can have dramatically different effects on performance. But compilers and runtimes can easily make things even more unpredictable, requiring things like warmup time to even hope to get an idea of what's going on. I think Go starts out a bit closer to achieving predictable performance; any ahead-of-time compiler has an advantage here. Other than the use of garbage collection, the whole language seems to be designed with predictable performance in mind over ease of use, and you can avoid GC where it matters. Also, recently they replaced segmented stacks with copying of contiguous stacks, which both improves performance and makes it more predictable (while a goroutine is running).

pron 12y ago
> good performance isn't necessarily more useful than predictable performance That may be true, but on modern machines it's pretty much a pipe dream, as even the hardware nowadays (let alone modern OSs) is unpredictable. So once you get to the compiler it's a little late to get predictability. I think we're better off foregoing predictability altogether -- unless when writing real time applications (and I'm using realtime in its original sense) -- and getting the best performance we can. JITs currently seem to be the best hope for top performance, especially for a given unit of manual programmer effort expended on optimization.
- _dps 12y ago
  (note: this comes mostly from a background in scientific/numerical programming in C) You are certainly right if one takes "predictable" to mean something close to "deterministic"; on the other hand I think there are numerous almost-deterministic characteristics that can be evaluated without resorting to plain-ol' empiricism (particularly involving likelihood of hits in L1/L2/L3 and likelihood of being able to speculatively execute particular branches of code). So I would agree that my a priori predictions may be a factor of 2-3x off; OTOH I often succeed in predicting that a particular tight loop will never have to leave L1, or that a low-entropy condition in the loop will be made essentially irrelevant by speculative execution. With that proviso, I share your belief that JITs seem to be the best hope for top performance (and I am a heavy user of LuaJIT).

_3u10 12y ago

tl;dr: When trying to make code run faster, use a profiler rather than doing things you think make the code faster.

joereggan190 12y ago

http://learn.hightail.com/t5/Ideas/12-00-Watch-Argentina-vs-Belgium-Live-Stream-2014-WOrld-Cup/idi-p/86046 http://learn.hightail.com/t5/Ideas/12-00-Watch-Argentina-vs-...