21 ms·

Google paper comparing performance of C++, Java, Scala, and Go [PDF]

200 points by pgbovine 15y ago

LiveTheDream 15y ago
Loaded directly as a PDF for me.
- rrrazdan 15y ago
  The scribd link is an alternate.
  - kristianp 15y ago
    A bit of cross-promotion, Scribd is a ycombinator company.
  - manveru 15y ago
    At least on my galaxy tab using opera mobile, the scribd link simply tells me to download the PDF.
copper 15y ago
The most instructive part of this are the listings from the changelogs alongwith the resulting improved performance. I'm really enjoying the C++ tunings. I wish they did show the results with the google hashes, though.
- btmorex 15y ago
  So, the C++ performance numbers were with the "public" version? It wasn't clear whether they left the internal version out completely (besides the changelog) or simple didn't release the code. The published C++ numbers are already way better than any other language in terms of speed and memory use. It would be surprising if there was a substantially faster version.
  - copper 15y ago
    The final list of changes is supposed to be with the internal version, and roughly seem to show a 20% change over and above what the basic std::map -> ext/hash_map replacement. I'm not sure if I should add the percentages, or multiply them along to get the final runtime, though that doesn't make much difference, really :) You're right that there may not be much improvement, but I've seen the google dense_hash_map do better sometimes than the basic SGI hash_map. Edit: the code repo has the unoptimized versions. Oh well, at least I can get the satisfaction of seeing the 30% myself :)
  - igouy 15y ago
    afaict Yes, the C++ performance numbers were with the "public" version. afaict Yes, they left the internal version out completely (besides the changelog). Notice C++ Dbg has the same wc -l as C++ Opt.
    - copper 15y ago
      Isaac, do you think you could add this problem to the shootout? I believe (and am therefore almost certainly wrong) that there isn't any benchmark there that derives almost purely from compiler theory.
pgbovine 15y ago
summary: author implemented a sophisticated compiler algorithm (loop header recognition) in a straightforward canonical way in C++, Java, Scala, and Go, measured performance/memory usage, and then (the most interesting part of the paper) he had colleagues who were experts in each respective language write highly-tuned optimized versions and reported on what it took to optimize in each language.
- CCs 15y ago
  So C++ wins the performance test by large margin, Scala follows (3.6x), then Java 64bit (5.8x), Go (7x) and finally Java 32bit (12.6x). Scala without any optimization is still better than Java with all the ninja-skills black magic applied. Scala optimized is pretty good - "just" 2.5x slower than C++
  - arron61 15y ago
    Actually it's odd. They say that in the Java tuning, they made some simple optimization that got it on par to the original C++ one but then they refused to do anything further; noting instead the same C++ optimization would have applied in Java as well. Does that mean the java version would have been just as fast? It seems in general though that Java is just as fast as C++ unless you are an ultimate C++ expert.
    - johnm 15y ago
      Indeed, this is ridiculous and makes their final conclusions weaker.
  - wtracy 15y ago
    Maybe I'm missing something, but I'm amazed to see Scala beating out Java.
    - gaius 15y ago
      Why? Remember that at this point in history, Java-the-language and Java-the-VM are related in name only.
skimbrel 15y ago
Wow. Looks like C++ is still one of the performance kings among current programming languages. Can't say I enjoy using it though. It also looks like Go and Scala have a long way to go on compiler optimization. Not surprising, since they're young languages compared to C++ and Java, and it's a decent amount of work to optimize the higher-order constructs they provide.
- latch 15y ago
  Languages that come from an era where total memory was measured in kbytes, cpus in megahertz and pointer arithmetic was as common as if statements, will always be performance kings. It'll take a pretty significant shift to change this. Something like 100+ core cpus each running slow, might be such a shift. Being surprised that C++ is faster than newer languages which provide all types of abstraction is like being surprise assembly is faster than C++.
  - anonymous246 15y ago
    You're mostly correct, but the code they wrote DID use fancy abstractions like hash tables, templates etc. C++ delivered on abstraction WITHOUT a performance penalty, which is what the C++ designers have claimed it could do since forever. Nice to see it verified again. I have no doubt that the C++0x syntactic sugar enhancements would have have also delivered the same high performance.
    - alexkcd 15y ago
      Indeed. I write a lot of Objective-C code, and still drop into C++ for the CPU heavy tasks. Swapping out NS* containers for std::* alone gives a fairly high boost in performance. All the C++ hate is blown way out of proportion. C++ is a great tool when you need performance. Tuning C++ isn't that difficult either. Contrary to what a commenter said above, steering clear of C++ at all costs is not what you should be doing. You should know when and how to use C++. It's useful more often than many people here seem to think. Especially if you develop for mobile devices.
scottjad 15y ago
Speaking of Scala Pro: "It should be noted that this version performs algorithmic improvements as well, and is therefore not directly comparable to the other Pro versions."
- enneff 15y ago
  "It should be noted"?! What an understatement! What's the point of going to the trouble of writing a paper if you're not even going to make your comparisons fair? Crazy.
  - Djehngo 15y ago
    Because the initial comparison was fair.
    - igouy 15y ago
      Really? "... the benchmark also used a HashMap<Type, Integer> when it should have just stored a primitive int in Type; it also used a LinkedList where it should have used an ArrayDeque." http://jeremymanson.blogspot.com/2011/06/scala-java-shootout.html http://jeremymanson.blogspot.com/2011/06/scala-java-shootout...
choxi 15y ago
This is cool but I think it's important to remember that they're comparing algorithmic benchmarks but modern software has many different bounds besides for raw, algorithmic processing. For example, I think I/O and database interactions are the bottleneck for most web apps. I really liked the way this article broke it down: http://pl.atyp.us/wordpress/?p=2947 http://pl.atyp.us/wordpress/?p=2947
- cdavid 15y ago
  Given the languages they are using, I think it is fair to say that they are interested in system-programming kind of tasks, which for google means the stacks below what you normally use for web programming (i.e. I would expect google to have their own web servers, or at least heavily customized from existing ones, etc...). For system programming, "language speed" matters much more (one definition I like for system programming, seen on lambda ultimate: "programming where both algorithm complexity's constant and memory representation matter"). Of course, in most cases, you're right. Given an infinite pool of very good programmers and infinite time, C will almost always be faster than most other languages. But in reality, you have a very limited of maybe not so good programmers, and the project needed to be done yesterday :)
- gtani 15y ago
  Thx for that. Java and scala-specifically, it's a pretty high-dimension space. Background: http://www.elis.ugent.be/JavaStats http://www.elis.ugent.be/JavaStats http://www.bestinclass.dk/index.clj/2010/02/benchmarking-jvm-languages.html http://www.bestinclass.dk/index.clj/2010/02/benchmarking-jvm... http://isthisclojure.blogspot.com/2011/02/benchmark-clojurejvm-5-degrees-of.html http://isthisclojure.blogspot.com/2011/02/benchmark-clojurej... https://docs.google.com/present/view?id=0AS8emH3-FLt3ZGRtbWJyOGdfMTFmcDZkcTk2cw&hl=en https://docs.google.com/present/view?id=0AS8emH3-FLt3ZGRtbWJ... Examples of benchmarks gone awry: http://stackoverflow.com/questions/6146182/why-is-my-scala-code-running-slow http://stackoverflow.com/questions/6146182/why-is-my-scala-c... http://groups.google.com/group/scala-language/browse_thread/thread/94740a10205dddd2/ http://groups.google.com/group/scala-language/browse_thread/...
  - igouy 15y ago
tectonic 15y ago
Their conclusions: "We ﬁnd that in regards to performance, C++ wins out by a large margin. However, it also required the most extensive tuning efforts, many of which were done at a level of sophistication that would not be available to the average programmer. Scala concise notation and powerful language features allowed for the best optimization of code complexity. The Java version was probably the simplest to implement, but the hardest to analyze for performance. Speciﬁcally the effects around garbage collection were complicated and very hard to tune. Since Scala runs on the JVM, it has the same issues. Go offers interesting language features, which also allow for a concise and standardized notation. The compilers for this language are still immature, which reﬂects in both performance and binary sizes."
- gaius 15y ago
  It's not that you have to tune C++ it's that you can.
  - keeperofdakeys 15y ago
    Untuned C++ will probably be slower then even tuned python, I can tune any language, but C++ (although less so then something like C) requires more tuning to get good performance. The performance is what we care about here.
    - Raphael_Amiard 15y ago
      > Untuned C++ will probably be slower then even tuned python [citation needed]
      - keeperofdakeys 15y ago
        By 'untuned' I'm also including using the wrong algorithm. For example, the using the usual recursive definition for the Fibonacci sequence in C++ as opposed to the loop or n^2 version of recursive Fibonacci in python. The python version will be faster in this case. It is probably true however that any decent programmer will churn out C++ that is faster than python for cpu intensive tasks most of the time.
ivan_ah 15y ago
link to the code: http://code.google.com/p/multi-language-bench/ http://code.google.com/p/multi-language-bench/ This is a great resource to learn the different languages. (I had never seen C++ template programming used in practice before) Challenge to the pythonistas: rewrite the python version so it uses the C++ code under the hood using scipy weave
- igouy 15y ago
  http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/
- lbolla 15y ago
  Templates are not used in the C++ benchmark.
jbpritts 15y ago
C++ code, when sophisticated template meta programming techniques are used, can be as fast as tuned C code, and can beat any language in terms of speed with the exception of hand-tuned assembly. However, techniques like expression templates, automatic loop unrolling, template specialization, and static polymorphism require a VERY high degree of sophistication from the programmer. In some sense, the programmer is forced to be a compiler. There are some individuals that, despite this demand, can be highly productive. The others wallow in complicated syntax, horrible type errors, and difficult to trace run-time errors. Unless speed is absolutely essential, C++ should be avoided at all costs.
- pnathan 15y ago
  C++ TMP is quite painful to the uninitiated. Outside of that, C++ is not that bad, especially as you build up your project-specific abstractions.
onan_barbarian 15y ago
Using a "older Pentium IV workstation". Older than what? Older than Grandpa? Older than dirt? Older than the dinosaurs? Why? Pentium IV - really? Let's see - we've had Core, Core 2, Nehalem and Sandy Bridge since then, not to mention 3 in-between process shrink versions of the same architecture. Perhaps we could pass the hat around so that Google could afford a Sandy Bridge workstation and discover what these interesting results look like on an architecture that dates from some point in this decade - or at least, at some point in the previous decade. This stuff actually, really, truly makes a difference. And not necessarily in favor or against any particular one of these languages...
- acqq 15y ago
  Imagine: there is a paper discussing the techniques of weight lifting. Your complaint is that there are newer models of weights than those they lifted.
  - jnhnum1 15y ago
    Newer processors don't just increase the performance of all programs equally. They have things like improved branch prediction, cache prefetching, better pipelining, and different cache sizes which can make a lot of performance optimizations that you get from C / C++ less relevant.
    - acqq 15y ago
      Except from the "most optimized C++ code" you have all sources of all benchmarks. Please try to run them on any newer x86/64 based processor and show us that the article conclusions don't hold. I'm not holding my breath though.
      - scott_s 15y ago
        I agree that it probably won't make a difference. But good experiments remove as many of those probablys as possible.
ksolanki 15y ago
We find that in regards to performance, C++ wins out by a large margin. However, it also required the most extensive tuning efforts, many of which were done at a level of sophistication that would not be available to the average programmer. I am a fan of C++, so I liked the first part (that C++ wins, which is not really surprising). However it was not clear if the sophistication they talk about is regarding the Google's internal data-structures or the list of optimizations listed in Section VI-D? Many of these optimizations are not big surprises to a C++ programmer (where possible use hash_map, vector instead of list, initialize data structure outside of loop and try to reuse, and so on). Overall, I do greatly appreciate the effort and their sharing the results.
- bad_user 15y ago
  The problem with possible optimizations and tunning is that it requires constant effort. Projects often have serious deadlines and you simply can't constantly think about memory or allocation footprint and still do a good job in the amount of time you have, unless you're well above average.
  - davidtgoldblatt 15y ago
    Most of the optimizations that got them big wins in C++ (using hash maps instead of tree-based maps, dynamic arrays instead of linked lists, .empty() rather than .size() > 0 on a linked list) are "library knowledge" issues more than memory / allocation ones. In fact, looking at the list of optimizations on page 9, the only one that involved a low-level understanding of the machine and resulted in a significant improvement was (unless I missed something) the use of InlinedVector (and this is only barely such a case - lots of C++ codebases have a similar class).
    - acqq 15y ago
      Exactly. Using hash maps and vectors are really elementary choices. Can you consider understanding enough in "systems programming" (see other comments for definition) without knowing that hash_map is implemented using hash tables and map using a tree? And that therefore adding elements to the map would incur rebalancing costs which hash_map wouldn't, as soon as you can estimate the number of elements to be processed. I consider it more a "meta" than a "language" decision.
zohebv 15y ago
One of the interesting details is that Go Pro required 786 lines of code as opposed to 850 lines of C++ code and 297 lines of Scala Pro code. Regular Go code was 902 lines. It certainly doesn't look like Go scores any expressiveness points over C++.
- cageface 15y ago
  It may just be because I've been writing a ton of C++ lately, but to me the C++ data declarations are the most readable of the bunch. The stuff with iterators is much uglier, of course, but so far I like the syntax of C++ a lot better than that of Go.
  - stephen_g 15y ago
    Yep, iterators are one of the most annoying things in C++. That's why I love range for in C++0x - instead of writing this, for example: for(std::vector<std::string>::iterator it = vec.begin(); it != vec.end(); it++) you can just do: for(std::string x : vec) So awesome!
    - dzorz 15y ago
      Even better: for (auto &x : vec) { }
      - cageface 15y ago
        I can't wait until I can start using this in XCode.
        dzorz 15y ago
        If you compile recent clang, it supports range based for loops. Using https://github.com/Rip-Rip/clang_complete https://github.com/Rip-Rip/clang_complete you can use smart context-aware completion with vim/emacs. For example if you had this code: vector<string> vec; for (auto &x : vec) { x and then if you typed period (.), it would show members from std::string. I'd say it works better than Intellisense (it's very precise and you use the same parser for code completion and final compilation).
nimrody 15y ago
Keep in mind that since C++ is usually used for performance critical code, its compilers are often highly tuned and produce good code (see Intel's ICC for number crunching code). A less popular language - or one that is usually used for less-critical code - will suffer. C++ wasn't always so fast. See Kernighan and Pike, "The Practice of Programming" for a simple example where Java surprisingly beats C++.
- cageface 15y ago
  True, but this is one of the reasons people choose to code in C++. The quality of tools & common libraries is as much a factor in language selection as the syntax and semantics of the language itself.
  - nimrody 15y ago
    You mean - "forced to use C++" :) I would love to use Digital Mars D or Scala for scientific applications if they had compilers/runtimes as good as C++. Perhaps one day the LLVM backend would be useful as a universal backend for many less popular languages.
aria 15y ago
Really shocked they would place numbers in a table like that when the 'Scala Pro' version is doing something algorithmically smarter. They note in the paper this distinction, but it should be marked in the table, because I'm sure 99% of people don't see that. Sloppy!
- sili 15y ago
  That, plus the fact that they refused to do same optimizations to Java as they did to C++ for some reason.
xedarius 15y ago
I wonder what JVM they used, as significant performance differences exist between Hotspot and JRockit. I also wonder what compiler they used, again Microsoft compiler will yield different results compared to gcc.
- igouy 15y ago
  Here's the Makefile - gcc http://code.google.com/p/multi-language-bench/source/browse/trunk/src/havlak/cpp/Makefile http://code.google.com/p/multi-language-bench/source/browse/...
mhd 15y ago
It probably won't take long until someone does the same exercise with other languages. Maybe they should add it to the shootout. Can't wait to see Modula-3's numbers…
- igouy 15y ago
  > Maybe they should... "they"? The space monsters who run the universe?
thesz 15y ago
What I'd like to see there is Haskell version and how all those implementations speed up when running on several cores. This is much more interesting considering our current reality. I once gained 10% speed up on two cores by changing just two lines of our optimization tool written on Haskell.
olifante 15y ago
TLDR: C++ fastest but hardest, Scala fast and elegant, Java simple but opaque, Go immature.
javanix 15y ago
This paper served as a great introduction to Go programming for me. I've been trying to work through my own basic program in Go for the past few weeks and their overview of the language features used cleared up a lot of loose ends for me.
- scott_s 15y ago
  Having the same algorithm implemented in several other languages helps, too. It enables you to look at the implementation in the language you're most comfortable with, then look at one of the other implementations and think "Oh, so that's how that's done in this language."
  - igouy 15y ago
    http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/ Although you shouldn't necessarily think "Oh, so that's how that's done in this language."
- supersillyus 15y ago
  Keep in mind, though, that the Go code shown there is not a good example of how idiomatic Go is written.
DonnyV 15y ago
Who said that when you release a paper that it needs to look like it came from a text book. I couldn't get past the first page because I felt like I was back in high school.
- sparky 15y ago
  >Who said that... The conference. From http://days2011.scala-lang.org/node/91 http://days2011.scala-lang.org/node/91 : Submissions must be in English and at most 12 pages total length in the standard ACM SIGPLAN two-column conference format (10pt). Nearly all academic papers in computer science and engineering look like this. Not to say they've converged on an optimal format, but it wasn't exactly a creative decision on the part of these authors.
cygwin98 15y ago
I was surprised by the fact that the Python version has 626 lines of code, compared to 297 to the Scala Pro. As Python is supposed to be more expressive, I was expecting something like 150ish LOC for Python. Edit: The Python version can be found at http://code.google.com/p/multi-language-bench/source/browse/trunk/src/havlak/python/LoopTesterApp.py http://code.google.com/p/multi-language-bench/source/browse/...
- euroclydon 15y ago
  I would imaging the expressiveness is overpowered by all the code necessary to make it performant. I'm thinking ctype libraries.
  - Spyro7 15y ago
    Well, if this is the code that is in question: http://code.google.com/p/multi-language-bench/source/browse/trunk/src/havlak/python/LoopTesterApp.py http://code.google.com/p/multi-language-bench/source/browse/... No, there are no ctype libraries in there.
- Spyro7 15y ago
  Actually, the python version has about 328 lines of code, while the scala version has about 216 lines of code. You should visit the Google project page and use something like cloc to analyze the files: http://code.google.com/p/multi-language-bench/ http://code.google.com/p/multi-language-bench/ http://cloc.sourceforge.net/ http://cloc.sourceforge.net/ I confess that I do not know a lot about Scala, but it looks like some of the functional aspects of the language allow for some savings in lines of code. If I had the time right now, I doubt that it would be too difficult to shrink the python version a bit. Just taking a glance at it, I don't see a single list comprehension throughout their code. I am sure there are some other language features that could probably have been better leveraged as well. Just out of curiosity, where did you get the 626 loc and the 297 loc from - I tried looking but I can't find it anywhere. Though that could just be a product of my lack of sleep right now.
  - cygwin98 15y ago
16s 15y ago
What is this InlinedVector<> they speak of in C++ tuning? Is that a Google specific container or is it in the std library (or some other library)?
- jongraehl 15y ago
  It probably holds small vectors directly inside the sizeof(InlinedVector<X>()) bytes, instead of indirecting. I think it's a pretty common technique.
mulander 15y ago
The paper uses wc -l (probably because sloccount doesn't handle go and scala?) to count the lines of code. Here is the output of sloccount: SLOC Directory SLOC-by-Language (Sorted) 595 java_pro java=595 591 java java=591 488 cpp cpp=488 328 python python=328 0 go (none) 0 go_pro (none) 0 scala (none) 0 scala_pro (none) generated using David A. Wheeler's 'SLOCCount'. Compare it to the paper. Benchmark wc -l C++ Dbg/Opt 850 Java 1068 Java Pro 1240 Scala 658 Scala Pro 297 Go 902 Go Pro 786
- igouy 15y ago
  Here are GZip minimally compressed source-code sizes after removal of comments and removal of duplicate whitespace characters - Benchmark GZ Bytes Factor (wc -l Factor from paper) Java Pro 5198 1.65x 1.9x Java 4403 1.40x 1.6x C++ 4229 1.34x 1.3x Go 3768 1.20x 1.4x Go Pro 3259 1.03x 1.2x Scala 3138 ==== === Python 2755 0.87x ???? Scala Pro 1929 0.61x 0.5x
- lbolla 15y ago
  This is the result using cloc: Bench files blank comment code cpp 4 145 236 501 go 5 153 239 546 go_pro 5 132 233 457 java 8 151 352 605 java_pro 8 157 351 772 python 2 110 204 334 scala 2 104 224 384 scala_pro 2 59 62 230
edabobojr 15y ago
The java version spends 75% of it's time gc? That smells fishy to me. edit: I just got to section VI of the paper, which does some gc tuning. In my playing, I also found about a 20% speedup by changing the HavlakLoopFinder.UnionFindMode class to being a statuc class.
- igouy 15y ago
  http://jeremymanson.blogspot.com/2011/06/scala-java-shootout.html http://jeremymanson.blogspot.com/2011/06/scala-java-shootout...

200 points by pgbovine 15y ago

LiveTheDream 15y ago

Loaded directly as a PDF for me.

rrrazdan 15y ago
The scribd link is an alternate.
- kristianp 15y ago
  A bit of cross-promotion, Scribd is a ycombinator company.
- manveru 15y ago
  At least on my galaxy tab using opera mobile, the scribd link simply tells me to download the PDF.

copper 15y ago

The most instructive part of this are the listings from the changelogs alongwith the resulting improved performance. I'm really enjoying the C++ tunings. I wish they did show the results with the google hashes, though.

btmorex 15y ago
So, the C++ performance numbers were with the "public" version? It wasn't clear whether they left the internal version out completely (besides the changelog) or simple didn't release the code. The published C++ numbers are already way better than any other language in terms of speed and memory use. It would be surprising if there was a substantially faster version.
- copper 15y ago
  The final list of changes is supposed to be with the internal version, and roughly seem to show a 20% change over and above what the basic std::map -> ext/hash_map replacement. I'm not sure if I should add the percentages, or multiply them along to get the final runtime, though that doesn't make much difference, really :) You're right that there may not be much improvement, but I've seen the google dense_hash_map do better sometimes than the basic SGI hash_map. Edit: the code repo has the unoptimized versions. Oh well, at least I can get the satisfaction of seeing the 30% myself :)
- igouy 15y ago
  afaict Yes, the C++ performance numbers were with the "public" version. afaict Yes, they left the internal version out completely (besides the changelog). Notice C++ Dbg has the same wc -l as C++ Opt.
  - copper 15y ago
    Isaac, do you think you could add this problem to the shootout? I believe (and am therefore almost certainly wrong) that there isn't any benchmark there that derives almost purely from compiler theory.

pgbovine 15y ago

summary: author implemented a sophisticated compiler algorithm (loop header recognition) in a straightforward canonical way in C++, Java, Scala, and Go, measured performance/memory usage, and then (the most interesting part of the paper) he had colleagues who were experts in each respective language write highly-tuned optimized versions and reported on what it took to optimize in each language.

CCs 15y ago
So C++ wins the performance test by large margin, Scala follows (3.6x), then Java 64bit (5.8x), Go (7x) and finally Java 32bit (12.6x). Scala without any optimization is still better than Java with all the ninja-skills black magic applied. Scala optimized is pretty good - "just" 2.5x slower than C++
- arron61 15y ago
  Actually it's odd. They say that in the Java tuning, they made some simple optimization that got it on par to the original C++ one but then they refused to do anything further; noting instead the same C++ optimization would have applied in Java as well. Does that mean the java version would have been just as fast? It seems in general though that Java is just as fast as C++ unless you are an ultimate C++ expert.
  - johnm 15y ago
    Indeed, this is ridiculous and makes their final conclusions weaker.
- wtracy 15y ago
  Maybe I'm missing something, but I'm amazed to see Scala beating out Java.
  - gaius 15y ago
    Why? Remember that at this point in history, Java-the-language and Java-the-VM are related in name only.

skimbrel 15y ago

Wow. Looks like C++ is still one of the performance kings among current programming languages. Can't say I enjoy using it though. It also looks like Go and Scala have a long way to go on compiler optimization. Not surprising, since they're young languages compared to C++ and Java, and it's a decent amount of work to optimize the higher-order constructs they provide.

latch 15y ago
Languages that come from an era where total memory was measured in kbytes, cpus in megahertz and pointer arithmetic was as common as if statements, will always be performance kings. It'll take a pretty significant shift to change this. Something like 100+ core cpus each running slow, might be such a shift. Being surprised that C++ is faster than newer languages which provide all types of abstraction is like being surprise assembly is faster than C++.
- anonymous246 15y ago
  You're mostly correct, but the code they wrote DID use fancy abstractions like hash tables, templates etc. C++ delivered on abstraction WITHOUT a performance penalty, which is what the C++ designers have claimed it could do since forever. Nice to see it verified again. I have no doubt that the C++0x syntactic sugar enhancements would have have also delivered the same high performance.
  - alexkcd 15y ago
    Indeed. I write a lot of Objective-C code, and still drop into C++ for the CPU heavy tasks. Swapping out NS* containers for std::* alone gives a fairly high boost in performance. All the C++ hate is blown way out of proportion. C++ is a great tool when you need performance. Tuning C++ isn't that difficult either. Contrary to what a commenter said above, steering clear of C++ at all costs is not what you should be doing. You should know when and how to use C++. It's useful more often than many people here seem to think. Especially if you develop for mobile devices.

scottjad 15y ago

Speaking of Scala Pro: "It should be noted that this version performs algorithmic improvements as well, and is therefore not directly comparable to the other Pro versions."

enneff 15y ago
"It should be noted"?! What an understatement! What's the point of going to the trouble of writing a paper if you're not even going to make your comparisons fair? Crazy.
- Djehngo 15y ago
  Because the initial comparison was fair.
  - igouy 15y ago
    Really? "... the benchmark also used a HashMap<Type, Integer> when it should have just stored a primitive int in Type; it also used a LinkedList where it should have used an ArrayDeque." http://jeremymanson.blogspot.com/2011/06/scala-java-shootout.html http://jeremymanson.blogspot.com/2011/06/scala-java-shootout...

choxi 15y ago

This is cool but I think it's important to remember that they're comparing algorithmic benchmarks but modern software has many different bounds besides for raw, algorithmic processing. For example, I think I/O and database interactions are the bottleneck for most web apps. I really liked the way this article broke it down: http://pl.atyp.us/wordpress/?p=2947 http://pl.atyp.us/wordpress/?p=2947

cdavid 15y ago
Given the languages they are using, I think it is fair to say that they are interested in system-programming kind of tasks, which for google means the stacks below what you normally use for web programming (i.e. I would expect google to have their own web servers, or at least heavily customized from existing ones, etc...). For system programming, "language speed" matters much more (one definition I like for system programming, seen on lambda ultimate: "programming where both algorithm complexity's constant and memory representation matter"). Of course, in most cases, you're right. Given an infinite pool of very good programmers and infinite time, C will almost always be faster than most other languages. But in reality, you have a very limited of maybe not so good programmers, and the project needed to be done yesterday :)
gtani 15y ago
Thx for that. Java and scala-specifically, it's a pretty high-dimension space. Background: http://www.elis.ugent.be/JavaStats http://www.elis.ugent.be/JavaStats http://www.bestinclass.dk/index.clj/2010/02/benchmarking-jvm-languages.html http://www.bestinclass.dk/index.clj/2010/02/benchmarking-jvm... http://isthisclojure.blogspot.com/2011/02/benchmark-clojurejvm-5-degrees-of.html http://isthisclojure.blogspot.com/2011/02/benchmark-clojurej... https://docs.google.com/present/view?id=0AS8emH3-FLt3ZGRtbWJyOGdfMTFmcDZkcTk2cw&hl=en https://docs.google.com/present/view?id=0AS8emH3-FLt3ZGRtbWJ... Examples of benchmarks gone awry: http://stackoverflow.com/questions/6146182/why-is-my-scala-code-running-slow http://stackoverflow.com/questions/6146182/why-is-my-scala-c... http://groups.google.com/group/scala-language/browse_thread/thread/94740a10205dddd2/ http://groups.google.com/group/scala-language/browse_thread/...
- igouy 15y ago

tectonic 15y ago

Their conclusions: "We ﬁnd that in regards to performance, C++ wins out by a large margin. However, it also required the most extensive tuning efforts, many of which were done at a level of sophistication that would not be available to the average programmer. Scala concise notation and powerful language features allowed for the best optimization of code complexity. The Java version was probably the simplest to implement, but the hardest to analyze for performance. Speciﬁcally the effects around garbage collection were complicated and very hard to tune. Since Scala runs on the JVM, it has the same issues. Go offers interesting language features, which also allow for a concise and standardized notation. The compilers for this language are still immature, which reﬂects in both performance and binary sizes."

gaius 15y ago
It's not that you have to tune C++ it's that you can.
- keeperofdakeys 15y ago
  Untuned C++ will probably be slower then even tuned python, I can tune any language, but C++ (although less so then something like C) requires more tuning to get good performance. The performance is what we care about here.
  - Raphael_Amiard 15y ago
    > Untuned C++ will probably be slower then even tuned python [citation needed]
    - keeperofdakeys 15y ago
      By 'untuned' I'm also including using the wrong algorithm. For example, the using the usual recursive definition for the Fibonacci sequence in C++ as opposed to the loop or n^2 version of recursive Fibonacci in python. The python version will be faster in this case. It is probably true however that any decent programmer will churn out C++ that is faster than python for cpu intensive tasks most of the time.

ivan_ah 15y ago

link to the code: http://code.google.com/p/multi-language-bench/ http://code.google.com/p/multi-language-bench/ This is a great resource to learn the different languages. (I had never seen C++ template programming used in practice before) Challenge to the pythonistas: rewrite the python version so it uses the C++ code under the hood using scipy weave

igouy 15y ago
http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/
lbolla 15y ago
Templates are not used in the C++ benchmark.

jbpritts 15y ago

C++ code, when sophisticated template meta programming techniques are used, can be as fast as tuned C code, and can beat any language in terms of speed with the exception of hand-tuned assembly. However, techniques like expression templates, automatic loop unrolling, template specialization, and static polymorphism require a VERY high degree of sophistication from the programmer. In some sense, the programmer is forced to be a compiler. There are some individuals that, despite this demand, can be highly productive. The others wallow in complicated syntax, horrible type errors, and difficult to trace run-time errors. Unless speed is absolutely essential, C++ should be avoided at all costs.

pnathan 15y ago
C++ TMP is quite painful to the uninitiated. Outside of that, C++ is not that bad, especially as you build up your project-specific abstractions.

onan_barbarian 15y ago

Using a "older Pentium IV workstation". Older than what? Older than Grandpa? Older than dirt? Older than the dinosaurs? Why? Pentium IV - really? Let's see - we've had Core, Core 2, Nehalem and Sandy Bridge since then, not to mention 3 in-between process shrink versions of the same architecture. Perhaps we could pass the hat around so that Google could afford a Sandy Bridge workstation and discover what these interesting results look like on an architecture that dates from some point in this decade - or at least, at some point in the previous decade. This stuff actually, really, truly makes a difference. And not necessarily in favor or against any particular one of these languages...

acqq 15y ago
Imagine: there is a paper discussing the techniques of weight lifting. Your complaint is that there are newer models of weights than those they lifted.
- jnhnum1 15y ago
  Newer processors don't just increase the performance of all programs equally. They have things like improved branch prediction, cache prefetching, better pipelining, and different cache sizes which can make a lot of performance optimizations that you get from C / C++ less relevant.
  - acqq 15y ago
    Except from the "most optimized C++ code" you have all sources of all benchmarks. Please try to run them on any newer x86/64 based processor and show us that the article conclusions don't hold. I'm not holding my breath though.
    - scott_s 15y ago
      I agree that it probably won't make a difference. But good experiments remove as many of those probablys as possible.

ksolanki 15y ago

We find that in regards to performance, C++ wins out by a large margin. However, it also required the most extensive tuning efforts, many of which were done at a level of sophistication that would not be available to the average programmer. I am a fan of C++, so I liked the first part (that C++ wins, which is not really surprising). However it was not clear if the sophistication they talk about is regarding the Google's internal data-structures or the list of optimizations listed in Section VI-D? Many of these optimizations are not big surprises to a C++ programmer (where possible use hash_map, vector instead of list, initialize data structure outside of loop and try to reuse, and so on). Overall, I do greatly appreciate the effort and their sharing the results.

bad_user 15y ago
The problem with possible optimizations and tunning is that it requires constant effort. Projects often have serious deadlines and you simply can't constantly think about memory or allocation footprint and still do a good job in the amount of time you have, unless you're well above average.
- davidtgoldblatt 15y ago
  Most of the optimizations that got them big wins in C++ (using hash maps instead of tree-based maps, dynamic arrays instead of linked lists, .empty() rather than .size() > 0 on a linked list) are "library knowledge" issues more than memory / allocation ones. In fact, looking at the list of optimizations on page 9, the only one that involved a low-level understanding of the machine and resulted in a significant improvement was (unless I missed something) the use of InlinedVector (and this is only barely such a case - lots of C++ codebases have a similar class).
  - acqq 15y ago
    Exactly. Using hash maps and vectors are really elementary choices. Can you consider understanding enough in "systems programming" (see other comments for definition) without knowing that hash_map is implemented using hash tables and map using a tree? And that therefore adding elements to the map would incur rebalancing costs which hash_map wouldn't, as soon as you can estimate the number of elements to be processed. I consider it more a "meta" than a "language" decision.

zohebv 15y ago

One of the interesting details is that Go Pro required 786 lines of code as opposed to 850 lines of C++ code and 297 lines of Scala Pro code. Regular Go code was 902 lines. It certainly doesn't look like Go scores any expressiveness points over C++.

cageface 15y ago
It may just be because I've been writing a ton of C++ lately, but to me the C++ data declarations are the most readable of the bunch. The stuff with iterators is much uglier, of course, but so far I like the syntax of C++ a lot better than that of Go.
- stephen_g 15y ago
  Yep, iterators are one of the most annoying things in C++. That's why I love range for in C++0x - instead of writing this, for example: for(std::vector<std::string>::iterator it = vec.begin(); it != vec.end(); it++) you can just do: for(std::string x : vec) So awesome!
  - dzorz 15y ago
    Even better: for (auto &x : vec) { }
    - cageface 15y ago
      I can't wait until I can start using this in XCode.
      - dzorz 15y ago
        If you compile recent clang, it supports range based for loops. Using https://github.com/Rip-Rip/clang_complete https://github.com/Rip-Rip/clang_complete you can use smart context-aware completion with vim/emacs. For example if you had this code: vector<string> vec; for (auto &x : vec) { x and then if you typed period (.), it would show members from std::string. I'd say it works better than Intellisense (it's very precise and you use the same parser for code completion and final compilation).

nimrody 15y ago

Keep in mind that since C++ is usually used for performance critical code, its compilers are often highly tuned and produce good code (see Intel's ICC for number crunching code). A less popular language - or one that is usually used for less-critical code - will suffer. C++ wasn't always so fast. See Kernighan and Pike, "The Practice of Programming" for a simple example where Java surprisingly beats C++.

cageface 15y ago
True, but this is one of the reasons people choose to code in C++. The quality of tools & common libraries is as much a factor in language selection as the syntax and semantics of the language itself.
- nimrody 15y ago
  You mean - "forced to use C++" :) I would love to use Digital Mars D or Scala for scientific applications if they had compilers/runtimes as good as C++. Perhaps one day the LLVM backend would be useful as a universal backend for many less popular languages.

aria 15y ago

Really shocked they would place numbers in a table like that when the 'Scala Pro' version is doing something algorithmically smarter. They note in the paper this distinction, but it should be marked in the table, because I'm sure 99% of people don't see that. Sloppy!

sili 15y ago
That, plus the fact that they refused to do same optimizations to Java as they did to C++ for some reason.

xedarius 15y ago

I wonder what JVM they used, as significant performance differences exist between Hotspot and JRockit. I also wonder what compiler they used, again Microsoft compiler will yield different results compared to gcc.

igouy 15y ago
Here's the Makefile - gcc http://code.google.com/p/multi-language-bench/source/browse/trunk/src/havlak/cpp/Makefile http://code.google.com/p/multi-language-bench/source/browse/...

mhd 15y ago

It probably won't take long until someone does the same exercise with other languages. Maybe they should add it to the shootout. Can't wait to see Modula-3's numbers…

igouy 15y ago
> Maybe they should... "they"? The space monsters who run the universe?

thesz 15y ago

What I'd like to see there is Haskell version and how all those implementations speed up when running on several cores. This is much more interesting considering our current reality. I once gained 10% speed up on two cores by changing just two lines of our optimization tool written on Haskell.

olifante 15y ago

TLDR: C++ fastest but hardest, Scala fast and elegant, Java simple but opaque, Go immature.

javanix 15y ago

This paper served as a great introduction to Go programming for me. I've been trying to work through my own basic program in Go for the past few weeks and their overview of the language features used cleared up a lot of loose ends for me.

scott_s 15y ago
Having the same algorithm implemented in several other languages helps, too. It enables you to look at the implementation in the language you're most comfortable with, then look at one of the other implementations and think "Oh, so that's how that's done in this language."
- igouy 15y ago
  http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/ Although you shouldn't necessarily think "Oh, so that's how that's done in this language."
supersillyus 15y ago
Keep in mind, though, that the Go code shown there is not a good example of how idiomatic Go is written.

DonnyV 15y ago

Who said that when you release a paper that it needs to look like it came from a text book. I couldn't get past the first page because I felt like I was back in high school.

sparky 15y ago
>Who said that... The conference. From http://days2011.scala-lang.org/node/91 http://days2011.scala-lang.org/node/91 : Submissions must be in English and at most 12 pages total length in the standard ACM SIGPLAN two-column conference format (10pt). Nearly all academic papers in computer science and engineering look like this. Not to say they've converged on an optimal format, but it wasn't exactly a creative decision on the part of these authors.

cygwin98 15y ago

I was surprised by the fact that the Python version has 626 lines of code, compared to 297 to the Scala Pro. As Python is supposed to be more expressive, I was expecting something like 150ish LOC for Python. Edit: The Python version can be found at http://code.google.com/p/multi-language-bench/source/browse/trunk/src/havlak/python/LoopTesterApp.py http://code.google.com/p/multi-language-bench/source/browse/...

euroclydon 15y ago
I would imaging the expressiveness is overpowered by all the code necessary to make it performant. I'm thinking ctype libraries.
- Spyro7 15y ago
  Well, if this is the code that is in question: http://code.google.com/p/multi-language-bench/source/browse/trunk/src/havlak/python/LoopTesterApp.py http://code.google.com/p/multi-language-bench/source/browse/... No, there are no ctype libraries in there.
Spyro7 15y ago
Actually, the python version has about 328 lines of code, while the scala version has about 216 lines of code. You should visit the Google project page and use something like cloc to analyze the files: http://code.google.com/p/multi-language-bench/ http://code.google.com/p/multi-language-bench/ http://cloc.sourceforge.net/ http://cloc.sourceforge.net/ I confess that I do not know a lot about Scala, but it looks like some of the functional aspects of the language allow for some savings in lines of code. If I had the time right now, I doubt that it would be too difficult to shrink the python version a bit. Just taking a glance at it, I don't see a single list comprehension throughout their code. I am sure there are some other language features that could probably have been better leveraged as well. Just out of curiosity, where did you get the 626 loc and the 297 loc from - I tried looking but I can't find it anywhere. Though that could just be a product of my lack of sleep right now.
- cygwin98 15y ago

16s 15y ago

What is this InlinedVector<> they speak of in C++ tuning? Is that a Google specific container or is it in the std library (or some other library)?

jongraehl 15y ago
It probably holds small vectors directly inside the sizeof(InlinedVector<X>()) bytes, instead of indirecting. I think it's a pretty common technique.

mulander 15y ago

The paper uses wc -l (probably because sloccount doesn't handle go and scala?) to count the lines of code. Here is the output of sloccount: SLOC Directory SLOC-by-Language (Sorted) 595 java_pro java=595 591 java java=591 488 cpp cpp=488 328 python python=328 0 go (none) 0 go_pro (none) 0 scala (none) 0 scala_pro (none) generated using David A. Wheeler's 'SLOCCount'. Compare it to the paper. Benchmark wc -l C++ Dbg/Opt 850 Java 1068 Java Pro 1240 Scala 658 Scala Pro 297 Go 902 Go Pro 786

igouy 15y ago
Here are GZip minimally compressed source-code sizes after removal of comments and removal of duplicate whitespace characters - Benchmark GZ Bytes Factor (wc -l Factor from paper) Java Pro 5198 1.65x 1.9x Java 4403 1.40x 1.6x C++ 4229 1.34x 1.3x Go 3768 1.20x 1.4x Go Pro 3259 1.03x 1.2x Scala 3138 ==== === Python 2755 0.87x ???? Scala Pro 1929 0.61x 0.5x
lbolla 15y ago
This is the result using cloc: Bench files blank comment code cpp 4 145 236 501 go 5 153 239 546 go_pro 5 132 233 457 java 8 151 352 605 java_pro 8 157 351 772 python 2 110 204 334 scala 2 104 224 384 scala_pro 2 59 62 230

edabobojr 15y ago

The java version spends 75% of it's time gc? That smells fishy to me. edit: I just got to section VI of the paper, which does some gc tuning. In my playing, I also found about a 20% speedup by changing the HavlakLoopFinder.UnionFindMode class to being a statuc class.

igouy 15y ago
http://jeremymanson.blogspot.com/2011/06/scala-java-shootout.html http://jeremymanson.blogspot.com/2011/06/scala-java-shootout...