3 ms·

I find microbenchmarks mostly useless. There are just too many variables in play. Also they tend to focus on the developer who optimizes code, not the journeym

I find microbenchmarks mostly useless. There are just too many variables in play. Also they tend to focus on the developer who optimizes code, not the journeyman developer in your company who actually writes code. C or Fortran looks fantastic on benchmarks, but if you forced all your developers to use C I doubt you'd see a similar outcome. They seem to push developers in the direction of coming up with simple rules of thumb like "Go is 2x slower than C" or "Python is 10x slower than C" which is complete nonsense. It really matters what the code is doing. In particular for server programs developers come to these conclusions without considering the multi-core environment they're working in. ie, redis is a really fast key-value store that will use 1 core on a 32 core machine. And as it turns out all the tips and tricks to improving performance in these cases works in all the other languages too. So just what are we to conclude from them? I suppose a nice simple math benchmark can demonstrate that LLVM or GCC is better at optimizing code than Go?

glangdale 8y ago
I have a lot of sympathy for your viewpoint, as I think it's abundantly clear that there are major productivity differences between languages. I doubt that anyone would be seriously running around doing an end-zone dance because C beats, say, Python. "Yeah, suck that, Python losers!". However, I think the fairly simple benchmarks where there looks to be a rough 1:1 correspondence between the structures of some similar Rust, Fortran, C/C++, Go code might demonstrate exactly the point you suggested (facetiously? I'm not quite sure how you are suggesting we compare compiler frameworks to a language, especially a language that can use both frameworks with alternate backends...) I don't know why this is controversial. The Go optimizer wasn't written primarily with utmost code quality in mind (it's pretty much a lift of the Plan 9 C compiler, which I worked with in 1994). It's designed to produce code quickly. It's also not rocket science that GC can be expensive despite years of claims to the contrary (it's been 'about to be the same or faster as manual memory management' since the mid-90s at least). Leaving aside the wacky benchmarks (binary trees and regex) the results seem pretty consistent with this. These rules of thumb are OK for what they are worth (not very much). Eventually we do have to think about how much performance can be extracted from a language if we really need it. The irritating thing about this whole benchmarking problem is that there is no natural population of programs to draw from, and there is constant thumb-on-the-scale cheating (who can intrinsic the hardest for Grandpa's ancient Core 2 duo, but no later?) - agreed. Essentially we're seeing a real-life version of the saying "the plural of anecdote is not 'data'". Nonetheless... it feels like we can say some programming languages are faster than others.
naasking 8y ago
I think you'll find the performance per gzipped line of code is a decently robust metric for a performance/productivity tradeoff, although it may have a wide variance. Performance optimizations tend to involve a lot of specialization, which means more code, which makes this ratio worse. The gzipped size is a proxy for the minimum description length/Kolmogorov complexity, which measures language expressiveness.
- vanderZwan 8y ago
  APL and co kind of undermine that metric, wouldn't you say?
  - naasking 8y ago
    Kolmogorov complexity is "ideal compression", in that it bundles the axioms and the expression to reproduce the desired output using those axioms. So it really is a perfect measure for this type of thing. Gzip is an approximation of this metric of course, and the only flaw in the benchmark suite is that it should really include the size of the runtime which provides the programming language's axioms. APL is also a good example of why it's important to include various programs in the suite for a valid comparison. Some APL programs are longer than more expressive equivalents when APL is ill-suited to the problem, and once you include the axioms.
    - vanderZwan 8y ago
      I know what Kolmogorov complexity is, I was trying to get at something else: removing boilerplate is one thing, but the other extreme isn't all that helpful either. When discussing information theory, one should always remember that white noise has the maximum amount of information, so the theoretical best scoring programming language of your measurement would be indistinguishable from white noise. Hardly productive, right? One of the key things about higher programming languages is that they add structure and patterns - in information theory terms: redundancy - to make it easier for programmers to follow what is going on. I mentioned APL because programs in it usually consist of a handful of symbols with very little repetition. The symbols/performance ratio is really good for APL. Or to be even more on the nose: think of code golfing languages[0]. [0] https://en.wikipedia.org/wiki/Code_golf#Dedicated_golfing_languages https://en.wikipedia.org/wiki/Code_golf#Dedicated_golfing_la...
rwj 8y ago
I always find this argument really strange. How is making a complicated decision (which language is best for this task) easier using less information (the benchmarks are only one facet of the decision)?
- jcranmer 8y ago
  The name of the website used to be the language shootout; it was subsequently changed to be called the benchmarks game to emphasize that its not a rigorous comparison between languages. Benchmarking is hard. You have to identify workloads that are realistic and representative, large enough to be sure you're hitting the steady-state performance yet small enough you can run them on a regular basis. And, of course, you have to make sure that you're not embarking on a journey of overfitting your benchmark suite. Microbenchmarks, where you look at a single kernel in isolation, are extremely hard to get right, especially because it can be easy to accidentally test how fast your computer can do nothing. Benchmarks can be incredibly useful, but an uncurated, gamified set of benchmarks are not going to be in the category of useful benchmarks. There is a risk that by turning a bad benchmark into a programming mantra, you can seriously harm the performance of your code in the future (Duff's device and object pooling are two "optimization" strategies that are now more harmful to your performance than not doing them).
  - glangdale 8y ago
    +1 for Duff's device. It's really not good now (as compared to the unrolled loop plus a peel) approach, as all those in-edges to a loop all through the body paralyze the compiler's ability to make good choices. An anecdote I often share is that we had a hotshit performance programmer who often enclosed his simple C versions of the code with: #ifdef I_DONT_CARE_ABOUT_PERFORMANCE The punchline was defining this macro and getting the C version usually improved performance over the giant festival of overly-Pentium-4-focused inline asm.
  - galangalalgol 8y ago
    Do the results of the game differ from your experience when you squint at them from a distance? They line up with mine pretty accurately, but I only have experience with a handful of those languages. Python is horrifically slow even with numpy in my experience. About 11x slower than native for an actual project. Java is 2-3x slower on everything I've used that had native and java versions of the same software. c# has always felt faster than these benchmarks indicate though. I'm interested in other peoples gut feels on this who have experience with go and maybe some who are better at using python? I really wanted to like it.

glangdale 8y ago

I have a lot of sympathy for your viewpoint, as I think it's abundantly clear that there are major productivity differences between languages. I doubt that anyone would be seriously running around doing an end-zone dance because C beats, say, Python. "Yeah, suck that, Python losers!". However, I think the fairly simple benchmarks where there looks to be a rough 1:1 correspondence between the structures of some similar Rust, Fortran, C/C++, Go code might demonstrate exactly the point you suggested (facetiously? I'm not quite sure how you are suggesting we compare compiler frameworks to a language, especially a language that can use both frameworks with alternate backends...) I don't know why this is controversial. The Go optimizer wasn't written primarily with utmost code quality in mind (it's pretty much a lift of the Plan 9 C compiler, which I worked with in 1994). It's designed to produce code quickly. It's also not rocket science that GC can be expensive despite years of claims to the contrary (it's been 'about to be the same or faster as manual memory management' since the mid-90s at least). Leaving aside the wacky benchmarks (binary trees and regex) the results seem pretty consistent with this. These rules of thumb are OK for what they are worth (not very much). Eventually we do have to think about how much performance can be extracted from a language if we really need it. The irritating thing about this whole benchmarking problem is that there is no natural population of programs to draw from, and there is constant thumb-on-the-scale cheating (who can intrinsic the hardest for Grandpa's ancient Core 2 duo, but no later?) - agreed. Essentially we're seeing a real-life version of the saying "the plural of anecdote is not 'data'". Nonetheless... it feels like we can say some programming languages are faster than others.

naasking 8y ago

I think you'll find the performance per gzipped line of code is a decently robust metric for a performance/productivity tradeoff, although it may have a wide variance. Performance optimizations tend to involve a lot of specialization, which means more code, which makes this ratio worse. The gzipped size is a proxy for the minimum description length/Kolmogorov complexity, which measures language expressiveness.

vanderZwan 8y ago
APL and co kind of undermine that metric, wouldn't you say?
- naasking 8y ago
  Kolmogorov complexity is "ideal compression", in that it bundles the axioms and the expression to reproduce the desired output using those axioms. So it really is a perfect measure for this type of thing. Gzip is an approximation of this metric of course, and the only flaw in the benchmark suite is that it should really include the size of the runtime which provides the programming language's axioms. APL is also a good example of why it's important to include various programs in the suite for a valid comparison. Some APL programs are longer than more expressive equivalents when APL is ill-suited to the problem, and once you include the axioms.
  - vanderZwan 8y ago
    I know what Kolmogorov complexity is, I was trying to get at something else: removing boilerplate is one thing, but the other extreme isn't all that helpful either. When discussing information theory, one should always remember that white noise has the maximum amount of information, so the theoretical best scoring programming language of your measurement would be indistinguishable from white noise. Hardly productive, right? One of the key things about higher programming languages is that they add structure and patterns - in information theory terms: redundancy - to make it easier for programmers to follow what is going on. I mentioned APL because programs in it usually consist of a handful of symbols with very little repetition. The symbols/performance ratio is really good for APL. Or to be even more on the nose: think of code golfing languages[0]. [0] https://en.wikipedia.org/wiki/Code_golf#Dedicated_golfing_languages https://en.wikipedia.org/wiki/Code_golf#Dedicated_golfing_la...

rwj 8y ago

I always find this argument really strange. How is making a complicated decision (which language is best for this task) easier using less information (the benchmarks are only one facet of the decision)?

jcranmer 8y ago
The name of the website used to be the language shootout; it was subsequently changed to be called the benchmarks game to emphasize that its not a rigorous comparison between languages. Benchmarking is hard. You have to identify workloads that are realistic and representative, large enough to be sure you're hitting the steady-state performance yet small enough you can run them on a regular basis. And, of course, you have to make sure that you're not embarking on a journey of overfitting your benchmark suite. Microbenchmarks, where you look at a single kernel in isolation, are extremely hard to get right, especially because it can be easy to accidentally test how fast your computer can do nothing. Benchmarks can be incredibly useful, but an uncurated, gamified set of benchmarks are not going to be in the category of useful benchmarks. There is a risk that by turning a bad benchmark into a programming mantra, you can seriously harm the performance of your code in the future (Duff's device and object pooling are two "optimization" strategies that are now more harmful to your performance than not doing them).
- glangdale 8y ago
  +1 for Duff's device. It's really not good now (as compared to the unrolled loop plus a peel) approach, as all those in-edges to a loop all through the body paralyze the compiler's ability to make good choices. An anecdote I often share is that we had a hotshit performance programmer who often enclosed his simple C versions of the code with: #ifdef I_DONT_CARE_ABOUT_PERFORMANCE The punchline was defining this macro and getting the C version usually improved performance over the giant festival of overly-Pentium-4-focused inline asm.
- galangalalgol 8y ago
  Do the results of the game differ from your experience when you squint at them from a distance? They line up with mine pretty accurately, but I only have experience with a handful of those languages. Python is horrifically slow even with numpy in my experience. About 11x slower than native for an actual project. Java is 2-3x slower on everything I've used that had native and java versions of the same software. c# has always felt faster than these benchmarks indicate though. I'm interested in other peoples gut feels on this who have experience with go and maybe some who are better at using python? I really wanted to like it.