3 ms·

I always find this argument really strange. How is making a complicated decision (which language is best for this task) easier using less information (the benc

by rwj 8y ago

I always find this argument really strange. How is making a complicated decision (which language is best for this task) easier using less information (the benchmarks are only one facet of the decision)?

jcranmer 8y ago
The name of the website used to be the language shootout; it was subsequently changed to be called the benchmarks game to emphasize that its not a rigorous comparison between languages. Benchmarking is hard. You have to identify workloads that are realistic and representative, large enough to be sure you're hitting the steady-state performance yet small enough you can run them on a regular basis. And, of course, you have to make sure that you're not embarking on a journey of overfitting your benchmark suite. Microbenchmarks, where you look at a single kernel in isolation, are extremely hard to get right, especially because it can be easy to accidentally test how fast your computer can do nothing. Benchmarks can be incredibly useful, but an uncurated, gamified set of benchmarks are not going to be in the category of useful benchmarks. There is a risk that by turning a bad benchmark into a programming mantra, you can seriously harm the performance of your code in the future (Duff's device and object pooling are two "optimization" strategies that are now more harmful to your performance than not doing them).
- glangdale 8y ago
  +1 for Duff's device. It's really not good now (as compared to the unrolled loop plus a peel) approach, as all those in-edges to a loop all through the body paralyze the compiler's ability to make good choices. An anecdote I often share is that we had a hotshit performance programmer who often enclosed his simple C versions of the code with: #ifdef I_DONT_CARE_ABOUT_PERFORMANCE The punchline was defining this macro and getting the C version usually improved performance over the giant festival of overly-Pentium-4-focused inline asm.
- galangalalgol 8y ago
  Do the results of the game differ from your experience when you squint at them from a distance? They line up with mine pretty accurately, but I only have experience with a handful of those languages. Python is horrifically slow even with numpy in my experience. About 11x slower than native for an actual project. Java is 2-3x slower on everything I've used that had native and java versions of the same software. c# has always felt faster than these benchmarks indicate though. I'm interested in other peoples gut feels on this who have experience with go and maybe some who are better at using python? I really wanted to like it.
- igouy 8y ago
  nit-pick -- The name was changed after the Virginia Tech shooting, because the search results were mostly about gun deaths and porn. Not a bright happy start to my day. Not "to emphasize" just about the benchmarks game, but more generally about cross-language comparisons -- and I had to pick a name ;-)

by rwj 8y ago

jcranmer 8y ago

The name of the website used to be the language shootout; it was subsequently changed to be called the benchmarks game to emphasize that its not a rigorous comparison between languages. Benchmarking is hard. You have to identify workloads that are realistic and representative, large enough to be sure you're hitting the steady-state performance yet small enough you can run them on a regular basis. And, of course, you have to make sure that you're not embarking on a journey of overfitting your benchmark suite. Microbenchmarks, where you look at a single kernel in isolation, are extremely hard to get right, especially because it can be easy to accidentally test how fast your computer can do nothing. Benchmarks can be incredibly useful, but an uncurated, gamified set of benchmarks are not going to be in the category of useful benchmarks. There is a risk that by turning a bad benchmark into a programming mantra, you can seriously harm the performance of your code in the future (Duff's device and object pooling are two "optimization" strategies that are now more harmful to your performance than not doing them).

glangdale 8y ago
+1 for Duff's device. It's really not good now (as compared to the unrolled loop plus a peel) approach, as all those in-edges to a loop all through the body paralyze the compiler's ability to make good choices. An anecdote I often share is that we had a hotshit performance programmer who often enclosed his simple C versions of the code with: #ifdef I_DONT_CARE_ABOUT_PERFORMANCE The punchline was defining this macro and getting the C version usually improved performance over the giant festival of overly-Pentium-4-focused inline asm.
galangalalgol 8y ago
Do the results of the game differ from your experience when you squint at them from a distance? They line up with mine pretty accurately, but I only have experience with a handful of those languages. Python is horrifically slow even with numpy in my experience. About 11x slower than native for an actual project. Java is 2-3x slower on everything I've used that had native and java versions of the same software. c# has always felt faster than these benchmarks indicate though. I'm interested in other peoples gut feels on this who have experience with go and maybe some who are better at using python? I really wanted to like it.
igouy 8y ago
nit-pick -- The name was changed after the Virginia Tech shooting, because the search results were mostly about gun deaths and porn. Not a bright happy start to my day. Not "to emphasize" just about the benchmarks game, but more generally about cross-language comparisons -- and I had to pick a name ;-)