5 ms·

I can't believe they used the Computer Language Benchmarks Game. Those benchmarks don't reflect real-world workloads at all, and the contest has fairly arbitrar

I can't believe they used the Computer Language Benchmarks Game. Those benchmarks don't reflect real-world workloads at all, and the contest has fairly arbitrary rules about implementations and widely differing implementation quality between languages. This should have been rejected by peer review.

igouy 9y ago
> Those benchmarks don't reflect real-world workloads at all… That's a very definite claim, for which you provide no supporting evidence ;-)
- oconnor663 9y ago
  Maybe we could get away with a couple smaller claims: - It would be weird to expect programs specifically optimized for a performance benchmark, to also be optimal for energy usage or memory. Maybe less weird for energy, if runtime is the biggest factor in how much energy gets used. But I'd expect there to be huge tradeoffs between runtime and memory usage, once we start really optimizing for memory. - Running the Benchmarks Game has meant doing a lot of work to figure out what the relationship between "optimal" and "representative" is, and what the rules of the game should be as a result. Even if we take it for granted that we've answered those questions in a way that gives us useful benchmarks for performance, it would be weird to expect the answers to be exactly the same in benchmarks for energy or memory.
  - jackmott 9y ago
    performance usually == energy efficiency on modern cpus. race to shutdown.
    - igouy 9y ago
      Does your comment agree or disagree with what they measure?
      - jackmott 9y ago
        The implementations in the benchmark game are all targeting performance. This will almost always also be the most energy efficient thing on modern cpus. It will not always be the most memory efficient way. I neither agree or disagree with what they are measuring, its just data. I would not look at this if I was trying to get an idea of what languages are most memory efficient, maybe.
        igouy 9y ago
        I wondered if you'd looked at all ;-)
  - igouy 9y ago
    Maybe we should look at "4 Threats to Validity" section of their paper and ask whether they have really done enough to suggest that their results should be considered general rather than specific.
  - yjftsjthsd-h 9y ago
    Perhaps I'm over simplifying to the point of making mistakes, but intuitively I would expect that performance and Energy Efficiency are strongly correlated, because I assume that each instruction executed takes a certain amount of time and a certain amount of energy, and that this skills roughly linearly. That is, executing n instructions takes A * n time and B * n energy, so of course making it finish sooner means less instructions means less energy.
- rbehrends 9y ago
  One of the worst offenders is probably the binary-trees benchmark. 1. It features an atypically high allocation rate compared to real-world programs. 2. As a synthetic micro benchmark that only ever allocates objects of one size, it makes pool allocators look disproportionately good. But in real-world applications with varying object sizes, overuse of pool allocators creates a serious risk for fragmentation. 3. The rules allow programs that use manual memory management to use pretty much any pool allocator library that they can get their hands on, but forbid garbage-collected languages from adjusting their GC parameters (last I checked, you could easily improve the performance of OCaml and Dart for that benchmark by a factor of 2-3 simply by adjusting GC settings, especially the size of the minor heap).
  - igouy 9y ago
    > The rules allow programs that use manual memory management to… And still there's outrage that programs with allocators custom-written for binary-trees are rejected :-)
  - kllrnohj 9y ago
    > It features an atypically high allocation rate compared to real-world programs. Do you have any supporting evidence for this? Real-world programs can be pretty allocation heavy, and if anything high allocation rates would "unfairly" benefit GC languages since native programs are banned from using arena allocators. > As a synthetic micro benchmark that only ever allocates objects of one size, it makes pool allocators look disproportionately good. But in real-world applications with varying object size, overuse of pool allocators is a serious risk for fragmentation. Huh? It's so incredibly common in the real-world for a structure to be of a single type that pretty much every language makes that first-class supported via templates. And since they are single-size there's no risk of fragmentation at all. Indeed object pools are typically how you avoid fragmentation, they are not things that cause it. So I'm not sure what you're trying to get at with that comment. > The rules allow programs that use manual memory management to use pretty much any pool allocator library that they can get their hand on, but forbid garbage-collected languages from adjusting their GC parameters (last I checked, you could easily improve the performance of OCaml and Dart for that benchmark by a factor of 2-3 simply by adjusting GC settings, especially the size of the nursery). Last I checked if I write a Dart library I can't adjust the GC settings in my library to make my library run better, either, but if I ship a C++ library I can absolutely use a pool allocator to make my library run better (and indeed this is exactly what real libraries do) So I'd call this a good rule, and allowing tweaking of GC parameters would be more about trying to over-tune the runtime to the benchmark rather than the benchmark illustrating what is feasible to achieve in the language as a library or as part of a larger system.
jackmott 9y ago
they reflect a number of different real world workloads.
devveppev 9y ago
This criticism is flawed. The fact that the code is imperfect or sub optimal actually makes their conclusions strong and more useful for real world comparisons. The code guidelines in their benchmarks represent real programming practices in those languages. They promote "idiomatic" code because it is meant to represent the typical quality of code in the real world. Obviously many of the examples could be made more performant, but in doing so you would made the code less representative of the real world. Example: the typescript code outputs wanky ES6 features that are slower than their old plain js counterparts (classes, arrow functions, let, etc). You could abuse the ts code until its output is identical to js, but you would have a pointless benchmark now. What would the benchmark honestly represent if the code looked nothing like code in the real world? The theoretical speed of the language just doesn't matter. In fact, JS engines have been optimizing their performance around numerical benchmarks for decades. The benchmark problems (nbody, etc) are actually highly unrepresentative of real javascript performance because real world javascript is touching strings and awful DOM apis and messing with dictionaries all day. Your 'real-world workload' should be stuff like text editor operations, a domain where JS's alleged 6x slowdown compared to C has not been remotely approached by current editors.
- Veedrac 9y ago
  > The code guidelines in their benchmarks represent real programming practices in those languages. They promote "idiomatic" code because it is meant to represent the typical quality of code in the real world. This could not be further from my observations when I tried contributing. These results are neither controlled nor the result of idiomatic programs.
  - devveppev 9y ago
    I'm talking about stuff like putting your code in classes, initializing with constructors, or using your language's standard library instead of writing your own stuff. Perhaps idiomatic was the wrong word. I mean that the code isn't supposed to be fighting the language. What were your observations when you were contributing? Which problems were you working on? Any chance you still have the code?
pmontra 9y ago
The results are what I have expected but yes, mandelbrot and n-body in Ruby? Ruby's typical workload is getting some strings from a web server, building a SQL query, sending it to a database, get a resultset, creating a bunch of objects to represent the result and send back another string to the web server. That would be more energy efficient if done in C but not as much as doing a mandelbrot. Then factor in the energy of the developer and its supporting environment for the extra development time (commuting, air conditioning / heating, laptop, monitors, etc.) The less the load on the app, the more important it gets.
- astrobe_ 9y ago
  > Then factor in the energy of the developer and its supporting environment for the extra development time (commuting, air conditioning / heating, laptop, monitors, etc.) The less the load on the app, the more important it gets. No, it's almost always negligible because it's essentially a one-time cost, whereas the program's inefficiencies are multiplied by an unknown number of users over an unknown period of time (cf. Cobol programs still being in use today) - but for sure your goal is often to maximize both of these unknown numbers. This fact is very obvious when you develop hardware/firmware combos. The firmware is often designed to minimize the hardware costs (RAM size, mass storage size,...) because the cost of producing the hardware is per piece, while the development cost is mostly a one-time cost that one rarely includes in the total cost of the product.

igouy 9y ago

> Those benchmarks don't reflect real-world workloads at all… That's a very definite claim, for which you provide no supporting evidence ;-)

oconnor663 9y ago
Maybe we could get away with a couple smaller claims: - It would be weird to expect programs specifically optimized for a performance benchmark, to also be optimal for energy usage or memory. Maybe less weird for energy, if runtime is the biggest factor in how much energy gets used. But I'd expect there to be huge tradeoffs between runtime and memory usage, once we start really optimizing for memory. - Running the Benchmarks Game has meant doing a lot of work to figure out what the relationship between "optimal" and "representative" is, and what the rules of the game should be as a result. Even if we take it for granted that we've answered those questions in a way that gives us useful benchmarks for performance, it would be weird to expect the answers to be exactly the same in benchmarks for energy or memory.
- jackmott 9y ago
  performance usually == energy efficiency on modern cpus. race to shutdown.
  - igouy 9y ago
    Does your comment agree or disagree with what they measure?
    - jackmott 9y ago
      The implementations in the benchmark game are all targeting performance. This will almost always also be the most energy efficient thing on modern cpus. It will not always be the most memory efficient way. I neither agree or disagree with what they are measuring, its just data. I would not look at this if I was trying to get an idea of what languages are most memory efficient, maybe.
      - igouy 9y ago
        I wondered if you'd looked at all ;-)
- igouy 9y ago
  Maybe we should look at "4 Threats to Validity" section of their paper and ask whether they have really done enough to suggest that their results should be considered general rather than specific.
- yjftsjthsd-h 9y ago
  Perhaps I'm over simplifying to the point of making mistakes, but intuitively I would expect that performance and Energy Efficiency are strongly correlated, because I assume that each instruction executed takes a certain amount of time and a certain amount of energy, and that this skills roughly linearly. That is, executing n instructions takes A * n time and B * n energy, so of course making it finish sooner means less instructions means less energy.
rbehrends 9y ago
One of the worst offenders is probably the binary-trees benchmark. 1. It features an atypically high allocation rate compared to real-world programs. 2. As a synthetic micro benchmark that only ever allocates objects of one size, it makes pool allocators look disproportionately good. But in real-world applications with varying object sizes, overuse of pool allocators creates a serious risk for fragmentation. 3. The rules allow programs that use manual memory management to use pretty much any pool allocator library that they can get their hands on, but forbid garbage-collected languages from adjusting their GC parameters (last I checked, you could easily improve the performance of OCaml and Dart for that benchmark by a factor of 2-3 simply by adjusting GC settings, especially the size of the minor heap).
- igouy 9y ago
  > The rules allow programs that use manual memory management to… And still there's outrage that programs with allocators custom-written for binary-trees are rejected :-)
- kllrnohj 9y ago
  > It features an atypically high allocation rate compared to real-world programs. Do you have any supporting evidence for this? Real-world programs can be pretty allocation heavy, and if anything high allocation rates would "unfairly" benefit GC languages since native programs are banned from using arena allocators. > As a synthetic micro benchmark that only ever allocates objects of one size, it makes pool allocators look disproportionately good. But in real-world applications with varying object size, overuse of pool allocators is a serious risk for fragmentation. Huh? It's so incredibly common in the real-world for a structure to be of a single type that pretty much every language makes that first-class supported via templates. And since they are single-size there's no risk of fragmentation at all. Indeed object pools are typically how you avoid fragmentation, they are not things that cause it. So I'm not sure what you're trying to get at with that comment. > The rules allow programs that use manual memory management to use pretty much any pool allocator library that they can get their hand on, but forbid garbage-collected languages from adjusting their GC parameters (last I checked, you could easily improve the performance of OCaml and Dart for that benchmark by a factor of 2-3 simply by adjusting GC settings, especially the size of the nursery). Last I checked if I write a Dart library I can't adjust the GC settings in my library to make my library run better, either, but if I ship a C++ library I can absolutely use a pool allocator to make my library run better (and indeed this is exactly what real libraries do) So I'd call this a good rule, and allowing tweaking of GC parameters would be more about trying to over-tune the runtime to the benchmark rather than the benchmark illustrating what is feasible to achieve in the language as a library or as part of a larger system.

jackmott 9y ago

they reflect a number of different real world workloads.

devveppev 9y ago

This criticism is flawed. The fact that the code is imperfect or sub optimal actually makes their conclusions strong and more useful for real world comparisons. The code guidelines in their benchmarks represent real programming practices in those languages. They promote "idiomatic" code because it is meant to represent the typical quality of code in the real world. Obviously many of the examples could be made more performant, but in doing so you would made the code less representative of the real world. Example: the typescript code outputs wanky ES6 features that are slower than their old plain js counterparts (classes, arrow functions, let, etc). You could abuse the ts code until its output is identical to js, but you would have a pointless benchmark now. What would the benchmark honestly represent if the code looked nothing like code in the real world? The theoretical speed of the language just doesn't matter. In fact, JS engines have been optimizing their performance around numerical benchmarks for decades. The benchmark problems (nbody, etc) are actually highly unrepresentative of real javascript performance because real world javascript is touching strings and awful DOM apis and messing with dictionaries all day. Your 'real-world workload' should be stuff like text editor operations, a domain where JS's alleged 6x slowdown compared to C has not been remotely approached by current editors.

Veedrac 9y ago
> The code guidelines in their benchmarks represent real programming practices in those languages. They promote "idiomatic" code because it is meant to represent the typical quality of code in the real world. This could not be further from my observations when I tried contributing. These results are neither controlled nor the result of idiomatic programs.
- devveppev 9y ago
  I'm talking about stuff like putting your code in classes, initializing with constructors, or using your language's standard library instead of writing your own stuff. Perhaps idiomatic was the wrong word. I mean that the code isn't supposed to be fighting the language. What were your observations when you were contributing? Which problems were you working on? Any chance you still have the code?

pmontra 9y ago

The results are what I have expected but yes, mandelbrot and n-body in Ruby? Ruby's typical workload is getting some strings from a web server, building a SQL query, sending it to a database, get a resultset, creating a bunch of objects to represent the result and send back another string to the web server. That would be more energy efficient if done in C but not as much as doing a mandelbrot. Then factor in the energy of the developer and its supporting environment for the extra development time (commuting, air conditioning / heating, laptop, monitors, etc.) The less the load on the app, the more important it gets.

astrobe_ 9y ago
> Then factor in the energy of the developer and its supporting environment for the extra development time (commuting, air conditioning / heating, laptop, monitors, etc.) The less the load on the app, the more important it gets. No, it's almost always negligible because it's essentially a one-time cost, whereas the program's inefficiencies are multiplied by an unknown number of users over an unknown period of time (cf. Cobol programs still being in use today) - but for sure your goal is often to maximize both of these unknown numbers. This fact is very obvious when you develop hardware/firmware combos. The firmware is often designed to minimize the hardware costs (RAM size, mass storage size,...) because the cost of producing the hardware is per piece, while the development cost is mostly a one-time cost that one rarely includes in the total cost of the product.