5 ms·

Please, for the love of God, can we ignore this useless paper? It has no new insights. It simply takes the Benchmarks Game and concludes that the ones that are

Please, for the love of God, can we ignore this useless paper? It has no new insights. It simply takes the Benchmarks Game and concludes that the ones that are fastest also consume least energy. Benchmarks Game is a fun exercise to kill some time on, but it doesn't indicate anything about anything. The solutions across different languages don't even have the same algorithmic complexity. How can these possibly be compared?

danuker 4y ago
What would you see as a "fair" benchmark? Here's another, for the server use case: https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes... I am stunned that a JS tool came out on top.
- nindalf 4y ago
  There are no fair benchmarks between programming languages. > When a measure becomes a target, it ceases to be a good measure. If you're deciding what language to use for your next project, there are better ways. Ask yourself these questions. - Can developers be hired/trained to use it? - Will they be happy using it? - What will our development velocity be on this language? What will be the operational burden of keeping it running in production? - Do we need to rely on third party code? Does that code already exist and what is it's quality? Importantly, how will the answers to these questions change over time? Will this language be just as easy to hire for in 5 years time? Comparing languages by benchmarks is a waste of time.
  - galangalalgol 4y ago
    Were those prioritized? I would have put velocity first,then will devs like it. Those are coupled. If they like it, it is less important if you can hire people who know it already, and likewise, if your velocity is high, maintainability becomes less important. All those are subject to physical constraints though. Much of the worlds code is embedded, and thus severely constrained in both compute and power. That limits your choices, but within those choices your criteria are still absolutely the correct ones.
    - nindalf 4y ago
      My list was not prioritised and not exhaustive either. As you correctly point out, physical constraints matter. If there are other requirements like real-time, low latency or high throughput, we have to take those into account as well.
  - jackmott42 4y ago
    Ignoring performance has gotten many companies in a terrible bind down the road. Not all people's projects are performance irrelevant.
- tiagod 4y ago
  > I am stunned that a JS tool came out on top I wanted to know why, and I found this article by the author: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01... also: https://github.com/just-js/just/issues/5#issuecomment-778673184 https://github.com/just-js/just/issues/5#issuecomment-778673...
- IshKebab 4y ago
  I always wanted to take code that hasn't been written specifically for a benchmark and test that. RosettaCode or leetcode might be a good source for that. leetcode even has many versions of each problem written by different people so you could see e.g. the distribution of expected runtimes for JavaScript programmers Vs Python programmers or whatever. It wouldn't quite be fair on the language. E.g. I would expect Typescript to give better performance than JavaScript simply because the fact that you're using Typescript means you know what you're doing more than a JavaScript programmer. Would be cool to see. Unfortunately I couldn't figure out a way to get unbiased sample programs for each problem.
  - igouy 4y ago
    For you — 2021 "Ranking programming languages by energy efficiency" https://haslab.github.io/SAFER/scp21.pdf https://haslab.github.io/SAFER/scp21.pdf "In addition, we further validate our results and rankings against implementations from a chrestomathy program repository, Rosetta Code., by reproducing our methodology and benchmarking system."
    - IshKebab 4y ago
      Interesting, though that seems to be the same authors so I'm not entirely sure I trust their methodology, given how much they messed it up the first time round. Also the results (page 48) show some obvious red flags, e.g. C and Chapel being suspiciously fast on Ackermann. Still the overall results mostly match my intuition.
      - igouy 4y ago
        > … how much they messed it up the first time round. With exploratory data analysis it is important to state how outliers will be identified and treated. https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.... The data tables published with that 2017 paper, show a 15x difference between the measured times of the selected JS and TS fannkuch-redux programs. That should explain the TS and JS average Time difference. There's an order of magnitude difference between the times of the selected C and C++ programs, for one thing — regex-redux. That should explain the C and C++ average Time difference. Even without looking for cause, they seem like outliers which could have been excluded. Without those outliers you'd be telling me that "the overall results mostly match my intuition".
deleted 4y ago
[deleted]
skohan 4y ago
That's just plain incorrect: > Through also measuring the execution time and peak memory usage, we were able to relate both to energy to understand not only how memory usage affects energy con- sumption, but also how time and energy relate. This allowed us to understand if a faster language is always the most energy efficient. As we saw, this is not always the case.
- nindalf 4y ago
  No doubt you read the graphs right after this statement? Look at the green dots (the ratio between energy consumption and time taken). Nearly all languages have comparable ratios, meaning a lower execution time leads to less energy consumed. > this is not always the case Just almost all the time, with a few exceptions. Execution time and energy consumption are very strongly correlated. Time taken is the single biggest factor in how much power is consumed in modern CPUs. The idea is to complete any task quickly and put the idle cores to sleep to conserve power.
  - skohan 4y ago
    It's explicitly a non-conclusion that there is a deterministic relationship between speed of execution and energy consumption as your initial comment implies. > Execution time behaves differently when compared to en- ergy efficiency. The results for the 3 benchmarks presented in Table 3 (and the remainder shown in the appendix) show several scenarios where a certain language energy consump- tion rank differs from the execution time rank (as the arrows in the first column indicate). In the fasta benchmark, for example, the Fortran language is second most energy effi- cient, while dropping 6 positions when it comes to execution time. Moreover, by observing the Ratio values in Figures 1 to 3 (and the remainder in the appendix under Results - C. En- ergy and Time Graphs), we clearly see a substantial variation between languages. This means that the average power is not constant, which further strengthens the previous point. There is a strong correlation, yes, which makes sense because more time will likely correlate to more work done by the hardware. However the paper proves that there are exceptions to this rule. That's interesting given that most people, including you, would have flatly assumed that performance is a proxy for energy efficiency. You could argue that it's not a finding which is going to be relevant to most software engineers, but it might be interesting if you are designing or implementing a programming language for instance.
avidphantasm 4y ago
[flagged]
- dang 4y ago
  "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." "Eschew flamebait. Avoid generic tangents." "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
sudosysgen 4y ago
This is not obvious at all. A program may very well run faster yet use more energy for a myriad of reasons. If there have been no publications about it, then this is valuable confirmation and a good citation if need be.
jackmott42 4y ago
> but it doesn't indicate anything about anything. Clearly not true. This is like when people say "IQ tests don't measure anything" but what they mean is "IQ tests don't measure what I think you think it does" You assume we are all incapable of understanding the nuance of the Benchmark game, maybe some of us are. Not all of us! >The solutions across different languages don't even have the same algorithmic complexity. How can these possibly be compared? You can compare anything you want. If you feel a particular languages solution is sub-optimal, you may go fix it rather than complain. There is no perfect way to compare language performances, this way is reasonable. If you know of a better way, set it up and do it.
- nindalf 4y ago
  By all means use the Benchmark game if you’re under the impression it’s meaningful. But let’s not call this attempt to spin it as a measurement of energy consumption novel or worthy of a paper. Of course programs that execute fewer instructions and put the CPU back to sleep will consume less power. That’s obvious. I’m not that interested in debating whether gamed benchmarks measure anything useful. You’re saying they do. Let’s agree to disagree.
igouy 4y ago
> Please, for the love of God, can we ignore…? Obviously not. Seems like the combination of climate emergency and my programming language beats your programming language has a fascination. At-least the OP posted the conference paper rather than a jpg of one table out-of-context.

danuker 4y ago

What would you see as a "fair" benchmark? Here's another, for the server use case: https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes... I am stunned that a JS tool came out on top.

nindalf 4y ago
There are no fair benchmarks between programming languages. > When a measure becomes a target, it ceases to be a good measure. If you're deciding what language to use for your next project, there are better ways. Ask yourself these questions. - Can developers be hired/trained to use it? - Will they be happy using it? - What will our development velocity be on this language? What will be the operational burden of keeping it running in production? - Do we need to rely on third party code? Does that code already exist and what is it's quality? Importantly, how will the answers to these questions change over time? Will this language be just as easy to hire for in 5 years time? Comparing languages by benchmarks is a waste of time.
- galangalalgol 4y ago
  Were those prioritized? I would have put velocity first,then will devs like it. Those are coupled. If they like it, it is less important if you can hire people who know it already, and likewise, if your velocity is high, maintainability becomes less important. All those are subject to physical constraints though. Much of the worlds code is embedded, and thus severely constrained in both compute and power. That limits your choices, but within those choices your criteria are still absolutely the correct ones.
  - nindalf 4y ago
    My list was not prioritised and not exhaustive either. As you correctly point out, physical constraints matter. If there are other requirements like real-time, low latency or high throughput, we have to take those into account as well.
- jackmott42 4y ago
  Ignoring performance has gotten many companies in a terrible bind down the road. Not all people's projects are performance irrelevant.
tiagod 4y ago
> I am stunned that a JS tool came out on top I wanted to know why, and I found this article by the author: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01... also: https://github.com/just-js/just/issues/5#issuecomment-778673184 https://github.com/just-js/just/issues/5#issuecomment-778673...
IshKebab 4y ago
I always wanted to take code that hasn't been written specifically for a benchmark and test that. RosettaCode or leetcode might be a good source for that. leetcode even has many versions of each problem written by different people so you could see e.g. the distribution of expected runtimes for JavaScript programmers Vs Python programmers or whatever. It wouldn't quite be fair on the language. E.g. I would expect Typescript to give better performance than JavaScript simply because the fact that you're using Typescript means you know what you're doing more than a JavaScript programmer. Would be cool to see. Unfortunately I couldn't figure out a way to get unbiased sample programs for each problem.
- igouy 4y ago
  For you — 2021 "Ranking programming languages by energy efficiency" https://haslab.github.io/SAFER/scp21.pdf https://haslab.github.io/SAFER/scp21.pdf "In addition, we further validate our results and rankings against implementations from a chrestomathy program repository, Rosetta Code., by reproducing our methodology and benchmarking system."
  - IshKebab 4y ago
    Interesting, though that seems to be the same authors so I'm not entirely sure I trust their methodology, given how much they messed it up the first time round. Also the results (page 48) show some obvious red flags, e.g. C and Chapel being suspiciously fast on Ackermann. Still the overall results mostly match my intuition.
    - igouy 4y ago
      > … how much they messed it up the first time round. With exploratory data analysis it is important to state how outliers will be identified and treated. https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.... The data tables published with that 2017 paper, show a 15x difference between the measured times of the selected JS and TS fannkuch-redux programs. That should explain the TS and JS average Time difference. There's an order of magnitude difference between the times of the selected C and C++ programs, for one thing — regex-redux. That should explain the C and C++ average Time difference. Even without looking for cause, they seem like outliers which could have been excluded. Without those outliers you'd be telling me that "the overall results mostly match my intuition".

deleted 4y ago

[deleted]

skohan 4y ago

That's just plain incorrect: > Through also measuring the execution time and peak memory usage, we were able to relate both to energy to understand not only how memory usage affects energy con- sumption, but also how time and energy relate. This allowed us to understand if a faster language is always the most energy efficient. As we saw, this is not always the case.

nindalf 4y ago
No doubt you read the graphs right after this statement? Look at the green dots (the ratio between energy consumption and time taken). Nearly all languages have comparable ratios, meaning a lower execution time leads to less energy consumed. > this is not always the case Just almost all the time, with a few exceptions. Execution time and energy consumption are very strongly correlated. Time taken is the single biggest factor in how much power is consumed in modern CPUs. The idea is to complete any task quickly and put the idle cores to sleep to conserve power.
- skohan 4y ago
  It's explicitly a non-conclusion that there is a deterministic relationship between speed of execution and energy consumption as your initial comment implies. > Execution time behaves differently when compared to en- ergy efficiency. The results for the 3 benchmarks presented in Table 3 (and the remainder shown in the appendix) show several scenarios where a certain language energy consump- tion rank differs from the execution time rank (as the arrows in the first column indicate). In the fasta benchmark, for example, the Fortran language is second most energy effi- cient, while dropping 6 positions when it comes to execution time. Moreover, by observing the Ratio values in Figures 1 to 3 (and the remainder in the appendix under Results - C. En- ergy and Time Graphs), we clearly see a substantial variation between languages. This means that the average power is not constant, which further strengthens the previous point. There is a strong correlation, yes, which makes sense because more time will likely correlate to more work done by the hardware. However the paper proves that there are exceptions to this rule. That's interesting given that most people, including you, would have flatly assumed that performance is a proxy for energy efficiency. You could argue that it's not a finding which is going to be relevant to most software engineers, but it might be interesting if you are designing or implementing a programming language for instance.

avidphantasm 4y ago

[flagged]

dang 4y ago
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." "Eschew flamebait. Avoid generic tangents." "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

sudosysgen 4y ago

This is not obvious at all. A program may very well run faster yet use more energy for a myriad of reasons. If there have been no publications about it, then this is valuable confirmation and a good citation if need be.

jackmott42 4y ago

> but it doesn't indicate anything about anything. Clearly not true. This is like when people say "IQ tests don't measure anything" but what they mean is "IQ tests don't measure what I think you think it does" You assume we are all incapable of understanding the nuance of the Benchmark game, maybe some of us are. Not all of us! >The solutions across different languages don't even have the same algorithmic complexity. How can these possibly be compared? You can compare anything you want. If you feel a particular languages solution is sub-optimal, you may go fix it rather than complain. There is no perfect way to compare language performances, this way is reasonable. If you know of a better way, set it up and do it.

nindalf 4y ago
By all means use the Benchmark game if you’re under the impression it’s meaningful. But let’s not call this attempt to spin it as a measurement of energy consumption novel or worthy of a paper. Of course programs that execute fewer instructions and put the CPU back to sleep will consume less power. That’s obvious. I’m not that interested in debating whether gamed benchmarks measure anything useful. You’re saying they do. Let’s agree to disagree.

igouy 4y ago

> Please, for the love of God, can we ignore…? Obviously not. Seems like the combination of climate emergency and my programming language beats your programming language has a fascination. At-least the OP posted the conference paper rather than a jpg of one table out-of-context.