4 ms·

What would you see as a "fair" benchmark? Here's another, for the server use case: https://www.techempower.com/benchmarks/#section=data-r21&test=composite htt

What would you see as a "fair" benchmark? Here's another, for the server use case: https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes... I am stunned that a JS tool came out on top.

nindalf 4y ago
There are no fair benchmarks between programming languages. > When a measure becomes a target, it ceases to be a good measure. If you're deciding what language to use for your next project, there are better ways. Ask yourself these questions. - Can developers be hired/trained to use it? - Will they be happy using it? - What will our development velocity be on this language? What will be the operational burden of keeping it running in production? - Do we need to rely on third party code? Does that code already exist and what is it's quality? Importantly, how will the answers to these questions change over time? Will this language be just as easy to hire for in 5 years time? Comparing languages by benchmarks is a waste of time.
- galangalalgol 4y ago
  Were those prioritized? I would have put velocity first,then will devs like it. Those are coupled. If they like it, it is less important if you can hire people who know it already, and likewise, if your velocity is high, maintainability becomes less important. All those are subject to physical constraints though. Much of the worlds code is embedded, and thus severely constrained in both compute and power. That limits your choices, but within those choices your criteria are still absolutely the correct ones.
  - nindalf 4y ago
    My list was not prioritised and not exhaustive either. As you correctly point out, physical constraints matter. If there are other requirements like real-time, low latency or high throughput, we have to take those into account as well.
- jackmott42 4y ago
  Ignoring performance has gotten many companies in a terrible bind down the road. Not all people's projects are performance irrelevant.
tiagod 4y ago
> I am stunned that a JS tool came out on top I wanted to know why, and I found this article by the author: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01... also: https://github.com/just-js/just/issues/5#issuecomment-778673184 https://github.com/just-js/just/issues/5#issuecomment-778673...
IshKebab 4y ago
I always wanted to take code that hasn't been written specifically for a benchmark and test that. RosettaCode or leetcode might be a good source for that. leetcode even has many versions of each problem written by different people so you could see e.g. the distribution of expected runtimes for JavaScript programmers Vs Python programmers or whatever. It wouldn't quite be fair on the language. E.g. I would expect Typescript to give better performance than JavaScript simply because the fact that you're using Typescript means you know what you're doing more than a JavaScript programmer. Would be cool to see. Unfortunately I couldn't figure out a way to get unbiased sample programs for each problem.
- igouy 4y ago
  For you — 2021 "Ranking programming languages by energy efficiency" https://haslab.github.io/SAFER/scp21.pdf https://haslab.github.io/SAFER/scp21.pdf "In addition, we further validate our results and rankings against implementations from a chrestomathy program repository, Rosetta Code., by reproducing our methodology and benchmarking system."
  - IshKebab 4y ago
    Interesting, though that seems to be the same authors so I'm not entirely sure I trust their methodology, given how much they messed it up the first time round. Also the results (page 48) show some obvious red flags, e.g. C and Chapel being suspiciously fast on Ackermann. Still the overall results mostly match my intuition.
    - igouy 4y ago
      > … how much they messed it up the first time round. With exploratory data analysis it is important to state how outliers will be identified and treated. https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.... The data tables published with that 2017 paper, show a 15x difference between the measured times of the selected JS and TS fannkuch-redux programs. That should explain the TS and JS average Time difference. There's an order of magnitude difference between the times of the selected C and C++ programs, for one thing — regex-redux. That should explain the C and C++ average Time difference. Even without looking for cause, they seem like outliers which could have been excluded. Without those outliers you'd be telling me that "the overall results mostly match my intuition".

nindalf 4y ago

There are no fair benchmarks between programming languages. > When a measure becomes a target, it ceases to be a good measure. If you're deciding what language to use for your next project, there are better ways. Ask yourself these questions. - Can developers be hired/trained to use it? - Will they be happy using it? - What will our development velocity be on this language? What will be the operational burden of keeping it running in production? - Do we need to rely on third party code? Does that code already exist and what is it's quality? Importantly, how will the answers to these questions change over time? Will this language be just as easy to hire for in 5 years time? Comparing languages by benchmarks is a waste of time.

galangalalgol 4y ago
Were those prioritized? I would have put velocity first,then will devs like it. Those are coupled. If they like it, it is less important if you can hire people who know it already, and likewise, if your velocity is high, maintainability becomes less important. All those are subject to physical constraints though. Much of the worlds code is embedded, and thus severely constrained in both compute and power. That limits your choices, but within those choices your criteria are still absolutely the correct ones.
- nindalf 4y ago
  My list was not prioritised and not exhaustive either. As you correctly point out, physical constraints matter. If there are other requirements like real-time, low latency or high throughput, we have to take those into account as well.
jackmott42 4y ago
Ignoring performance has gotten many companies in a terrible bind down the road. Not all people's projects are performance irrelevant.

tiagod 4y ago

> I am stunned that a JS tool came out on top I wanted to know why, and I found this article by the author: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01... also: https://github.com/just-js/just/issues/5#issuecomment-778673184 https://github.com/just-js/just/issues/5#issuecomment-778673...

IshKebab 4y ago

I always wanted to take code that hasn't been written specifically for a benchmark and test that. RosettaCode or leetcode might be a good source for that. leetcode even has many versions of each problem written by different people so you could see e.g. the distribution of expected runtimes for JavaScript programmers Vs Python programmers or whatever. It wouldn't quite be fair on the language. E.g. I would expect Typescript to give better performance than JavaScript simply because the fact that you're using Typescript means you know what you're doing more than a JavaScript programmer. Would be cool to see. Unfortunately I couldn't figure out a way to get unbiased sample programs for each problem.

igouy 4y ago
For you — 2021 "Ranking programming languages by energy efficiency" https://haslab.github.io/SAFER/scp21.pdf https://haslab.github.io/SAFER/scp21.pdf "In addition, we further validate our results and rankings against implementations from a chrestomathy program repository, Rosetta Code., by reproducing our methodology and benchmarking system."
- IshKebab 4y ago
  Interesting, though that seems to be the same authors so I'm not entirely sure I trust their methodology, given how much they messed it up the first time round. Also the results (page 48) show some obvious red flags, e.g. C and Chapel being suspiciously fast on Ackermann. Still the overall results mostly match my intuition.
  - igouy 4y ago
    > … how much they messed it up the first time round. With exploratory data analysis it is important to state how outliers will be identified and treated. https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm https://www.itl.nist.gov/div898/handbook/prc/section1/prc16.... The data tables published with that 2017 paper, show a 15x difference between the measured times of the selected JS and TS fannkuch-redux programs. That should explain the TS and JS average Time difference. There's an order of magnitude difference between the times of the selected C and C++ programs, for one thing — regex-redux. That should explain the C and C++ average Time difference. Even without looking for cause, they seem like outliers which could have been excluded. Without those outliers you'd be telling me that "the overall results mostly match my intuition".