5 ms·

I didn't read the original paper[0] but it is based on the Debian Benchmark Game[1] as it was in 2017 and the game relies on user submitted code for each langua

I didn't read the original paper[0] but it is based on the Debian Benchmark Game[1] as it was in 2017 and the game relies on user submitted code for each language separately, so most likely when the researchers checked the results, the TypeScript tests didn't have as optimized tests as the JavaScript tests - which makes sense as the latter is way more popular than the former, so there are less people to bother with it (and in fact the benchmarks nowadays do not include TypeScript at all). You can confirm it via archive.org too: the TypeScript[2] page shows both less implementations and the implementations that are shown are often slower than those in the JavaScript[3] page. As for if that makes sense, well, IMO using the benchmark game for judging how good a language is at optimizations is flawed in the first place as not only there is a bias towards the more popular languages but also a lot of the top entries use approaches that in practice you wouldn't find in real projects. [0] https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleFinal.pdf https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sle... [1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [2] http://web.archive.org/web/20170425064751/https://benchmarksgame.alioth.debian.org/u64q/typescript.html http://web.archive.org/web/20170425064751/https://benchmarks... [3] http://web.archive.org/web/20170425114504/https://benchmarksgame.alioth.debian.org/u64q/javascript.html http://web.archive.org/web/20170425114504/https://benchmarks...

IshKebab 3y ago
I fixed the Typescript implementation 3 years ago when this paper was debunked back then. Didn't submit it though because they make it difficult. https://news.ycombinator.com/item?id=24826453 https://news.ycombinator.com/item?id=24826453 (the whole thread is worth a read)
- igouy 3y ago
  And-yet your program was accepted and included, and then with a later TypeScript update it stopped working. spectralnorm.typescript-6.ts(115,29): error TS2794: Expected 1 arguments, but got 0. Did you forget to include 'void' in your type argument to 'Promise'? Something like that is probably what stopped the authors of "Energy Efficiency across Programming Languages". https://sites.google.com/view/energy-efficiency-languages https://sites.google.com/view/energy-efficiency-languages
  - IshKebab 3y ago
    Yeah Typescript doesn't guarantee perfect backwards compatibility. You normally have the compiler as part of your dependencies and pin it. I guess they just didn't know. I don't know what you mean with your last sentence. That's the same paper...
BackBlast 3y ago
I appreciate the thoughtful reply. Every time I see this list trotted out I can't get past the obvious red flag. Nice to have some more background on it.
- igouy 3y ago
  --alwaysStrict So when the JavaScript doesn't type check, the authors measured a different program that does type check. Even so, that only messes up the results because the mean is used rather than the median, and the data tables published with that 2017 paper, show a 15x difference between the measured times for a single outlier the selected JS and TS fannkuch-redux programs. That single outlier is enough to distort TS and JS "mean" Time difference. https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#averages https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
igouy 3y ago
A single outlier is enough to distort the TS and JS "mean" Time difference. https://news.ycombinator.com/item?id=36524073 https://news.ycombinator.com/item?id=36524073
- badsectoracula 3y ago
  Check the pages i linked at, or even better the one that compares the two[0] around 2017. While there is a single case with wildly different results, there is more than one with results that also have a large difference in JavaScript's favor. [0] http://web.archive.org/web/20170715120019/http://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=typescript&lang2=node http://web.archive.org/web/20170715120019/http://benchmarksg...
  - igouy 3y ago
    Please read the original paper and base your comments on the repo that the authors provided. Like you, most readers haven't seen the paper. Like you, most readers have only seen "Table 4. Normalized global results for Energy, Time, and Memory" taken out-of-context (often without any way to find the original source). Most readers notice the too-large differences C/C++ and JS/TS and start speculating about how those differences were caused (because that's fun). I went back to the time measurements the authors provided and calculated the Mean, Geometric Mean and Median. Simply using a more appropriate summary statistic would have presented average values which readers would have found acceptable: JS 7.25 times slower than C TS 7.8 times slower than C :even though they were based on different programs and included an outlier. (Similar story with C/C++.)

IshKebab 3y ago

I fixed the Typescript implementation 3 years ago when this paper was debunked back then. Didn't submit it though because they make it difficult. https://news.ycombinator.com/item?id=24826453 https://news.ycombinator.com/item?id=24826453 (the whole thread is worth a read)

igouy 3y ago
And-yet your program was accepted and included, and then with a later TypeScript update it stopped working. spectralnorm.typescript-6.ts(115,29): error TS2794: Expected 1 arguments, but got 0. Did you forget to include 'void' in your type argument to 'Promise'? Something like that is probably what stopped the authors of "Energy Efficiency across Programming Languages". https://sites.google.com/view/energy-efficiency-languages https://sites.google.com/view/energy-efficiency-languages
- IshKebab 3y ago
  Yeah Typescript doesn't guarantee perfect backwards compatibility. You normally have the compiler as part of your dependencies and pin it. I guess they just didn't know. I don't know what you mean with your last sentence. That's the same paper...

BackBlast 3y ago

I appreciate the thoughtful reply. Every time I see this list trotted out I can't get past the obvious red flag. Nice to have some more background on it.

igouy 3y ago
--alwaysStrict So when the JavaScript doesn't type check, the authors measured a different program that does type check. Even so, that only messes up the results because the mean is used rather than the median, and the data tables published with that 2017 paper, show a 15x difference between the measured times for a single outlier the selected JS and TS fannkuch-redux programs. That single outlier is enough to distort TS and JS "mean" Time difference. https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#averages https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

igouy 3y ago

A single outlier is enough to distort the TS and JS "mean" Time difference. https://news.ycombinator.com/item?id=36524073 https://news.ycombinator.com/item?id=36524073

badsectoracula 3y ago
Check the pages i linked at, or even better the one that compares the two[0] around 2017. While there is a single case with wildly different results, there is more than one with results that also have a large difference in JavaScript's favor. [0] http://web.archive.org/web/20170715120019/http://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=typescript&lang2=node http://web.archive.org/web/20170715120019/http://benchmarksg...
- igouy 3y ago
  Please read the original paper and base your comments on the repo that the authors provided. Like you, most readers haven't seen the paper. Like you, most readers have only seen "Table 4. Normalized global results for Energy, Time, and Memory" taken out-of-context (often without any way to find the original source). Most readers notice the too-large differences C/C++ and JS/TS and start speculating about how those differences were caused (because that's fun). I went back to the time measurements the authors provided and calculated the Mean, Geometric Mean and Median. Simply using a more appropriate summary statistic would have presented average values which readers would have found acceptable: JS 7.25 times slower than C TS 7.8 times slower than C :even though they were based on different programs and included an outlier. (Similar story with C/C++.)