3 ms·
It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on
by pvg 8y ago
It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on earth it was there.
- andrewprock 8y agoThat's not how the game works. The community "plays the game" by submitting better benchmarks when they come up with better solutions.
- pvg 8y agoIt's not really about how the game works, it's whether the study is using sane data. In this case, it very much isn't and it should have been obvious to them the data is bad.
- andrewprock 8y agoThe site has been through several iterations. Here is the description from 12 years ago: https://web.archive.org/web/20070503205039/http://shootout.alioth.debian.org/gp4/faq.php#play https://web.archive.org/web/20070503205039/http://shootout.a...
- pvg 8y agoI don't understand what that's supposed to show.
- igouy 8y agoIf we assume they understood the relationship between JavaScript and TypeScript then maybe it should have been noticed. However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now -- https://news.ycombinator.com/item?id=15249289 https://news.ycombinator.com/item?id=15249289 https://www.google.com/search?q=energy+efficiency+programming+languages+site:www.reddit.com&client=safari&hl=en-us&prmd=niv&sa=X&ved=2ahUKEwiV_qKs0KrhAhUPVa0KHZ0gDkYQrQIoBDAPegQIChAI&biw=1024&bih=728 https://www.google.com/search?q=energy+efficiency+programmin...
- pvg 8y agomaybe it should have been noticed It's a study presented at some conference so while not exactly the Higgs boson, they're showing other people data and the conclusions they derived from it. It's 100% their job to understand what their data measures and to notice that one of the measurements is completely bogus for their purposes. The fact other people hadn't necessarily noticed on messageboards before is mildly curious but it's not really their job.
- igouy 8y agoafaict the evidence is - not - that "one of the measurements is completely bogus". On the contrary; we can see from archived web pages that other measurements showed the same relatively-poor performance, with those old versions of TypeScript.
- pvg 8y agoWhat do the archived pages have to do with this study? The study starts with a snapshot of benchmark game sources. That's a perfectly sensible way to get a bunch of implementations to bootstrap the study. But some of those implementations might be unsuitable for the study, just as (at least) one was, in their case. They don't seem to have noticed that. What's a good, benign explanation that they didn't?
- igouy 8y ago> But some of those implementations might be unsuitable for the study, just as (at least) one was, in their case. Unsuitable because?
- pvg 8y agoBecause it's a different implementation that happens to be 15 times slower than the implementation used in the straight JS version. That's fine for the benchmark game, it's a garbage input to a 'how energy efficient are these languages' study. Let's say I want to measure the 'energy efficiency' of x86 assembly and JS. I'll use sorting an array of 1000 integers. In my JS implementation, I call Array.sort. In my x86 implementation, I randomly shuffle the array and check if it's sorted, if not repeat until it is. Does measuring the execution times of these tell me anything about the 'energy efficiency' of Javascript vs x86 assembly?