5 ms·

it is absolutely a flaw in the methodology, but one I think is unavoidable without enormous resources.

kbenson 8y ago
I agree. I just wanted to point it out in case people took the analysis as entirely accurate without looking a little closer. I suspect the entries towards the top of the list are fairly accurate, by nature of their competitive standing in the benchmark games and their focus as languages on performance. It's much easier for an implementation difference to hide in the natural drift you see in the languages with VMs and interpreters that don't receive as much attention. That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.
- 52-6F-62 8y ago
  I, by nature of my current job, have to write JavaScript regularly. In those cases I almost always opt for TypeScript for my sanity. So as a regular writer of TypeScript I had the exact same question as you.
pvg 8y ago
It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on earth it was there.
- andrewprock 8y ago
  That's not how the game works. The community "plays the game" by submitting better benchmarks when they come up with better solutions.
  - pvg 8y ago
    It's not really about how the game works, it's whether the study is using sane data. In this case, it very much isn't and it should have been obvious to them the data is bad.
    - andrewprock 8y ago
      The site has been through several iterations. Here is the description from 12 years ago: https://web.archive.org/web/20070503205039/http://shootout.alioth.debian.org/gp4/faq.php#play https://web.archive.org/web/20070503205039/http://shootout.a...
      - pvg 8y ago
        I don't understand what that's supposed to show.
- igouy 8y ago
  If we assume they understood the relationship between JavaScript and TypeScript then maybe it should have been noticed. However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now -- https://news.ycombinator.com/item?id=15249289 https://news.ycombinator.com/item?id=15249289 https://www.google.com/search?q=energy+efficiency+programming+languages+site:www.reddit.com&client=safari&hl=en-us&prmd=niv&sa=X&ved=2ahUKEwiV_qKs0KrhAhUPVa0KHZ0gDkYQrQIoBDAPegQIChAI&biw=1024&bih=728 https://www.google.com/search?q=energy+efficiency+programmin...
  - pvg 8y ago
    maybe it should have been noticed It's a study presented at some conference so while not exactly the Higgs boson, they're showing other people data and the conclusions they derived from it. It's 100% their job to understand what their data measures and to notice that one of the measurements is completely bogus for their purposes. The fact other people hadn't necessarily noticed on messageboards before is mildly curious but it's not really their job.
    - igouy 8y ago
      afaict the evidence is - not - that "one of the measurements is completely bogus". On the contrary; we can see from archived web pages that other measurements showed the same relatively-poor performance, with those old versions of TypeScript.
      - pvg 8y ago
        What do the archived pages have to do with this study? The study starts with a snapshot of benchmark game sources. That's a perfectly sensible way to get a bunch of implementations to bootstrap the study. But some of those implementations might be unsuitable for the study, just as (at least) one was, in their case. They don't seem to have noticed that. What's a good, benign explanation that they didn't?
        8y ago

I agree. I just wanted to point it out in case people took the analysis as entirely accurate without looking a little closer. I suspect the entries towards the top of the list are fairly accurate, by nature of their competitive standing in the benchmark games and their focus as languages on performance. It's much easier for an implementation difference to hide in the natural drift you see in the languages with VMs and interpreters that don't receive as much attention. That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.

52-6F-62 8y ago
I, by nature of my current job, have to write JavaScript regularly. In those cases I almost always opt for TypeScript for my sanity. So as a regular writer of TypeScript I had the exact same question as you.

pvg 8y ago

It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on earth it was there.

andrewprock 8y ago
That's not how the game works. The community "plays the game" by submitting better benchmarks when they come up with better solutions.
- pvg 8y ago
  It's not really about how the game works, it's whether the study is using sane data. In this case, it very much isn't and it should have been obvious to them the data is bad.
  - andrewprock 8y ago
    The site has been through several iterations. Here is the description from 12 years ago: https://web.archive.org/web/20070503205039/http://shootout.alioth.debian.org/gp4/faq.php#play https://web.archive.org/web/20070503205039/http://shootout.a...
    - pvg 8y ago
      I don't understand what that's supposed to show.
igouy 8y ago
If we assume they understood the relationship between JavaScript and TypeScript then maybe it should have been noticed. However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now -- https://news.ycombinator.com/item?id=15249289 https://news.ycombinator.com/item?id=15249289 https://www.google.com/search?q=energy+efficiency+programming+languages+site:www.reddit.com&client=safari&hl=en-us&prmd=niv&sa=X&ved=2ahUKEwiV_qKs0KrhAhUPVa0KHZ0gDkYQrQIoBDAPegQIChAI&biw=1024&bih=728 https://www.google.com/search?q=energy+efficiency+programmin...
- pvg 8y ago
  maybe it should have been noticed It's a study presented at some conference so while not exactly the Higgs boson, they're showing other people data and the conclusions they derived from it. It's 100% their job to understand what their data measures and to notice that one of the measurements is completely bogus for their purposes. The fact other people hadn't necessarily noticed on messageboards before is mildly curious but it's not really their job.
  - igouy 8y ago
    afaict the evidence is - not - that "one of the measurements is completely bogus". On the contrary; we can see from archived web pages that other measurements showed the same relatively-poor performance, with those old versions of TypeScript.
    - pvg 8y ago
      What do the archived pages have to do with this study? The study starts with a snapshot of benchmark game sources. That's a perfectly sensible way to get a bunch of implementations to bootstrap the study. But some of those implementations might be unsuitable for the study, just as (at least) one was, in their case. They don't seem to have noticed that. What's a good, benign explanation that they didn't?
      - 8y ago