4 ms·

submit a better implementation. it is not possible for any one entity to produce optimal implementations for many languages, it is up to us. or, if you dont l

submit a better implementation. it is not possible for any one entity to produce optimal implementations for many languages, it is up to us. or, if you dont like their rules, make your own benchmark system with new rules.

kbenson 8y ago
I don't use Typescript, and use Javascript only for what I have to. I don't care about the ranking of either of them, I'm just pointing out a possible flaw in the methodology that should be taken into account when looking at the numbers presented, and what I suspect is a concrete example of that. This also isn't a criticism of the benchmark game, it's well known that not every implementation is equivalent in the time and effort put into optimizing it. It serves its purpose about as well as can be expected. Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated into results that are not always obviously an artifact of the underlying system, as I suspect this is. Making that obvious by pointing it out can be useful.
- gameswithgo 8y ago
  it is absolutely a flaw in the methodology, but one I think is unavoidable without enormous resources.
  - kbenson 8y ago
    I agree. I just wanted to point it out in case people took the analysis as entirely accurate without looking a little closer. I suspect the entries towards the top of the list are fairly accurate, by nature of their competitive standing in the benchmark games and their focus as languages on performance. It's much easier for an implementation difference to hide in the natural drift you see in the languages with VMs and interpreters that don't receive as much attention. That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.
    - 52-6F-62 8y ago
      I, by nature of my current job, have to write JavaScript regularly. In those cases I almost always opt for TypeScript for my sanity. So as a regular writer of TypeScript I had the exact same question as you.
  - pvg 8y ago
    It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on earth it was there.
    - andrewprock 8y ago
      That's not how the game works. The community "plays the game" by submitting better benchmarks when they come up with better solutions.
      - pvg 8y ago
        It's not really about how the game works, it's whether the study is using sane data. In this case, it very much isn't and it should have been obvious to them the data is bad.
        andrewprock 8y ago
        The site has been through several iterations. Here is the description from 12 years ago: https://web.archive.org/web/20070503205039/http://shootout.alioth.debian.org/gp4/faq.php#play https://web.archive.org/web/20070503205039/http://shootout.a...
        pvg 8y ago
        I don't understand what that's supposed to show.
    - igouy 8y ago
      If we assume they understood the relationship between JavaScript and TypeScript then maybe it should have been noticed. However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now -- https://news.ycombinator.com/item?id=15249289 https://news.ycombinator.com/item?id=15249289 https://www.google.com/search?q=energy+efficiency+programming+languages+site:www.reddit.com&client=safari&hl=en-us&prmd=niv&sa=X&ved=2ahUKEwiV_qKs0KrhAhUPVa0KHZ0gDkYQrQIoBDAPegQIChAI&biw=1024&bih=728 https://www.google.com/search?q=energy+efficiency+programmin...
- igouy 8y ago
  > Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated You seem not to have considered the possibility that the authors may have simply made a mistake, unrelated to the origin of the programs. The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report -- https://web.archive.org/web/20170918163900/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170918163900/http://benchmarks...
  - kbenson 8y ago
    > Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report I think the Sep 1st 2017 benchmark does, though.[1] At that point it's 1,204.93 seconds, compared to the 131.39 seconds on September 18th you referenced. That makes sense, since the paper could have been finished quite a bit prior to the conference. 1: https://web.archive.org/web/20170901020804/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170901020804/http://benchmarks...
    - deleted 8y ago
      [deleted]
    - igouy 8y ago
      That's interesting. Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program? 5 July, Node 8.1.3, TypeScript 2.4.1 https://web.archive.org/web/20170715120038/http://benchmarksgame.alioth.debian.org/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170715120038/http://benchmarks... 1 Sep, Node 8.4.0, TypeScript 2.5.2 https://web.archive.org/web/20170922144419/http://benchmarksgame.alioth.debian.org:80/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170922144419/http://benchmarks... ---- How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?

kbenson 8y ago

I don't use Typescript, and use Javascript only for what I have to. I don't care about the ranking of either of them, I'm just pointing out a possible flaw in the methodology that should be taken into account when looking at the numbers presented, and what I suspect is a concrete example of that. This also isn't a criticism of the benchmark game, it's well known that not every implementation is equivalent in the time and effort put into optimizing it. It serves its purpose about as well as can be expected. Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated into results that are not always obviously an artifact of the underlying system, as I suspect this is. Making that obvious by pointing it out can be useful.

gameswithgo 8y ago
it is absolutely a flaw in the methodology, but one I think is unavoidable without enormous resources.
- kbenson 8y ago
  I agree. I just wanted to point it out in case people took the analysis as entirely accurate without looking a little closer. I suspect the entries towards the top of the list are fairly accurate, by nature of their competitive standing in the benchmark games and their focus as languages on performance. It's much easier for an implementation difference to hide in the natural drift you see in the languages with VMs and interpreters that don't receive as much attention. That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.
  - 52-6F-62 8y ago
    I, by nature of my current job, have to write JavaScript regularly. In those cases I almost always opt for TypeScript for my sanity. So as a regular writer of TypeScript I had the exact same question as you.
- pvg 8y ago
  It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on earth it was there.
  - andrewprock 8y ago
    That's not how the game works. The community "plays the game" by submitting better benchmarks when they come up with better solutions.
    - pvg 8y ago
      It's not really about how the game works, it's whether the study is using sane data. In this case, it very much isn't and it should have been obvious to them the data is bad.
      - andrewprock 8y ago
        The site has been through several iterations. Here is the description from 12 years ago: https://web.archive.org/web/20070503205039/http://shootout.alioth.debian.org/gp4/faq.php#play https://web.archive.org/web/20070503205039/http://shootout.a...
        pvg 8y ago
        I don't understand what that's supposed to show.
  - igouy 8y ago
    If we assume they understood the relationship between JavaScript and TypeScript then maybe it should have been noticed. However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now -- https://news.ycombinator.com/item?id=15249289 https://news.ycombinator.com/item?id=15249289 https://www.google.com/search?q=energy+efficiency+programming+languages+site:www.reddit.com&client=safari&hl=en-us&prmd=niv&sa=X&ved=2ahUKEwiV_qKs0KrhAhUPVa0KHZ0gDkYQrQIoBDAPegQIChAI&biw=1024&bih=728 https://www.google.com/search?q=energy+efficiency+programmin...
igouy 8y ago
> Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated You seem not to have considered the possibility that the authors may have simply made a mistake, unrelated to the origin of the programs. The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report -- https://web.archive.org/web/20170918163900/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170918163900/http://benchmarks...
- kbenson 8y ago
  > Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report I think the Sep 1st 2017 benchmark does, though.[1] At that point it's 1,204.93 seconds, compared to the 131.39 seconds on September 18th you referenced. That makes sense, since the paper could have been finished quite a bit prior to the conference. 1: https://web.archive.org/web/20170901020804/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170901020804/http://benchmarks...
  - deleted 8y ago
    [deleted]
  - igouy 8y ago
    That's interesting. Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program? 5 July, Node 8.1.3, TypeScript 2.4.1 https://web.archive.org/web/20170715120038/http://benchmarksgame.alioth.debian.org/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170715120038/http://benchmarks... 1 Sep, Node 8.4.0, TypeScript 2.5.2 https://web.archive.org/web/20170922144419/http://benchmarksgame.alioth.debian.org:80/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170922144419/http://benchmarks... ---- How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?