6 ms·

> 5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there. Apparently 5.3, but I am very skep

by etiene 9y ago

> 5. I wonder which implementation of Lua they used. Lua can be pretty fast, one of the fastest dynamic languages out there. Apparently 5.3, but I am very skeptical of these results. Also the way the data is organised is very weird. One should not compare a JIT compiled VM (JavaScript) with interpreters (Lua, Python, Ruby...). An honest approach would have used LuaJIT, PyPy etc. or separated the languages into "families" better. Not to mention the choice of algorithms they used. Some of the algorithms in the benchmark game have many flaws. As it was already stated in this thread, some of the Python ones were multithreaded but the ones in Lua weren't. Why? In a different benchmark I've seen algorithms using C wrappers for Python, but not in Lua. While I understand wrapping C and multithreading reflect the industry use in Python, so they do in Lua. Benchmarks are useless if you are not consistent in the implementations of the things you are comparing with. It disgusts me, to be honest, when I see benchmarks comparing JIT compilers with vanilla Lua just to say "hey we're faster", as LuaJIT does not even exist. It looks like pure dishonesty. In the case of this paper in particular, as they do not represent any language, it strikes me as lack of research / understanding.

igouy 9y ago
> … but the ones in Lua weren't. Why? No mystery: someone has made the effort to contribute all those multi-process Python programs; no one has made the effort to contribute all those multi-process Lua programs.
- etiene 9y ago
  Oh yes, that is true. But the person making the benchmarks should be aware of that and either look for better matching algorithms or implement them themselves.
  - igouy 9y ago
    Why should a comparison be restricted just because the best Lua programs-contributed fail to use multi core? (Tail wagging the dog). If someone who didn't know Lua wrote those programs, then you'd complain that the programs weren't written by an expert. If Lua wasn't included then I think you'd complain about that too.
    - etiene 9y ago
      If your intention is not to make a good comparison, then sure. I've seen other papers on language implementation and benchmarks, and researchers were way more careful than this, individually inspecting the algorithms they were using. I happened to spot a problem on this one on the Lua part and also on some other parts concerning how they classify things, because it happens to be things I know more. I'm not going out of my way to look for problems in the rest of the paper because I'm not a reviewer and I have better things to do. But if I was the one way conducting this academic research, then I would want the results not to be bogus. The difference here is this is not a blog post about some bullshit comparison someone is making. If that was the case, then alright, if public contributed algorithms failed to represent exactly what was being looked for, why not go for whatever is out there. I just expect a more rigorous procedure from academics.
      - igouy 9y ago
        > I just expect a more rigorous procedure from academics Are they likely to be expert in all those programming languages? ---- If Lua wasn't included would you complain ? ( Like this: https://news.ycombinator.com/item?id=15255427 https://news.ycombinator.com/item?id=15255427 https://news.ycombinator.com/item?id=15251242 https://news.ycombinator.com/item?id=15251242 https://news.ycombinator.com/item?id=15251144 https://news.ycombinator.com/item?id=15251144 )
        etiene 9y ago
        > Are they likely to be expert in all those programming languages? No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf https://arxiv.org/pdf/1602.00602.pdf > If Lua wasn't included would you complain ? Maybe I would, who knows. I did also mention in another comment it would be nice if Julia was there. It depends how relevant they are for the research being conducted. As a language used very often in microcontrollers, Lua is very relevant for energy efficiency research. I don't know anything about Forth, but I wouldn't just dismiss people saying it should be there with "then contribute forth algorithms to the benchmarks game yourself".
igouy 9y ago
> One should not compare a JIT … with interpreters Because ?
- etiene 9y ago
  Because they're "different families" of language implementation. A Just In Time compiler is a compiler. It looks like an interpreter, but it compiles the code to machine code and runs it.
  - igouy 9y ago
    And obviously the performance of programs using those language implementations on the same computer under the same workload may be measured and compared. Why should "different families" of language implementation not be compared?
    - etiene 9y ago
      Firstly, the paper separates languages into compiled languages and interpreted languages to begin with. So I was expecting it would follow this pattern correctly. Secondly, why would you compare the performance of C and Ruby, for example? Ruby is bound to be slower for design reasons. Doing it to say "Look, C is fast!" doesn't mean anything. There are different reasons one would compare C and Rust, or C and Ruby. The way the results are displayed and what it tries to convey is very important. If you're not making a baseline, or being very clear, etc. then it's just misleading to have a benchmark portraying Javascript as faster than Ruby, Python and Lua. Either you put those into their correct classifications clearly, or you compare it with LuaJIT, PyPy, Julia etc.
      - igouy 9y ago
        > … the paper separates languages into… The paper, both shows tables that include all the results (ordered by Energy consumed) and shows separate charts for what the authors classify as "either a compiled, interpreted, or virtual-machine language". What is "an interpreted language" ? "Although we refer to Lua as an interpreted language, Lua always precompiles source code to an intermediate form before running it. … The presence of a compilation phase may sound out of place in an interpreted language like Lua. However, the distinguishing feature of interpreted languages is not that they are not compiled, but that any eventual compiler is part of the language runtime and that, therefore, it is possible (and easy) to execute code generated on the fly." p57 "Programming in Lua" (2003) > … why would you compare the performance of C and Ruby, for example? Ruby is bound to be slower… Except when the C program is measured to be 50x slower than the Ruby program -- http://benchmarksgame.alioth.debian.org/u32/compare.php?lang=cint&lang2=yarv http://benchmarksgame.alioth.debian.org/u32/compare.php?lang...

by etiene 9y ago

igouy 9y ago

> … but the ones in Lua weren't. Why? No mystery: someone has made the effort to contribute all those multi-process Python programs; no one has made the effort to contribute all those multi-process Lua programs.

etiene 9y ago
Oh yes, that is true. But the person making the benchmarks should be aware of that and either look for better matching algorithms or implement them themselves.
- igouy 9y ago
  Why should a comparison be restricted just because the best Lua programs-contributed fail to use multi core? (Tail wagging the dog). If someone who didn't know Lua wrote those programs, then you'd complain that the programs weren't written by an expert. If Lua wasn't included then I think you'd complain about that too.
  - etiene 9y ago
    If your intention is not to make a good comparison, then sure. I've seen other papers on language implementation and benchmarks, and researchers were way more careful than this, individually inspecting the algorithms they were using. I happened to spot a problem on this one on the Lua part and also on some other parts concerning how they classify things, because it happens to be things I know more. I'm not going out of my way to look for problems in the rest of the paper because I'm not a reviewer and I have better things to do. But if I was the one way conducting this academic research, then I would want the results not to be bogus. The difference here is this is not a blog post about some bullshit comparison someone is making. If that was the case, then alright, if public contributed algorithms failed to represent exactly what was being looked for, why not go for whatever is out there. I just expect a more rigorous procedure from academics.
    - igouy 9y ago
      > I just expect a more rigorous procedure from academics Are they likely to be expert in all those programming languages? ---- If Lua wasn't included would you complain ? ( Like this: https://news.ycombinator.com/item?id=15255427 https://news.ycombinator.com/item?id=15255427 https://news.ycombinator.com/item?id=15251242 https://news.ycombinator.com/item?id=15251242 https://news.ycombinator.com/item?id=15251144 https://news.ycombinator.com/item?id=15251144 )
      - etiene 9y ago
        > Are they likely to be expert in all those programming languages? No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf https://arxiv.org/pdf/1602.00602.pdf > If Lua wasn't included would you complain ? Maybe I would, who knows. I did also mention in another comment it would be nice if Julia was there. It depends how relevant they are for the research being conducted. As a language used very often in microcontrollers, Lua is very relevant for energy efficiency research. I don't know anything about Forth, but I wouldn't just dismiss people saying it should be there with "then contribute forth algorithms to the benchmarks game yourself".

igouy 9y ago

> One should not compare a JIT … with interpreters Because ?

etiene 9y ago
Because they're "different families" of language implementation. A Just In Time compiler is a compiler. It looks like an interpreter, but it compiles the code to machine code and runs it.
- igouy 9y ago
  And obviously the performance of programs using those language implementations on the same computer under the same workload may be measured and compared. Why should "different families" of language implementation not be compared?
  - etiene 9y ago
    Firstly, the paper separates languages into compiled languages and interpreted languages to begin with. So I was expecting it would follow this pattern correctly. Secondly, why would you compare the performance of C and Ruby, for example? Ruby is bound to be slower for design reasons. Doing it to say "Look, C is fast!" doesn't mean anything. There are different reasons one would compare C and Rust, or C and Ruby. The way the results are displayed and what it tries to convey is very important. If you're not making a baseline, or being very clear, etc. then it's just misleading to have a benchmark portraying Javascript as faster than Ruby, Python and Lua. Either you put those into their correct classifications clearly, or you compare it with LuaJIT, PyPy, Julia etc.
    - igouy 9y ago
      > … the paper separates languages into… The paper, both shows tables that include all the results (ordered by Energy consumed) and shows separate charts for what the authors classify as "either a compiled, interpreted, or virtual-machine language". What is "an interpreted language" ? "Although we refer to Lua as an interpreted language, Lua always precompiles source code to an intermediate form before running it. … The presence of a compilation phase may sound out of place in an interpreted language like Lua. However, the distinguishing feature of interpreted languages is not that they are not compiled, but that any eventual compiler is part of the language runtime and that, therefore, it is possible (and easy) to execute code generated on the fly." p57 "Programming in Lua" (2003) > … why would you compare the performance of C and Ruby, for example? Ruby is bound to be slower… Except when the C program is measured to be 50x slower than the Ruby program -- http://benchmarksgame.alioth.debian.org/u32/compare.php?lang=cint&lang2=yarv http://benchmarksgame.alioth.debian.org/u32/compare.php?lang...