4 ms·

There is very limited data for non-artificial and non-numeric benchmarks for Lua and Python, so treat it all with a grain of salt. PyPy mostly cares about larg

by fijal 14y ago

There is very limited data for non-artificial and non-numeric benchmarks for Lua and Python, so treat it all with a grain of salt. PyPy mostly cares about large scale programs (thanks to a large quantity of Python libraries it's not that hard), while LuaJIT usually cares for smaller stuff. Last time I tried to run richards (which is the largest example I found for both Lua and Python), PyPy was able to eventually outperform LuaJIT slightly, given enough iterations, however LuaJIT warmup time was very very impressive and it was much faster at the beginning. From a very unscientific perspective I would expect LuaJIT to mostly outperform PyPy, especially on stuff where it's feasible to compare, like by implementing small numeric kernels in both Lua and Python, but I haven't measured a whole lot in this area. LuaJIT definitely outperforms PyPy on the computer language shootout benchmarks. I would also expect the field to be more leveled if you implement say a webserver in both, caring about performance, but this is really on the guessing side.

haberman 14y ago
> PyPy was able to eventually outperform LuaJIT slightly, given enough iterations Is this benchmark data-intensive? One known weak point of LuaJIT is its GC, which is slated for a rewrite: http://wiki.luajit.org/New-Garbage-Collector http://wiki.luajit.org/New-Garbage-Collector If "richards" allocates lots of objects that accumulate and need to be GC'd, this could explain why PyPy wins eventually.
- igouy 14y ago
  "[Richards] simulates the task dispatcher in the kernel of an operating system." http://labs.oracle.com/people/mario/java_benchmarking/richards/richards.html http://labs.oracle.com/people/mario/java_benchmarking/richar...
  - haberman 14y ago
    That doesn't answer the question of whether the benchmark is written in a way that generates lots of objects that need to be GC'd. I'd be very surprised if PyPy beat LuaJIT for anything that was numerical or otherwise had a fixed working set.
- ltratt 14y ago
  [Richards] > Is this benchmark data-intensive? A draft paper where C F Bolz and I look at this is here: http://tratt.net/laurie/research/publications/papers/bolz_tratt__the_impact_of_metatracing_on_vm_design_and_implementation.pdf http://tratt.net/laurie/research/publications/papers/bolz_tr... You can also download the full experimental suite and check out the particular benchmark(s) yourself. In my opinion, fijal has provided an accurate summary of the situation as we currently understand it.
stcredzero 14y ago
> PyPy was able to eventually outperform LuaJIT slightly, given enough iterations It would be awesome to save the state of the JIT. It would also be awesome to export the type inference information.
- fijal 14y ago
  The problem really is Python import semantics. As soon as you can provide me with a way to say "this imported code is identical to the one you loaded before", I'm fine with providing a way to save the JIT state. Note that "identical" means also module globals, state of sys.modules etc. etc. In short, the assumptions are all the same.
  - stcredzero 14y ago
    > As soon as you can provide me with a way to say "this imported code is identical to the one you loaded before", I'm fine with providing a way to save the JIT state. How about an SHA2 hash of the source files?
    - ot 14y ago
      That would not be sufficient. Python modules can execute arbitrary code at import time, so the behaviour could depend on a number of external factors, such as environment variables, other files, current time, response from a remote service (?!), etc...
      - stcredzero 14y ago
        That would not be sufficient in some corner cases. I think for a competent group running web application servers, which knows it needs to control such external factors, it could well be enough.
        fijal 14y ago
        but this is not the attitude you have to have when writing compilers. from the compiler standpoint, it has to be correct, even in corner cases (which are not even as corner as you would like them to be). A competent group can leave a dormant process that communicates over pipe for example and then the warmup issue disappears (it does not really appear for long running web application servers to start with)
mikemike 14y ago
<rant> Oh, well ... The Lua port of Richards I've seen is naive and nobody within his right mind would write Lua code like that. As has been pointed out elsewhere, Richards is a very atypical benchmark and different implementation styles have very different performance characteristics. In fact, PyPy wins ONLY on Richards by 1.2x and ONLY for a specific iteration count. But LuaJIT wins substantially on ALL OTHER BENCHMARKS where something is actually compiled (i.e. not regex stuff). Your paper clearly shows that in the appendix, but no matter what, you're still pointing out that one little result. And, even worse, you're trying to draw conclusions from this outlier. I have absolutely no idea how you could possibly come to the conclusion that this makes PyPy more suitable for 'large scale' programs? And that LuaJIT is not? That's not just unscientific -- it's dishonest and disparaging. Also, constantly repeating that 'hand-crafted JIT compilers' are somehow inferior to PyPy's overly complex approach is getting really old by now. LuaJIT runs on more architectures and more platforms than PyPy ever will. And it's vastly ahead in terms of optimizations. Now, please compare the timeframe all of this was implemented and the size of our teams. Ok, so what was the better deal, again? And, no, the semantics of Lua and Python are not that far apart as you'd like everyone to believe. Dispatch is more complex in Python, but that can be taken care of in a central place in the code. Most library code can be written in terms of lower-level code. And so on. There are simple solutions for all of that. And none of that warrants a clumsy meta-tracing approach. </rant>
- ltratt 14y ago
  I know myself that I had a high regard for LuaJIT before benchmarking, and an even higher regard for it afterwards. Richards is an oldish benchmark, and none of the subsequent versions (Lua, Python, Converge) is exactly "native", so I'm not sure that any of those languages has an advantage. It's a datapoint, no more, no less. As to which VM would be faster on "large" programs - well, personally I agree with your implication that it's a matter of conjecture, and we'll probably never know because it's hard to write large cross-language benchmarks. I do, however, think you might be a little harsh on meta-tracing. It's an interesting approach, in no small part because it allows VMs to be written for languages other than Python. LuaJIT can be retargeted too (I've been impressed by Thomas Schilling's work), but I think it would be fair to say that it's harder work than writing a new RPython interpreter. In other words, IMHO, there is plenty of scope for both LuaJIT and RPython/PyPy to co-exist happily alongside each other.

by fijal 14y ago

haberman 14y ago

> PyPy was able to eventually outperform LuaJIT slightly, given enough iterations Is this benchmark data-intensive? One known weak point of LuaJIT is its GC, which is slated for a rewrite: http://wiki.luajit.org/New-Garbage-Collector http://wiki.luajit.org/New-Garbage-Collector If "richards" allocates lots of objects that accumulate and need to be GC'd, this could explain why PyPy wins eventually.

igouy 14y ago
"[Richards] simulates the task dispatcher in the kernel of an operating system." http://labs.oracle.com/people/mario/java_benchmarking/richards/richards.html http://labs.oracle.com/people/mario/java_benchmarking/richar...
- haberman 14y ago
  That doesn't answer the question of whether the benchmark is written in a way that generates lots of objects that need to be GC'd. I'd be very surprised if PyPy beat LuaJIT for anything that was numerical or otherwise had a fixed working set.
ltratt 14y ago
[Richards] > Is this benchmark data-intensive? A draft paper where C F Bolz and I look at this is here: http://tratt.net/laurie/research/publications/papers/bolz_tratt__the_impact_of_metatracing_on_vm_design_and_implementation.pdf http://tratt.net/laurie/research/publications/papers/bolz_tr... You can also download the full experimental suite and check out the particular benchmark(s) yourself. In my opinion, fijal has provided an accurate summary of the situation as we currently understand it.

stcredzero 14y ago

> PyPy was able to eventually outperform LuaJIT slightly, given enough iterations It would be awesome to save the state of the JIT. It would also be awesome to export the type inference information.

fijal 14y ago
The problem really is Python import semantics. As soon as you can provide me with a way to say "this imported code is identical to the one you loaded before", I'm fine with providing a way to save the JIT state. Note that "identical" means also module globals, state of sys.modules etc. etc. In short, the assumptions are all the same.
- stcredzero 14y ago
  > As soon as you can provide me with a way to say "this imported code is identical to the one you loaded before", I'm fine with providing a way to save the JIT state. How about an SHA2 hash of the source files?
  - ot 14y ago
    That would not be sufficient. Python modules can execute arbitrary code at import time, so the behaviour could depend on a number of external factors, such as environment variables, other files, current time, response from a remote service (?!), etc...
    - stcredzero 14y ago
      That would not be sufficient in some corner cases. I think for a competent group running web application servers, which knows it needs to control such external factors, it could well be enough.
      - fijal 14y ago
        but this is not the attitude you have to have when writing compilers. from the compiler standpoint, it has to be correct, even in corner cases (which are not even as corner as you would like them to be). A competent group can leave a dormant process that communicates over pipe for example and then the warmup issue disappears (it does not really appear for long running web application servers to start with)

mikemike 14y ago

<rant> Oh, well ... The Lua port of Richards I've seen is naive and nobody within his right mind would write Lua code like that. As has been pointed out elsewhere, Richards is a very atypical benchmark and different implementation styles have very different performance characteristics. In fact, PyPy wins ONLY on Richards by 1.2x and ONLY for a specific iteration count. But LuaJIT wins substantially on ALL OTHER BENCHMARKS where something is actually compiled (i.e. not regex stuff). Your paper clearly shows that in the appendix, but no matter what, you're still pointing out that one little result. And, even worse, you're trying to draw conclusions from this outlier. I have absolutely no idea how you could possibly come to the conclusion that this makes PyPy more suitable for 'large scale' programs? And that LuaJIT is not? That's not just unscientific -- it's dishonest and disparaging. Also, constantly repeating that 'hand-crafted JIT compilers' are somehow inferior to PyPy's overly complex approach is getting really old by now. LuaJIT runs on more architectures and more platforms than PyPy ever will. And it's vastly ahead in terms of optimizations. Now, please compare the timeframe all of this was implemented and the size of our teams. Ok, so what was the better deal, again? And, no, the semantics of Lua and Python are not that far apart as you'd like everyone to believe. Dispatch is more complex in Python, but that can be taken care of in a central place in the code. Most library code can be written in terms of lower-level code. And so on. There are simple solutions for all of that. And none of that warrants a clumsy meta-tracing approach. </rant>

ltratt 14y ago
I know myself that I had a high regard for LuaJIT before benchmarking, and an even higher regard for it afterwards. Richards is an oldish benchmark, and none of the subsequent versions (Lua, Python, Converge) is exactly "native", so I'm not sure that any of those languages has an advantage. It's a datapoint, no more, no less. As to which VM would be faster on "large" programs - well, personally I agree with your implication that it's a matter of conjecture, and we'll probably never know because it's hard to write large cross-language benchmarks. I do, however, think you might be a little harsh on meta-tracing. It's an interesting approach, in no small part because it allows VMs to be written for languages other than Python. LuaJIT can be retargeted too (I've been impressed by Thomas Schilling's work), but I think it would be fair to say that it's harder work than writing a new RPython interpreter. In other words, IMHO, there is plenty of scope for both LuaJIT and RPython/PyPy to co-exist happily alongside each other.