9 ms·

My experience with the computer language shootout

95 points by kingkilr 16y ago

wriq 16y ago
It's a shame that it all seems to be up to one person and what he will allow considering how often the shootout is referenced in discussions. I wonder if Mike Pall's experience was similar when he posted the Lua/LuaJIT versions.
- scythe 16y ago
  Lua still has no regex-dna because the standard string functions aren't proper regexes (no choice operator) and LPeg isn't available as a Debian package (which is the real problem). 'course, it'd be hard for LuaJIT to do much better in the shootout than it does now. It's beating C#.
  - rtaycher 16y ago
    Better(actually faster on 1/3, smaller on a 1/3) then mono!!!/c# http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=luajit&lang2=csharp http://shootout.alioth.debian.org/u32/benchmark.php?test=all... , mostly slower then java -server http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=luajit&lang2=java http://shootout.alioth.debian.org/u32/benchmark.php?test=all... . Still impressive.
  - mikemike 16y ago
    If the submissions using low-level types would be included in the total score, LuaJIT would show at around 1.5x instead of 2.5x.
- pygy_ 16y ago
  AFAIK, the LuaJIT entries use custom, optimized scripts.
  - igouy 16y ago
    They certainly use Mike Pall's expert skill as a Lua programmer and implementer of LuaJIT ;-) But they are Lua programs measured on both the Lua interpreter and LuaJIT. (Programs that rely on the LuaJIT only FFI library and won't work with the Lua interpreter are shown separately.)
- deleted 16y ago
  [deleted]
- mikemike 16y ago
  Same experience here with LuaJIT. My submissions using low-level types (byte arrays and such) were put into 'interesting alternative', too. Almost all other languages can use byte arrays, when they are the appropriate data structure for the job. The C submissions make heavy use of GCC extensions, Haskell gets to use mutable (OMG!) byte arrays and Free Pascal has about as much in common with Wirth's Pascal as the name. But Python and Lua are not allowed to do that? Apparently not all languages are treated equally. Dismissing submissions by resorting to a flawed definition of 'standard' and then suppressing further debate is really lame. I contributed almost all of the Lua programs to the shootout, but I do not feel particularly encouraged to continue contributing any programs.
  - igouy 16y ago
    Here's how Alex Gaynor compared his program to your programs- "It's not used to do any crazy hackery like the LuaJIT one, just to access the libc write() function."
    - mikemike 16y ago
      So what? Look at the C, Ada or ATS submissions for even more 'crazy hackery'. My programs are quite straightforward in comparison.
  - igouy 16y ago
    Please show that there's some substance behind your "suppressing further debate" accusation. Obviously your opinion wasn't suppressed on proggit. Obviously your opinion wasn't suppressed here. And nothings been done to stop you posting in the discussion forum or commenting in the tracker.
- igouy 16y ago
  It's up to you! Take the measurement scripts (download from the Help page), measure programs and publish your measurements.
glenjamin 16y ago
I don't see why the same program has to run on PyPy, CPython and Python3. Isn't the idea to do different implementations for each language. I know technically they expose the same (or a similar) language, but they're dissimilar under the hood. I'd imagine the implementation varies across the other languages by more than just syntax. The point that worries me the most, is the amount of microoptimisation being applied to these "benchmark" programs, making the results more or less pointless for real-world use.
- deleted 16y ago
  [deleted]
- igouy 16y ago
  There's nothing to say that all of CPython, Python 3, and PyPy must be measured. For the moment, they all are being measured, so it's interesting to see that programs written for CPython might perform badly with PyPy, and programs written for PyPy might perform badly with CPython.
  - glenjamin 16y ago
    I think in general, programs written in "Python" will perform better in PyPy then CPython, but the current submissions are hyper-optimised for the implementation details of CPython.
    - igouy 16y ago
      PyPy was shown for 2 years. PyPy performance with programs hyper-optimised for PyPy would have been shown if someone had contributed them. Is libc.write a great example of programs written in "Python"?
jerf 16y ago
It's true that it's arbitrary; the problem is, there's no non-arbitrary alternative, so it's a weak charge to make. Unless you're going to allow every language to converge on simply blasting some bytes into RAM and executing optimized machine-language code and watch all the languages cluster to the exact same location give or take how long it takes to load the machine code, you have to draw some lines about what's a "real" implementation and what isn't. If I were running it, I'd be even more strict and insist that the solutions must be "idiomatic"... and what's idiomatic and what isn't would also be arbitrary, because there is no escape from the "arbitrary". Yet the end result is useful. Not totally determinative, but it's not anywhere near "useless" either.
- akkartik 16y ago
  Combining your comment and DarkShikari's (http://news.ycombinator.com/item?id=2404278 http://news.ycombinator.com/item?id=2404278), it seems like a good shootout should simply compare the simplest, most idiomatic solution possible in each language.
  - stewbrew 16y ago
    The problem with a comparison of idiomatic solutions is that they rely in different degrees on what's in their libraries. IIRC the shootout tries to avoid this by comparing similar solutions using the same algorithm that are actually implemented with the interpreter/compiler in question. IMHO that's the more sensible way to do this.
    - akkartik 16y ago
      Perhaps that concern is eliminated with good test programs? As an extreme example, if I wanted to compare how different languages do array slicing it's meaningless to force python to implement array slicing without the [:] operator. But then this whole test program is meaningless anyway. You're right that there's a lot of nuance here, and perhaps there's no better solution. It's not even clear to me if there's a meaningful question to be answered. "This ocean's warmer than that one? What part? In summer or winter? Did you measure near a blue whale's fart?"
DarkShikari 16y ago
In my experience language shootouts in general are often terrible by design -- with these being just a prime example. A lot of the problem is in the questions they're designed to answer versus the questions people use them to answer. For example, if I'm comparing Python and C, I typically want to know "how much slower would my program be in Python?", not "how much slower is my program in Python if I spent so much time hyper-optimizing it that I might as well have written it in C?" But the test cases usually try to answer the latter, not the former.
- dkarl 16y ago
  It might be more reasonable than it seems at first glance. It's true that it's good to know how fast typical code runs, but there's another important question: when I run into performance problems and need to optimize a bottleneck, how fast can I make it before I have to resort to non-portable code or C extensions that complicate my build process?
  - dagw 16y ago
    Anybody writing, for example, Python code to solve these sort of problems in the real world would instantly reach for numpy. Which, while not part of the core language distribution, is pretty close to being a standard library for most python programmers. I'm sure several of the other languages have similar libraries that are being ignored in these benchmarks. Without taking things like that into account, theses results don't say too many useful things about real world performance.
    - igouy 16y ago
      They say what difference numpy might make! http://shootout.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=python3&id=2 http://shootout.alioth.debian.org/u64q/program.php?test=spec... http://shootout.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=python3&id=6 http://shootout.alioth.debian.org/u64q/program.php?test=spec...
      - 16y ago
onan_barbarian 16y ago
The Shootout's C implementations do occasionally turn into a festival of SSE intrinsics and the associated unrolling. Some benchmarks, anyway. Still, not using SSE when it's available is dumb and perhaps all the other languages need to start playing, too. A little tricker in Python, of course....
- kingkilr 16y ago
  Well, that's what the compiler is for! PyPy uses SSE under the hood for various float handing things.
  - onan_barbarian 16y ago
    Unfortunately, SSE - especially in integer-land, so I suppose we're talking SSE2 and beyond - often requires more specialist care and feeding than any automatic method (compiler or run-time) can provide. Some cases aren't hard to pick up (e.g. bulk operations on big arrays) but others require trickery of the kind that compilers usually don't (or couldn't) have. This isn't made easier by the notoriously non-orthogonal nature of the SSE integer operations and the rather limited number of ways that you can get in and out of SSE-land (to, say, affect a conditional or get something into a GPR).
  - igouy 16y ago
    Also "J2SE platform version 1.4.2 now uses SSE and SSE2 instruction sets for floating point computations".
- Luyt 16y ago
  I once wrote a function which uses MMX to convert a (possibly very large) string to lowercase. That was pretty much faster than the library's strlwr() function. http://codepad.org/BeDqS1Ws http://codepad.org/BeDqS1Ws
  - onan_barbarian 16y ago
    Great stuff. I think there are some potential tricks here to reduce the number of comparisons - maybe do a parallel subtract by k to pull down 'A' to -128 (smallest possible byte) then do your comparison against (ord('Z')-k). Or maybe push up 'Z' to +127..? That way you can get a single comparison and can replace a pcmpgtb and pand with a single subtract. Then switch it to SSE2, unroll and you're good to go. Alternately, http://www.azillionmonkeys.com/qed/asmexample.html http://www.azillionmonkeys.com/qed/asmexample.html in section 11 ("Converting Uppercase") contains a brainsmashing version of this entirely in SWAR ("SIMD Within a Register"), which could be adapted with a certain amount of pain (largely due to the absence of a double-quadword bitshift in SSE2, which is retardlepated).
scythe 16y ago
Why not at the least accept the first submission for the PyPy entries? It shouldn't matter that it doesn't run on CPython -- if it's idiomatic Python and it runs on PyPy it looks valid. At the same time, I can see where the guy is coming from with ctypes.
robryan 16y ago
Maybe you should have to submit and have it ran against a secret program that covers many areas of an implementation, so it's unknown precisely what needs to be optimized to improve on the test. Have a limit for how often you can retest it to so the only real way to be the fastest is to optimize lots of stuff which is a net win for language users.

95 points by kingkilr 16y ago

wriq 16y ago

It's a shame that it all seems to be up to one person and what he will allow considering how often the shootout is referenced in discussions. I wonder if Mike Pall's experience was similar when he posted the Lua/LuaJIT versions.

scythe 16y ago
Lua still has no regex-dna because the standard string functions aren't proper regexes (no choice operator) and LPeg isn't available as a Debian package (which is the real problem). 'course, it'd be hard for LuaJIT to do much better in the shootout than it does now. It's beating C#.
- rtaycher 16y ago
  Better(actually faster on 1/3, smaller on a 1/3) then mono!!!/c# http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=luajit&lang2=csharp http://shootout.alioth.debian.org/u32/benchmark.php?test=all... , mostly slower then java -server http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=luajit&lang2=java http://shootout.alioth.debian.org/u32/benchmark.php?test=all... . Still impressive.
- mikemike 16y ago
  If the submissions using low-level types would be included in the total score, LuaJIT would show at around 1.5x instead of 2.5x.
pygy_ 16y ago
AFAIK, the LuaJIT entries use custom, optimized scripts.
- igouy 16y ago
  They certainly use Mike Pall's expert skill as a Lua programmer and implementer of LuaJIT ;-) But they are Lua programs measured on both the Lua interpreter and LuaJIT. (Programs that rely on the LuaJIT only FFI library and won't work with the Lua interpreter are shown separately.)
deleted 16y ago
[deleted]
mikemike 16y ago
Same experience here with LuaJIT. My submissions using low-level types (byte arrays and such) were put into 'interesting alternative', too. Almost all other languages can use byte arrays, when they are the appropriate data structure for the job. The C submissions make heavy use of GCC extensions, Haskell gets to use mutable (OMG!) byte arrays and Free Pascal has about as much in common with Wirth's Pascal as the name. But Python and Lua are not allowed to do that? Apparently not all languages are treated equally. Dismissing submissions by resorting to a flawed definition of 'standard' and then suppressing further debate is really lame. I contributed almost all of the Lua programs to the shootout, but I do not feel particularly encouraged to continue contributing any programs.
- igouy 16y ago
  Here's how Alex Gaynor compared his program to your programs- "It's not used to do any crazy hackery like the LuaJIT one, just to access the libc write() function."
  - mikemike 16y ago
    So what? Look at the C, Ada or ATS submissions for even more 'crazy hackery'. My programs are quite straightforward in comparison.
- igouy 16y ago
  Please show that there's some substance behind your "suppressing further debate" accusation. Obviously your opinion wasn't suppressed on proggit. Obviously your opinion wasn't suppressed here. And nothings been done to stop you posting in the discussion forum or commenting in the tracker.
igouy 16y ago
It's up to you! Take the measurement scripts (download from the Help page), measure programs and publish your measurements.

glenjamin 16y ago

I don't see why the same program has to run on PyPy, CPython and Python3. Isn't the idea to do different implementations for each language. I know technically they expose the same (or a similar) language, but they're dissimilar under the hood. I'd imagine the implementation varies across the other languages by more than just syntax. The point that worries me the most, is the amount of microoptimisation being applied to these "benchmark" programs, making the results more or less pointless for real-world use.

deleted 16y ago
[deleted]
igouy 16y ago
There's nothing to say that all of CPython, Python 3, and PyPy must be measured. For the moment, they all are being measured, so it's interesting to see that programs written for CPython might perform badly with PyPy, and programs written for PyPy might perform badly with CPython.
- glenjamin 16y ago
  I think in general, programs written in "Python" will perform better in PyPy then CPython, but the current submissions are hyper-optimised for the implementation details of CPython.
  - igouy 16y ago
    PyPy was shown for 2 years. PyPy performance with programs hyper-optimised for PyPy would have been shown if someone had contributed them. Is libc.write a great example of programs written in "Python"?

jerf 16y ago

It's true that it's arbitrary; the problem is, there's no non-arbitrary alternative, so it's a weak charge to make. Unless you're going to allow every language to converge on simply blasting some bytes into RAM and executing optimized machine-language code and watch all the languages cluster to the exact same location give or take how long it takes to load the machine code, you have to draw some lines about what's a "real" implementation and what isn't. If I were running it, I'd be even more strict and insist that the solutions must be "idiomatic"... and what's idiomatic and what isn't would also be arbitrary, because there is no escape from the "arbitrary". Yet the end result is useful. Not totally determinative, but it's not anywhere near "useless" either.

akkartik 16y ago
Combining your comment and DarkShikari's (http://news.ycombinator.com/item?id=2404278 http://news.ycombinator.com/item?id=2404278), it seems like a good shootout should simply compare the simplest, most idiomatic solution possible in each language.
- stewbrew 16y ago
  The problem with a comparison of idiomatic solutions is that they rely in different degrees on what's in their libraries. IIRC the shootout tries to avoid this by comparing similar solutions using the same algorithm that are actually implemented with the interpreter/compiler in question. IMHO that's the more sensible way to do this.
  - akkartik 16y ago
    Perhaps that concern is eliminated with good test programs? As an extreme example, if I wanted to compare how different languages do array slicing it's meaningless to force python to implement array slicing without the [:] operator. But then this whole test program is meaningless anyway. You're right that there's a lot of nuance here, and perhaps there's no better solution. It's not even clear to me if there's a meaningful question to be answered. "This ocean's warmer than that one? What part? In summer or winter? Did you measure near a blue whale's fart?"

DarkShikari 16y ago

In my experience language shootouts in general are often terrible by design -- with these being just a prime example. A lot of the problem is in the questions they're designed to answer versus the questions people use them to answer. For example, if I'm comparing Python and C, I typically want to know "how much slower would my program be in Python?", not "how much slower is my program in Python if I spent so much time hyper-optimizing it that I might as well have written it in C?" But the test cases usually try to answer the latter, not the former.

dkarl 16y ago
It might be more reasonable than it seems at first glance. It's true that it's good to know how fast typical code runs, but there's another important question: when I run into performance problems and need to optimize a bottleneck, how fast can I make it before I have to resort to non-portable code or C extensions that complicate my build process?
- dagw 16y ago
  Anybody writing, for example, Python code to solve these sort of problems in the real world would instantly reach for numpy. Which, while not part of the core language distribution, is pretty close to being a standard library for most python programmers. I'm sure several of the other languages have similar libraries that are being ignored in these benchmarks. Without taking things like that into account, theses results don't say too many useful things about real world performance.
  - igouy 16y ago
    They say what difference numpy might make! http://shootout.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=python3&id=2 http://shootout.alioth.debian.org/u64q/program.php?test=spec... http://shootout.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=python3&id=6 http://shootout.alioth.debian.org/u64q/program.php?test=spec...
    - 16y ago

onan_barbarian 16y ago

The Shootout's C implementations do occasionally turn into a festival of SSE intrinsics and the associated unrolling. Some benchmarks, anyway. Still, not using SSE when it's available is dumb and perhaps all the other languages need to start playing, too. A little tricker in Python, of course....

kingkilr 16y ago
Well, that's what the compiler is for! PyPy uses SSE under the hood for various float handing things.
- onan_barbarian 16y ago
  Unfortunately, SSE - especially in integer-land, so I suppose we're talking SSE2 and beyond - often requires more specialist care and feeding than any automatic method (compiler or run-time) can provide. Some cases aren't hard to pick up (e.g. bulk operations on big arrays) but others require trickery of the kind that compilers usually don't (or couldn't) have. This isn't made easier by the notoriously non-orthogonal nature of the SSE integer operations and the rather limited number of ways that you can get in and out of SSE-land (to, say, affect a conditional or get something into a GPR).
- igouy 16y ago
  Also "J2SE platform version 1.4.2 now uses SSE and SSE2 instruction sets for floating point computations".
Luyt 16y ago
I once wrote a function which uses MMX to convert a (possibly very large) string to lowercase. That was pretty much faster than the library's strlwr() function. http://codepad.org/BeDqS1Ws http://codepad.org/BeDqS1Ws
- onan_barbarian 16y ago
  Great stuff. I think there are some potential tricks here to reduce the number of comparisons - maybe do a parallel subtract by k to pull down 'A' to -128 (smallest possible byte) then do your comparison against (ord('Z')-k). Or maybe push up 'Z' to +127..? That way you can get a single comparison and can replace a pcmpgtb and pand with a single subtract. Then switch it to SSE2, unroll and you're good to go. Alternately, http://www.azillionmonkeys.com/qed/asmexample.html http://www.azillionmonkeys.com/qed/asmexample.html in section 11 ("Converting Uppercase") contains a brainsmashing version of this entirely in SWAR ("SIMD Within a Register"), which could be adapted with a certain amount of pain (largely due to the absence of a double-quadword bitshift in SSE2, which is retardlepated).

scythe 16y ago

Why not at the least accept the first submission for the PyPy entries? It shouldn't matter that it doesn't run on CPython -- if it's idiomatic Python and it runs on PyPy it looks valid. At the same time, I can see where the guy is coming from with ctypes.

robryan 16y ago

Maybe you should have to submit and have it ran against a secret program that covers many areas of an implementation, so it's unknown precisely what needs to be optimized to improve on the test. Have a limit for how often you can retest it to so the only real way to be the fastest is to optimize lots of stuff which is a net win for language users.