6 ms·

PyPy (together with LuaJIT and tracemonkey) has been banned from shootout, partly because we complained about unfair rules. Looking at shootout to asses the rel

by fijal 14y ago

PyPy (together with LuaJIT and tracemonkey) has been banned from shootout, partly because we complained about unfair rules. Looking at shootout to asses the relative performance of languages where you cannot compare high-performance implementations of a language is a very very bad idea. Looking at shootout to compare would very likely be a bad idea anyhow, but right now it's just completely useless.

lmm 14y ago
Can you give details and/or a citation for the complaints and the banning? I use the shootout a lot to give me a rough idea of a specific language's properties, so if there are valid criticisms then I'd like to hear them, but your post doesn't really stand on its own to someone who doesn't know the backstory.
- igouy 14y ago
  >>a rough idea of a specific language's properties<< No, but maybe a rough idea of a specific language implementation's properties. >> if there are valid criticisms << There are plenty ;-) http://shootout.alioth.debian.org/dont-jump-to-conclusions.php http://shootout.alioth.debian.org/dont-jump-to-conclusions.p... Readers obviously jump to conclusions based on box plots and summary statistics -- when they really need to look at the individual tasks and look at the program source code, because there's so much variation. I think there would need to be incredibly tight constraints on minimum program run-time, source code size, and memory use; before summary statistics could be taken at face value.
- fijal 14y ago
  There is pypy-dev discussion about the subject that starts here: http://mail.python.org/pipermail/pypy-dev/2011-April/007139.html http://mail.python.org/pipermail/pypy-dev/2011-April/007139.... It's a long and boring read, but here you go. As in the actual complaints, as far as I remember, but this was a bit ago: * our versions got rejected because they perform slower on CPython (while being faster on PyPy) and because they're 'unobvious'. At the same time, the C implementation of a lot of things there is very unobvious. * we were not allowed to use array.array module, despite being in standard library. * you can use gmp library for C, CPython, but not PyPy via ctypes * C can use a bad random (that's fast, but also gives a really bad distribution), while PyPy not because Python comes with a random in stdlib * custom malloc libraries for C are fine, tuning JIT parameters per-benchmark is not. And various complaints like this. After some discussion, Isaac just kicked us out alltogether. Overall, my personal impression of the experience was that shootout is there to showcase things that are predetermined and we're trying to attack the status quo. This is obviously very personal, I don't have any actual data to back up this claim.
  - igouy 14y ago
    READER BEWARE! >>our versions got rejected because they perform slower on CPython (while being faster on PyPy)<< Maciej Fijalkowski knows that I asked him to contribute a PyPy version of n-body because the written-for-CPython version failed with PyPy. Maciej Fijalkowski knows that his own PyPy version of n-body "performed slower on CPython (while being faster on PyPy)" -- AND WAS NOT REJECTED. http://mail.python.org/pipermail/pypy-dev/2011-April/007177.html http://mail.python.org/pipermail/pypy-dev/2011-April/007177.... Justify your accusation or take it back! >> you can use gmp library for C, CPython, but not PyPy via ctypes << Joe La Fata's pi-digits code worked first time on x86 and x64, on PyPy and CPython and Python3; and used ctypes to get to GMP -- AND WAS NOT REJECTED http://anonscm.debian.org/viewvc/shootout/shootout/bench/pidigits/pidigits.python3-6.python3?hideattic=0&revision=1.1&view=markup http://anonscm.debian.org/viewvc/shootout/shootout/bench/pid... Justify your accusation or take it back! >> C can use a bad random (that's fast, but also gives a really bad distribution), while PyPy not because Python comes with a random in stdlib << The fasta task uses random numbers, and every fasta program (including C and Python) implements the same random function. http://shootout.alioth.debian.org/u32/program.php?test=fasta&lang=gcc&id=4 http://shootout.alioth.debian.org/u32/program.php?test=fasta... http://shootout.alioth.debian.org/u32/program.php?test=fasta&lang=python3&id=2 http://shootout.alioth.debian.org/u32/program.php?test=fasta... Justify your accusation or take it back! ---- Tell Maciej Fijalkowski to justify his accusations!
    - fijal 14y ago
      Cool that you're linking to this mail, this shows yet another problem that I forgot about - that pypy extensions are uncool, while GCC extensions are cool (and say a custom malloc library for GCbench - after all the benchmark does not do anything, an empty malloc would do just fine).
    - fijal 14y ago
      Look Igor -- I don't care how things are now since PyPy got kicked out anyway, so I seriously don't care any more. gmpy uses C extension, it's not using ctypes. We disagreed over a ctypes version at some point. random - indeed it got fixed (and it's too late to edit a parent if you wonder)
jerf 14y ago
I was confining myself to CPython on the grounds that it is still what people complain about in terms of adding multicore to it. People don't complain about PyPy not having it since it's still developing, and it is not yet known whether PyPy will be fast enough to be worth worrying about multicore. To be honest I'm skeptical, but open to the possibility. But I rather suspect in the end that Python will forever be a slower language than the competition; all the dynamicness doesn't come for free. Also the nicer primitives that IMHO the really-useful multicore languages are building on are either impossible (Haskell) or too late (Go/Erlang) for Python to deeply embrace due to massive legacy libraries and code. If you want to make threading "work" in Python, go nuts, it will bring much benefit, but Python will simply never be the first choice for tasks in which multicore performance and safety is the first or second priority, rather than the fifth or sixth. And that's fine. I'd really rather see Python become a better Python than see it become a crappy Go or something.
- fijal 14y ago
  PyPy (on it's good day) is within 2x mark from the equivalent C. We're planning on closing this gap, but also we're planning on making every day the good day (right now you kind of have to know what you're doing in case you want to hit the sweet spot). This might be an interesting read for you: https://www2.cisl.ucar.edu/sites/default/files/2cameron_sparr_FINAL.pdf https://www2.cisl.ucar.edu/sites/default/files/2cameron_spar...
- deleted 14y ago
  [deleted]
igouy 14y ago
>> partly because we complained about unfair rules << That's not true.
- fijal 14y ago
  I take your word for it - I genuinely don't know, the correlation is however hard to miss.
  - igouy 14y ago
    I also stopped making measurements for Java -Xint, LuaJIT, Tracemonkey, CPython, Iron Python, and Ruby 1.8.7 Post hoc ergo propter hoc is a well known fallacy. >> I genuinely don't know << When you don't know, don't make accusations.

by fijal 14y ago

lmm 14y ago

Can you give details and/or a citation for the complaints and the banning? I use the shootout a lot to give me a rough idea of a specific language's properties, so if there are valid criticisms then I'd like to hear them, but your post doesn't really stand on its own to someone who doesn't know the backstory.

igouy 14y ago
>>a rough idea of a specific language's properties<< No, but maybe a rough idea of a specific language implementation's properties. >> if there are valid criticisms << There are plenty ;-) http://shootout.alioth.debian.org/dont-jump-to-conclusions.php http://shootout.alioth.debian.org/dont-jump-to-conclusions.p... Readers obviously jump to conclusions based on box plots and summary statistics -- when they really need to look at the individual tasks and look at the program source code, because there's so much variation. I think there would need to be incredibly tight constraints on minimum program run-time, source code size, and memory use; before summary statistics could be taken at face value.
fijal 14y ago
There is pypy-dev discussion about the subject that starts here: http://mail.python.org/pipermail/pypy-dev/2011-April/007139.html http://mail.python.org/pipermail/pypy-dev/2011-April/007139.... It's a long and boring read, but here you go. As in the actual complaints, as far as I remember, but this was a bit ago: * our versions got rejected because they perform slower on CPython (while being faster on PyPy) and because they're 'unobvious'. At the same time, the C implementation of a lot of things there is very unobvious. * we were not allowed to use array.array module, despite being in standard library. * you can use gmp library for C, CPython, but not PyPy via ctypes * C can use a bad random (that's fast, but also gives a really bad distribution), while PyPy not because Python comes with a random in stdlib * custom malloc libraries for C are fine, tuning JIT parameters per-benchmark is not. And various complaints like this. After some discussion, Isaac just kicked us out alltogether. Overall, my personal impression of the experience was that shootout is there to showcase things that are predetermined and we're trying to attack the status quo. This is obviously very personal, I don't have any actual data to back up this claim.
- igouy 14y ago
  READER BEWARE! >>our versions got rejected because they perform slower on CPython (while being faster on PyPy)<< Maciej Fijalkowski knows that I asked him to contribute a PyPy version of n-body because the written-for-CPython version failed with PyPy. Maciej Fijalkowski knows that his own PyPy version of n-body "performed slower on CPython (while being faster on PyPy)" -- AND WAS NOT REJECTED. http://mail.python.org/pipermail/pypy-dev/2011-April/007177.html http://mail.python.org/pipermail/pypy-dev/2011-April/007177.... Justify your accusation or take it back! >> you can use gmp library for C, CPython, but not PyPy via ctypes << Joe La Fata's pi-digits code worked first time on x86 and x64, on PyPy and CPython and Python3; and used ctypes to get to GMP -- AND WAS NOT REJECTED http://anonscm.debian.org/viewvc/shootout/shootout/bench/pidigits/pidigits.python3-6.python3?hideattic=0&revision=1.1&view=markup http://anonscm.debian.org/viewvc/shootout/shootout/bench/pid... Justify your accusation or take it back! >> C can use a bad random (that's fast, but also gives a really bad distribution), while PyPy not because Python comes with a random in stdlib << The fasta task uses random numbers, and every fasta program (including C and Python) implements the same random function. http://shootout.alioth.debian.org/u32/program.php?test=fasta&lang=gcc&id=4 http://shootout.alioth.debian.org/u32/program.php?test=fasta... http://shootout.alioth.debian.org/u32/program.php?test=fasta&lang=python3&id=2 http://shootout.alioth.debian.org/u32/program.php?test=fasta... Justify your accusation or take it back! ---- Tell Maciej Fijalkowski to justify his accusations!
  - fijal 14y ago
    Cool that you're linking to this mail, this shows yet another problem that I forgot about - that pypy extensions are uncool, while GCC extensions are cool (and say a custom malloc library for GCbench - after all the benchmark does not do anything, an empty malloc would do just fine).
  - fijal 14y ago
    Look Igor -- I don't care how things are now since PyPy got kicked out anyway, so I seriously don't care any more. gmpy uses C extension, it's not using ctypes. We disagreed over a ctypes version at some point. random - indeed it got fixed (and it's too late to edit a parent if you wonder)

jerf 14y ago

I was confining myself to CPython on the grounds that it is still what people complain about in terms of adding multicore to it. People don't complain about PyPy not having it since it's still developing, and it is not yet known whether PyPy will be fast enough to be worth worrying about multicore. To be honest I'm skeptical, but open to the possibility. But I rather suspect in the end that Python will forever be a slower language than the competition; all the dynamicness doesn't come for free. Also the nicer primitives that IMHO the really-useful multicore languages are building on are either impossible (Haskell) or too late (Go/Erlang) for Python to deeply embrace due to massive legacy libraries and code. If you want to make threading "work" in Python, go nuts, it will bring much benefit, but Python will simply never be the first choice for tasks in which multicore performance and safety is the first or second priority, rather than the fifth or sixth. And that's fine. I'd really rather see Python become a better Python than see it become a crappy Go or something.

fijal 14y ago
PyPy (on it's good day) is within 2x mark from the equivalent C. We're planning on closing this gap, but also we're planning on making every day the good day (right now you kind of have to know what you're doing in case you want to hit the sweet spot). This might be an interesting read for you: https://www2.cisl.ucar.edu/sites/default/files/2cameron_sparr_FINAL.pdf https://www2.cisl.ucar.edu/sites/default/files/2cameron_spar...
deleted 14y ago
[deleted]

igouy 14y ago

>> partly because we complained about unfair rules << That's not true.

fijal 14y ago
I take your word for it - I genuinely don't know, the correlation is however hard to miss.
- igouy 14y ago
  I also stopped making measurements for Java -Xint, LuaJIT, Tracemonkey, CPython, Iron Python, and Ruby 1.8.7 Post hoc ergo propter hoc is a well known fallacy. >> I genuinely don't know << When you don't know, don't make accusations.