4 ms·

It completely depends on the application. I have yet to see a use case that benefits from PyPy. Here's an example I tried with CPython2, 3 and PyPy yesterday: I

by andor 11y ago

It completely depends on the application. I have yet to see a use case that benefits from PyPy. Here's an example I tried with CPython2, 3 and PyPy yesterday: It loads the most probable translations for all unigrams from a Moses phrase translation table (11GB) into a dict, and using that dict translates 73 MB of text word by word. Most of the time is spent filling the dict/reading the large file. $ time python literal_translation.py phrase-table en test4 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 1m14.345s user 1m9.836s sys 0m4.157s $ time python3 literal_translation.py phrase-table en test5 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 2m17.680s user 2m2.605s sys 0m4.987s $ time pypy literal_translation.py phrase-table en test6 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 1m13.347s user 1m0.033s sys 0m4.734s https://gist.github.com/andreasf/af6bdc00cf5a928712b5 https://gist.github.com/andreasf/af6bdc00cf5a928712b5

314 11y ago
If your problem is memory-bound then no compiler will help you. Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.
- unhammer 11y ago
  1m for 11GB sounds IO-bound, but yeah that's still not a use-case where switching to pypy will make a difference. But did you try using it as a service (ie. avoid the startup time) and timing the actual translation? At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.
  - andor 11y ago
    No, I didn't try to optimize this any further. I just checked whether any of the new interpreters is really faster. It's a simple one-off experiment, and compared to real statistical machine translation with Moses on the same data the translation is pretty much instant. Moses needs something on the order of 1600 CPU-hours (50 nodes * 8 threads * 4 hours).
andor 11y ago
To whoever downvoted this -- how about posting a more comprehensive set of benchmarks instead? My single benchmark is not in any way less meaningful than the parents. But as I said, I often try PyPy on longer-running Python code, and so far saw no significant gains.
- fijal 11y ago
  Your benchmark is terrible - it's impossible to run it because you don't provide a dataset that you used (or I'm missing some description). It also stresses one aspect of the language (processing strings, in a very specific way) and yet you claim this is representative. However, we're more than happy to consider it a pypy bug if we're not faster than CPython. Please provide clearer instructions how to reproduce it and ideally also other benchmarks that did not work for you.
  - brendano 11y ago
    "Terrible" is a little harsh. I found the benchmark to be interesting because it's similar to many NLP-style analysis tasks I have myself. Of course reproducible ones would be great too.
  - andor 11y ago
    Great, next time I'll hit your bug tracker directly ;-) The script and data are available here: http://zentrale1.com/~an/literal-translation.tar.xz http://zentrale1.com/~an/literal-translation.tar.xz The texts were scraped from the European Parliament web site. The phrase table was built with Moses from the Europarl parallel corpus [0] minus the texts. The script itself was supposed the be baseline input for another experiment, basically it's the worst way of translating something i could think of. [0] http://statmt.org/europarl/ http://statmt.org/europarl/
    - fijal 11y ago
      can you please provide me the data in somehow reproducible form (I have no clue how to use all the tools you mentioned for example), ideally as a .tar.bz2 file somewhere I can download and run without any prior knowledge. I'm happy to improve pypy for that particular use case.
GFK_of_xmaspast 11y ago
I notice python3 is twice as slow.
syllogism 11y ago
NLP stuff tends to be more data-bound, which PyPy can't really help with. If it's worth the effort, use Cython, design your data structures carefully, and you'll get C level performance.
- andor 11y ago
  Hey, thanks for spaCy! It's a great library.
zo1 11y ago
This isn't an isolated test, and tells us almost nothing. Because you've clumped together IO as well as the bulk of the actual processing work. Rather, when you do the test, only test the "mapping" part, and only test the "IO" part. That way you can clearly see the difference between the two.

by andor 11y ago

314 11y ago

If your problem is memory-bound then no compiler will help you. Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.

unhammer 11y ago
1m for 11GB sounds IO-bound, but yeah that's still not a use-case where switching to pypy will make a difference. But did you try using it as a service (ie. avoid the startup time) and timing the actual translation? At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.
- andor 11y ago
  No, I didn't try to optimize this any further. I just checked whether any of the new interpreters is really faster. It's a simple one-off experiment, and compared to real statistical machine translation with Moses on the same data the translation is pretty much instant. Moses needs something on the order of 1600 CPU-hours (50 nodes * 8 threads * 4 hours).

andor 11y ago

To whoever downvoted this -- how about posting a more comprehensive set of benchmarks instead? My single benchmark is not in any way less meaningful than the parents. But as I said, I often try PyPy on longer-running Python code, and so far saw no significant gains.

fijal 11y ago
Your benchmark is terrible - it's impossible to run it because you don't provide a dataset that you used (or I'm missing some description). It also stresses one aspect of the language (processing strings, in a very specific way) and yet you claim this is representative. However, we're more than happy to consider it a pypy bug if we're not faster than CPython. Please provide clearer instructions how to reproduce it and ideally also other benchmarks that did not work for you.
- brendano 11y ago
  "Terrible" is a little harsh. I found the benchmark to be interesting because it's similar to many NLP-style analysis tasks I have myself. Of course reproducible ones would be great too.
- andor 11y ago
  Great, next time I'll hit your bug tracker directly ;-) The script and data are available here: http://zentrale1.com/~an/literal-translation.tar.xz http://zentrale1.com/~an/literal-translation.tar.xz The texts were scraped from the European Parliament web site. The phrase table was built with Moses from the Europarl parallel corpus [0] minus the texts. The script itself was supposed the be baseline input for another experiment, basically it's the worst way of translating something i could think of. [0] http://statmt.org/europarl/ http://statmt.org/europarl/
  - fijal 11y ago
    can you please provide me the data in somehow reproducible form (I have no clue how to use all the tools you mentioned for example), ideally as a .tar.bz2 file somewhere I can download and run without any prior knowledge. I'm happy to improve pypy for that particular use case.

GFK_of_xmaspast 11y ago

I notice python3 is twice as slow.

syllogism 11y ago

NLP stuff tends to be more data-bound, which PyPy can't really help with. If it's worth the effort, use Cython, design your data structures carefully, and you'll get C level performance.

andor 11y ago
Hey, thanks for spaCy! It's a great library.

zo1 11y ago

This isn't an isolated test, and tells us almost nothing. Because you've clumped together IO as well as the bulk of the actual processing work. Rather, when you do the test, only test the "mapping" part, and only test the "IO" part. That way you can clearly see the difference between the two.