4 ms·

$ time wc big-data.csv 1252045 1252045 17805309 big-data.csv real 0m0.417

$ time wc big-data.csv 1252045 1252045 17805309 big-data.csv real 0m0.417s user 0m0.411s sys 0m0.004s $ time ./sum-ocaml big-data.csv 6143290.,13087196.,14473656.,50128757.,3765822.,1290420. real 0m3.680s user 0m3.666s sys 0m0.008s $ time python3 sum.py big-data.csv 6143290.0,13087196.0,14473656.0,50128757.0,3765822.0,1290420.0 real 0m7.186s user 0m7.131s sys 0m0.036s $ time python2 sum.py big-data.csv 6143290.0,13087196.0,14473656.0,50128757.0,3765822.0,1290420.0 real 0m5.146s user 0m5.111s sys 0m0.012s $ time pypy2 sum.py big-data.csv 6143290.0,13087196.0,14473656.0,50128757.0,3765822.0,1290420.0 real 0m2.754s user 0m2.697s sys 0m0.052s I really like ocaml, but have to admit, pypy is pretty amazing.

deleted 12y ago
[deleted]
pron 12y ago
And ZipPy -- a Python implementation for the JVM based on Truffle/Graal -- supposedly performs even better than pypy.
- fijal 12y ago
  [citation needed]. In the minute or so of idle browsing I did not find that claim on their website nor a claim how much python they actually implement.
  - pron 12y ago
    https://docs.google.com/spreadsheets/d/1fFMWcRIuPKt7wSAM5Ox9Rho4BBRhA5xgX6oemGhVxAA/edit#gid=1 https://docs.google.com/spreadsheets/d/1fFMWcRIuPKt7wSAM5Ox9... It seems like it's often much faster than PyPy (up to 4x) and almost never slower.
andor 12y ago
It completely depends on the application. I have yet to see a use case that benefits from PyPy. Here's an example I tried with CPython2, 3 and PyPy yesterday: It loads the most probable translations for all unigrams from a Moses phrase translation table (11GB) into a dict, and using that dict translates 73 MB of text word by word. Most of the time is spent filling the dict/reading the large file. $ time python literal_translation.py phrase-table en test4 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 1m14.345s user 1m9.836s sys 0m4.157s $ time python3 literal_translation.py phrase-table en test5 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 2m17.680s user 2m2.605s sys 0m4.987s $ time pypy literal_translation.py phrase-table en test6 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 1m13.347s user 1m0.033s sys 0m4.734s https://gist.github.com/andreasf/af6bdc00cf5a928712b5 https://gist.github.com/andreasf/af6bdc00cf5a928712b5
- 314 12y ago
  If your problem is memory-bound then no compiler will help you. Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.
  - unhammer 12y ago
    1m for 11GB sounds IO-bound, but yeah that's still not a use-case where switching to pypy will make a difference. But did you try using it as a service (ie. avoid the startup time) and timing the actual translation? At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.
cwyers 11y ago
I feel like if a program runs for less than 10 seconds, I don't much care if it's 2 seconds or 7 seconds. And a benchmark for a program like that tells me very little about how the various runtimes affect larger programs.

deleted 12y ago

[deleted]

pron 12y ago

And ZipPy -- a Python implementation for the JVM based on Truffle/Graal -- supposedly performs even better than pypy.

fijal 12y ago
[citation needed]. In the minute or so of idle browsing I did not find that claim on their website nor a claim how much python they actually implement.
- pron 12y ago
  https://docs.google.com/spreadsheets/d/1fFMWcRIuPKt7wSAM5Ox9Rho4BBRhA5xgX6oemGhVxAA/edit#gid=1 https://docs.google.com/spreadsheets/d/1fFMWcRIuPKt7wSAM5Ox9... It seems like it's often much faster than PyPy (up to 4x) and almost never slower.

andor 12y ago

It completely depends on the application. I have yet to see a use case that benefits from PyPy. Here's an example I tried with CPython2, 3 and PyPy yesterday: It loads the most probable translations for all unigrams from a Moses phrase translation table (11GB) into a dict, and using that dict translates 73 MB of text word by word. Most of the time is spent filling the dict/reading the large file. $ time python literal_translation.py phrase-table en test4 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 1m14.345s user 1m9.836s sys 0m4.157s $ time python3 literal_translation.py phrase-table en test5 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 2m17.680s user 2m2.605s sys 0m4.987s $ time pypy literal_translation.py phrase-table en test6 translation table contains 76885 elements ...................................................................... 070 ....................................................... 125 files translated real 1m13.347s user 1m0.033s sys 0m4.734s https://gist.github.com/andreasf/af6bdc00cf5a928712b5 https://gist.github.com/andreasf/af6bdc00cf5a928712b5

314 12y ago
If your problem is memory-bound then no compiler will help you. Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.
- unhammer 12y ago
  1m for 11GB sounds IO-bound, but yeah that's still not a use-case where switching to pypy will make a difference. But did you try using it as a service (ie. avoid the startup time) and timing the actual translation? At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.

cwyers 11y ago

I feel like if a program runs for less than 10 seconds, I don't much care if it's 2 seconds or 7 seconds. And a benchmark for a program like that tells me very little about how the various runtimes affect larger programs.