9 ms·

PyPy 2.5.0 released

297 points by cyber1 12y ago

tbrock 12y ago
I wish you could compile scons with pypi.
- ginko 12y ago
  What OS are you using scons on? I found that scons is way slower on Windows than it is on Linux.
- mikeash 12y ago
  A nitpick I only bring up because the similarity sometimes confuses me: you mean pypy, which is python-on-python done fast. Pypi is the Python Package Index, a separate thing.
  - tbrock 12y ago
    Yes sorry iPhone autocorrect, I'm talking about pypy
- mattip 12y ago
  You mean "I wish you could run scons with PyPY". It looks to me like scons does contents = open(file).read() where they should be doing with open(file) as fid: contents = fid.read()
  - anon4 12y ago
    What is the difference between the two?
    - fijal 12y ago
      It's documented in PyPy differences, but generally open('x').read() does not guarantee that a file is closed. In CPython it's not a problem due to refcounting, but in pypy, the file might be closed at some later stage. There is additionally a silly low number of open file descriptors, which limits how many files you can have open at once.
      - masklinn 12y ago
        > In CPython it's not a problem due to refcounting, but in pypy, the file might be closed at some later stage. Technically the guarantee in CPython is "soft" so it can be a problem, just rarely is: if the file gets referenced from an object accessible from a cycle the file won't be released (and closed) until the cycle-breaking GC is triggered.
    - objectified 12y ago
      The latter uses a Python context manager to open the file, the former doesn't. By using the open() context manager like this, you don't need to worry about closing the file yourself, since the context manager takes care of it. The code that runs inside the 'with' block gets yielded inside the context manager. The former version of the code does not store a reference to the file handle, thus cannot close the file handle. Of course you can close file handles without using a context manager too: f = open('foo.txt') contents = f.read() f.close() But I'd prefer this way: with open('foo.txt') as f: contents = f.read() Context managers can of course be used for all kinds of things. For more information, see https://docs.python.org/2/library/contextlib.html https://docs.python.org/2/library/contextlib.html
      - codefisher 12y ago
        The equivalent code with out the context manager is actually this: f = open('foo.txt') try: contents = f.read() finally: f.close() So yes, extra reason to prefer the context manager.
rcarmo 12y ago
FYI, you'll still need to compile a specific gevent branch if you want to use it with this. lxml built fine, uWSGI seems OK too (except for the lack of gevent workers in my build). Things seem adequately speedy, haven't investigated the network throughput tweaks yet.
- kapilvt 12y ago
  which branch is that? [update] https://travis-ci.org/gevent/gevent https://travis-ci.org/gevent/gevent passes w/ pypy fwiw.
  - rcarmo 12y ago
    Just tried the current pip package and it doesn't build for me on OSX or Ubuntu. I have a local snapshot of the version I used with 2.4.0, but can't remember where I got it from now.
deleted 12y ago
[deleted]
kbd 12y ago
Congrats to the PyPy team on what sounds like a pretty big release! Something in the release notes caught my eye: > The past months have seen pypy mature and grow, as rpython becomes the goto solution for writing fast dynamic language interpreters. I asked this question[1] on the Perl 6 thread from a few days ago but didn't get an answer. Does anyone know why on earth the Perl 6 folks created yet another dynamic language VM+JIT with MoarVM instead of taking advantage of all the great work done with PyPy? Does anyone know whether PyPy was even considered as a target before writing MoarVM? [1] https://news.ycombinator.com/item?id=8982229 https://news.ycombinator.com/item?id=8982229
- slimsag 12y ago
  I don't know, but just from your description it sounds like a case of NIH[1]. [1] http://en.wikipedia.org/wiki/Not_invented_here http://en.wikipedia.org/wiki/Not_invented_here
- bkeroack 12y ago
  Perl6 started long before PyPy (2000 vs ~2007).
  - scott_karana 12y ago
    And yet, work began on MoarVM only in 2012. http://en.wikipedia.org/wiki/MoarVM http://en.wikipedia.org/wiki/MoarVM
    - simula67 12y ago
      I think MOAR was designed specifically to run NQP efficiently. NQP is a subset of Perl 6 which the Perl 6 compiler is written in. That compiler takes a program written in Perl 6 and converts it into NQP. That NQP program is then compiled to different targets such as JVM, Parrot etc. This is my understanding, of course.
      - sitkack 12y ago
        Sounds like someone needs to write an NQP JIT in rpython.
mrmagooey 12y ago
Assuming trunk and 2.5.0 are roughly the same thing it seems like a decent performance increase http://speed.pypy.org/ http://speed.pypy.org/
- mikeash 12y ago
  I just tried it on a heavy Python workload I run regularly and it looks like a substantial improvement for my purposes. I'm just measuring by eye but for this particular code I'd say it's around a factor of two faster. This code is really dict-heavy so I'd guess I'm benefitting a lot from those improvements. I love this project and the continual improvements it puts out.
  - mattip 12y ago
    PyPy dev here. Could you give us some info on how to help you more, for instance, a small benchmark of where we are still slow? We love to hear how PyPy is being used "in the real world"
    - mikeash 12y ago
      I was too optimistic in my eyeballed assessment, but a real measurement shows there's still a good improvement. I gave my program a typical workload and tested with both PyPy 2.4.0 and 2.5.0, and the result was 13 minutes 31 seconds on 2.4.0, and 11:22 on 2.5.0. That's great! What would be the best way to profile this thing under PyPy to see where it's spending time? I'm barely familiar with Python profiling in general, and totally unfamiliar with PyPy specifically. As for how I'm using it, I belong to a glider club. For every weekend day we operate (which is every weekend from around the end of February through mid-December) we need to assign four club members to carry out various tasks for the day. (Specifically, we need a designated tow pilot, an instructor, a duty officer, and an assistant.) I'm the scheduling guy, and I wrote a Python program to generate the schedules for me. I put in various constraints (when people are unavailable, what days they prefer, etc.) and then the program uses a hill-climbing algorithm along with a bunch of scoring routines to find a good schedule. The actual workload operates on Schedule objects, which just contain a list of dates and the members assigned to each position on each date. Then I make a ton of duplicates, each with one adjustment, score them all, pick out the best, repeat. I can also optionally go for two adjustments, which takes much longer but gives better results, and that's what ends up taking 10+ minutes as above.
ngoldbaum 12y ago
Does anyone have any experience with numpypy? Is it useful for real work yet?
- Derbasti 12y ago
  In my experience, almost everything works. Also, workarounds are usually easy. http://buildbot.pypy.org/numpy-status/latest.html http://buildbot.pypy.org/numpy-status/latest.html
Fede_V 12y ago
I'm really looking forward to the numpy specific announcements. Numpy is THE basic building for every single scientific library - if pypy can get a high performance numpy, that will go a long way towards allowing scientific users to use pypy (there is still the detail of libraries that use the c-api to wrap c libraries, but cffi is pretty neat).
- rcthompson 12y ago
  If most of your code is numpy stuff, will you actually see a speedup from PyPy? (I mean hypothetically, once NumPyPy is properly optimized.)
  - jre 12y ago
    In my experience, you'll have some parts of your code in which you have to do a for loop or something else that cannot be (cleanly) expressed using numpy and that's where performance starts to suffer. The current solution is to use something like cython, but if pypy simplifies that, I think that's great.
  - dagss 12y ago
    Yes, ignoring that most people need a loop now and then, even pure NumPy-using code has a lot of potential. Consider "a + b + c + d". For large arrays, the problem is that it creates many temporary results of same size as original arrays that must be streamed over the memory bus. And since FLOPs are free and your computation is limited by memory bandwidth, you pay a large penalty for using NumPy (that gets worse as expression gets more complex). Or "a + a.T"... here you can get lots of speedup using basic tiling techniques, to fully use cache lines rather than read a cache line only to get one number and discard it. And so on. For numeric computation, there are large gains (like 2-10x) from properly using the memory bus, that NumPy can't take advantage of. So you have projects like numexpr and Theano that mainly focus on speeding up non-loop NumPy-using code.
  - Fede_V 12y ago
    It highly depends. If it's just large linear algebra operations, then it won't matter, since all the work will be done by BLAS. If it's lots of small operations, I think pypy can inline them and you might see a significant speed up.
deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
aidos 12y ago
"We would like to thank our donors for the continued support of the PyPy project, and for those who donate to our three sub-projects, as well as our volunteers and contributors (10 new commiters joined PyPy since the last release). We’ve shown quite a bit of progress, but we’re slowly running out of funds. Please consider donating more, or even better convince your employer to donate, so we can finish those projects! The three sub-projects are: " PyPy are doing such incredible work and they seem to ask for very little in the way of funding to make it happen (and they're very explicit about where they're going to spend the money). Why is it that there aren't more donations from the industry? Is it just a marketing issue? Do they need to do a snazzy kickstarter video to build the hype?
- fijal 12y ago
  PyPy dev here. I would be interested to explore the ideas what can be done to improve our funding situation. We seem to have moderate success in those fundraisers, but nothing REALLY huge.
  - aidos 12y ago
    Unfortunately, it's not something I have answers to. I guess I just look at the success of something like the Micro Python campaign [0] (also a cool project) and wonder if something similar could be achieved. There are plenty of companies that could benefit from PyPy doing well though. How many python servers do youtube / reddit etc run? With some investment in PyPy they could reduce their on-going costs. Youtube could easily dump $100k into the PyPy project, so what's stopping them? (I suspect the answer in that case is that Google just aren't interested) I'm in a startup that doesn't have money to invest in PyPy, but if we we had more cash we'd definitely look at putting some into PyPy (though our use case is the hairy NumPy side of things). I guess in the case of PyPy it's maybe less interesting to individuals but maybe there's some way of reframing it to make it interesting. In a way, I'm sort of the target market here, because I use python all the time and I haven't donated to PyPy. So why haven't I donated (other than being quite broke because I'm starting a business)? I think part of it is the urgency. There are 3 specific campaigns you're running [1] to get funding, along with funding for the general project itself. And the campaigns themselves are great - you have good overviews of the issues that you're solving, a realistic approach and a proven track record of being able to execute. But I don't know when those campaigns started and what the cut-off date is. They sort of feel ongoing and it feels like there's no loss in not donating right now. Compare with something like kickstarter, where you have a date by which if there hasn't been enough funding you may end up losing out on something really cool. Sorry for the rather rambling brain-dump, but maybe there's some useful perspective here. [0] https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers https://www.kickstarter.com/projects/214379695/micro-python-... [1] http://pypy.org/tmdonate2.html http://pypy.org/tmdonate2.html http://pypy.org/py3donate.html http://pypy.org/py3donate.html http://pypy.org/numpydonate.html http://pypy.org/numpydonate.html
ldng 12y ago
Now if only someone could fix swig to be compatible with Pypy ..
- sitkack 12y ago
  What packages are you using that use Swig? Can cffi or cppyy work for you?
  - ldng 12y ago
    I'm thinking of Mapscript bindings that allows to interact with Mapserver. They generate biding for several languages through Swig. Some projects uses Swig precisely because of that characteristic. Project with "low bandwidth" for whom maintaining different binding generator for every language they support would be painful. In my case, I could probably hack a minimal cffi binding. But solving the problem at the root would be a better solution for everyone. Swig already use some form of cffi for Common Lisp apparently so maybe python-cffi support could be derived from that. I don't know for sure, just thinking out loud.
    - fijal 12y ago
      The problem with SWIG is that current SWIG bindings would not be reusable. Any major project using SWIG for Python already has a bit of glue and this glue tends to be CPython C API. SWIG has other modes, but it has not been the case in the past.
      - ldng 12y ago
        Sure, but still, having a way within SWIG to generate both standard python binding and more cffi-friendly python could help transitioning. That way, new projects can still benefit from the strength of SWIG that allows them to propose bindings for multiple languages with a single tool. Because, let's not kid ourselves, using specific binding libs for every single language you want to support has a cost. And in that case backward compatibility does not matter much. All the while giving a transition path for projects still using what I would call the legacy way. Or those who want/need to poke in the CPython API.
bmoresbest55 12y ago
This is very exciting. I will have to look into using PyPy more regularly.

PyPy 2.5.0 released

297 points by cyber1 12y ago

tbrock 12y ago

I wish you could compile scons with pypi.

ginko 12y ago
What OS are you using scons on? I found that scons is way slower on Windows than it is on Linux.
mikeash 12y ago
A nitpick I only bring up because the similarity sometimes confuses me: you mean pypy, which is python-on-python done fast. Pypi is the Python Package Index, a separate thing.
- tbrock 12y ago
  Yes sorry iPhone autocorrect, I'm talking about pypy
mattip 12y ago
You mean "I wish you could run scons with PyPY". It looks to me like scons does contents = open(file).read() where they should be doing with open(file) as fid: contents = fid.read()
- anon4 12y ago
  What is the difference between the two?
  - fijal 12y ago
    It's documented in PyPy differences, but generally open('x').read() does not guarantee that a file is closed. In CPython it's not a problem due to refcounting, but in pypy, the file might be closed at some later stage. There is additionally a silly low number of open file descriptors, which limits how many files you can have open at once.
    - masklinn 12y ago
      > In CPython it's not a problem due to refcounting, but in pypy, the file might be closed at some later stage. Technically the guarantee in CPython is "soft" so it can be a problem, just rarely is: if the file gets referenced from an object accessible from a cycle the file won't be released (and closed) until the cycle-breaking GC is triggered.
  - objectified 12y ago
    The latter uses a Python context manager to open the file, the former doesn't. By using the open() context manager like this, you don't need to worry about closing the file yourself, since the context manager takes care of it. The code that runs inside the 'with' block gets yielded inside the context manager. The former version of the code does not store a reference to the file handle, thus cannot close the file handle. Of course you can close file handles without using a context manager too: f = open('foo.txt') contents = f.read() f.close() But I'd prefer this way: with open('foo.txt') as f: contents = f.read() Context managers can of course be used for all kinds of things. For more information, see https://docs.python.org/2/library/contextlib.html https://docs.python.org/2/library/contextlib.html
    - codefisher 12y ago
      The equivalent code with out the context manager is actually this: f = open('foo.txt') try: contents = f.read() finally: f.close() So yes, extra reason to prefer the context manager.

rcarmo 12y ago

FYI, you'll still need to compile a specific gevent branch if you want to use it with this. lxml built fine, uWSGI seems OK too (except for the lack of gevent workers in my build). Things seem adequately speedy, haven't investigated the network throughput tweaks yet.

kapilvt 12y ago
which branch is that? [update] https://travis-ci.org/gevent/gevent https://travis-ci.org/gevent/gevent passes w/ pypy fwiw.
- rcarmo 12y ago
  Just tried the current pip package and it doesn't build for me on OSX or Ubuntu. I have a local snapshot of the version I used with 2.4.0, but can't remember where I got it from now.

deleted 12y ago

[deleted]

kbd 12y ago

Congrats to the PyPy team on what sounds like a pretty big release! Something in the release notes caught my eye: > The past months have seen pypy mature and grow, as rpython becomes the goto solution for writing fast dynamic language interpreters. I asked this question[1] on the Perl 6 thread from a few days ago but didn't get an answer. Does anyone know why on earth the Perl 6 folks created yet another dynamic language VM+JIT with MoarVM instead of taking advantage of all the great work done with PyPy? Does anyone know whether PyPy was even considered as a target before writing MoarVM? [1] https://news.ycombinator.com/item?id=8982229 https://news.ycombinator.com/item?id=8982229

slimsag 12y ago
I don't know, but just from your description it sounds like a case of NIH[1]. [1] http://en.wikipedia.org/wiki/Not_invented_here http://en.wikipedia.org/wiki/Not_invented_here
bkeroack 12y ago
Perl6 started long before PyPy (2000 vs ~2007).
- scott_karana 12y ago
  And yet, work began on MoarVM only in 2012. http://en.wikipedia.org/wiki/MoarVM http://en.wikipedia.org/wiki/MoarVM
  - simula67 12y ago
    I think MOAR was designed specifically to run NQP efficiently. NQP is a subset of Perl 6 which the Perl 6 compiler is written in. That compiler takes a program written in Perl 6 and converts it into NQP. That NQP program is then compiled to different targets such as JVM, Parrot etc. This is my understanding, of course.
    - sitkack 12y ago
      Sounds like someone needs to write an NQP JIT in rpython.

mrmagooey 12y ago

Assuming trunk and 2.5.0 are roughly the same thing it seems like a decent performance increase http://speed.pypy.org/ http://speed.pypy.org/

mikeash 12y ago
I just tried it on a heavy Python workload I run regularly and it looks like a substantial improvement for my purposes. I'm just measuring by eye but for this particular code I'd say it's around a factor of two faster. This code is really dict-heavy so I'd guess I'm benefitting a lot from those improvements. I love this project and the continual improvements it puts out.
- mattip 12y ago
  PyPy dev here. Could you give us some info on how to help you more, for instance, a small benchmark of where we are still slow? We love to hear how PyPy is being used "in the real world"
  - mikeash 12y ago
    I was too optimistic in my eyeballed assessment, but a real measurement shows there's still a good improvement. I gave my program a typical workload and tested with both PyPy 2.4.0 and 2.5.0, and the result was 13 minutes 31 seconds on 2.4.0, and 11:22 on 2.5.0. That's great! What would be the best way to profile this thing under PyPy to see where it's spending time? I'm barely familiar with Python profiling in general, and totally unfamiliar with PyPy specifically. As for how I'm using it, I belong to a glider club. For every weekend day we operate (which is every weekend from around the end of February through mid-December) we need to assign four club members to carry out various tasks for the day. (Specifically, we need a designated tow pilot, an instructor, a duty officer, and an assistant.) I'm the scheduling guy, and I wrote a Python program to generate the schedules for me. I put in various constraints (when people are unavailable, what days they prefer, etc.) and then the program uses a hill-climbing algorithm along with a bunch of scoring routines to find a good schedule. The actual workload operates on Schedule objects, which just contain a list of dates and the members assigned to each position on each date. Then I make a ton of duplicates, each with one adjustment, score them all, pick out the best, repeat. I can also optionally go for two adjustments, which takes much longer but gives better results, and that's what ends up taking 10+ minutes as above.

ngoldbaum 12y ago

Does anyone have any experience with numpypy? Is it useful for real work yet?

Derbasti 12y ago
In my experience, almost everything works. Also, workarounds are usually easy. http://buildbot.pypy.org/numpy-status/latest.html http://buildbot.pypy.org/numpy-status/latest.html

Fede_V 12y ago

I'm really looking forward to the numpy specific announcements. Numpy is THE basic building for every single scientific library - if pypy can get a high performance numpy, that will go a long way towards allowing scientific users to use pypy (there is still the detail of libraries that use the c-api to wrap c libraries, but cffi is pretty neat).

rcthompson 12y ago
If most of your code is numpy stuff, will you actually see a speedup from PyPy? (I mean hypothetically, once NumPyPy is properly optimized.)
- jre 12y ago
  In my experience, you'll have some parts of your code in which you have to do a for loop or something else that cannot be (cleanly) expressed using numpy and that's where performance starts to suffer. The current solution is to use something like cython, but if pypy simplifies that, I think that's great.
- dagss 12y ago
  Yes, ignoring that most people need a loop now and then, even pure NumPy-using code has a lot of potential. Consider "a + b + c + d". For large arrays, the problem is that it creates many temporary results of same size as original arrays that must be streamed over the memory bus. And since FLOPs are free and your computation is limited by memory bandwidth, you pay a large penalty for using NumPy (that gets worse as expression gets more complex). Or "a + a.T"... here you can get lots of speedup using basic tiling techniques, to fully use cache lines rather than read a cache line only to get one number and discard it. And so on. For numeric computation, there are large gains (like 2-10x) from properly using the memory bus, that NumPy can't take advantage of. So you have projects like numexpr and Theano that mainly focus on speeding up non-loop NumPy-using code.
- Fede_V 12y ago
  It highly depends. If it's just large linear algebra operations, then it won't matter, since all the work will be done by BLAS. If it's lots of small operations, I think pypy can inline them and you might see a significant speed up.

deleted 12y ago

[deleted]

deleted 12y ago

[deleted]

aidos 12y ago

"We would like to thank our donors for the continued support of the PyPy project, and for those who donate to our three sub-projects, as well as our volunteers and contributors (10 new commiters joined PyPy since the last release). We’ve shown quite a bit of progress, but we’re slowly running out of funds. Please consider donating more, or even better convince your employer to donate, so we can finish those projects! The three sub-projects are: " PyPy are doing such incredible work and they seem to ask for very little in the way of funding to make it happen (and they're very explicit about where they're going to spend the money). Why is it that there aren't more donations from the industry? Is it just a marketing issue? Do they need to do a snazzy kickstarter video to build the hype?

fijal 12y ago
PyPy dev here. I would be interested to explore the ideas what can be done to improve our funding situation. We seem to have moderate success in those fundraisers, but nothing REALLY huge.
- aidos 12y ago
  Unfortunately, it's not something I have answers to. I guess I just look at the success of something like the Micro Python campaign [0] (also a cool project) and wonder if something similar could be achieved. There are plenty of companies that could benefit from PyPy doing well though. How many python servers do youtube / reddit etc run? With some investment in PyPy they could reduce their on-going costs. Youtube could easily dump $100k into the PyPy project, so what's stopping them? (I suspect the answer in that case is that Google just aren't interested) I'm in a startup that doesn't have money to invest in PyPy, but if we we had more cash we'd definitely look at putting some into PyPy (though our use case is the hairy NumPy side of things). I guess in the case of PyPy it's maybe less interesting to individuals but maybe there's some way of reframing it to make it interesting. In a way, I'm sort of the target market here, because I use python all the time and I haven't donated to PyPy. So why haven't I donated (other than being quite broke because I'm starting a business)? I think part of it is the urgency. There are 3 specific campaigns you're running [1] to get funding, along with funding for the general project itself. And the campaigns themselves are great - you have good overviews of the issues that you're solving, a realistic approach and a proven track record of being able to execute. But I don't know when those campaigns started and what the cut-off date is. They sort of feel ongoing and it feels like there's no loss in not donating right now. Compare with something like kickstarter, where you have a date by which if there hasn't been enough funding you may end up losing out on something really cool. Sorry for the rather rambling brain-dump, but maybe there's some useful perspective here. [0] https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers https://www.kickstarter.com/projects/214379695/micro-python-... [1] http://pypy.org/tmdonate2.html http://pypy.org/tmdonate2.html http://pypy.org/py3donate.html http://pypy.org/py3donate.html http://pypy.org/numpydonate.html http://pypy.org/numpydonate.html

ldng 12y ago

Now if only someone could fix swig to be compatible with Pypy ..

sitkack 12y ago
What packages are you using that use Swig? Can cffi or cppyy work for you?
- ldng 12y ago
  I'm thinking of Mapscript bindings that allows to interact with Mapserver. They generate biding for several languages through Swig. Some projects uses Swig precisely because of that characteristic. Project with "low bandwidth" for whom maintaining different binding generator for every language they support would be painful. In my case, I could probably hack a minimal cffi binding. But solving the problem at the root would be a better solution for everyone. Swig already use some form of cffi for Common Lisp apparently so maybe python-cffi support could be derived from that. I don't know for sure, just thinking out loud.
  - fijal 12y ago
    The problem with SWIG is that current SWIG bindings would not be reusable. Any major project using SWIG for Python already has a bit of glue and this glue tends to be CPython C API. SWIG has other modes, but it has not been the case in the past.
    - ldng 12y ago
      Sure, but still, having a way within SWIG to generate both standard python binding and more cffi-friendly python could help transitioning. That way, new projects can still benefit from the strength of SWIG that allows them to propose bindings for multiple languages with a single tool. Because, let's not kid ourselves, using specific binding libs for every single language you want to support has a cost. And in that case backward compatibility does not matter much. All the while giving a transition path for projects still using what I would call the legacy way. Or those who want/need to poke in the CPython API.

bmoresbest55 12y ago

This is very exciting. I will have to look into using PyPy more regularly.