14 ms·

PyPy 1.6 Released - Full Python 2.7.1 Implementation

244 points by jparise 15y ago

voyvf 15y ago
> has beta level support for loading CPython C extensions. Is this via ctypes, or "real" support in much the same as how CPython would behave? I ask because this is one of the features that I've been waiting (impatiently) for - I've run some Flask projects using PyPy and gunicorn, and love how fast it goes, but really want to be able to use the rest of my codebase, which unfortunately does rely on some C (and Cython) extensions. (:
- vgnet 15y ago
  It's "real" support, using your criteria. However, it's slower in PyPy than in CPython and than what the same thing based on ctypes would be on PyPy. If there's a pure-Python version of the C extension, it might be faster on PyPy than the C extension support that cpyext (PyPy C-API compatibility module) provides. Cython-based code is currently incompatible (it goes well beyond the public C-API), but a GSoC project to generate ctypes-based pure-Python code from Cython is (was?) going on.
  - voyvf 15y ago
    > It's "real" support, using your criteria. However, it's slower in PyPy than in CPython and than what the same thing based on ctypes would be on PyPy. > If there's a pure-Python version of the C extension, it might be faster on PyPy than the C extension support that cpyext (PyPy C-API compatibility module) provides. That's fine, the mere ability to run them will be nice. (: > Cython-based code is currently incompatible (it goes well beyond the public C-API), but a GSoC project to generate ctypes-based pure-Python code from Cython is (was?) going on. I remember that GSoC! Any idea as to what happened to it, if it was scrapped, or whatever?
    - vgnet 15y ago
      > I remember that GSoC! Any idea as to what happened to it, > if it was scrapped, or whatever? I know it made a lot of progress in Cython-land (updates at http://rguillebert.blogspot.com/ http://rguillebert.blogspot.com/) but no real overview or high level status update has been given yet (AFAIK). Since it's from GSoC 2011, it's about time for that to happen... and for other people to start contributing code <hint, hint> :).
- pjenvey 15y ago
  Not via ctypes, via CPyExt, a layer emulating the CPython C extension API (it actually implements a reference counting GC and everything)
sylvinus 15y ago
I'm always blown away by the consistent performance gains they reach with each new version. Congrats!
kristofferR 15y ago
In general I'm very happy with my choice of Ruby/Rails instead of Python/Django, but PyPy is one of the few things I envy Python developers for. I wish something similar could be developed for Ruby.
- postfuturist 15y ago
  http://rubini.us/ http://rubini.us/
- lobo_tuerto 15y ago
  Maybe rubinius will fill this need? http://rubini.us/ http://rubini.us/ Also, this could be an interesting read: http://www.engineyard.com/blog/2010/making-ruby-fast-the-rubinius-jit/ http://www.engineyard.com/blog/2010/making-ruby-fast-the-rub...
  - bryanwb 15y ago
    but what is the status of rubinius? is it nearly as far along as pypy? I would think that the metaprogramming features of ruby which make it so much fun would also make it n times harder to build a JIT for.
    - lobo_tuerto 15y ago
      I think the second link I posted in my previous comment could help you get a grasp about what they are doing in respect to the dynamic nature of the language and JIT. Here is the current status towards 1.9 implementation: http://status.rubini.us/ http://status.rubini.us/ From the frontpage: "How compatible is Rubinius? From the start, compatibility has been critical to us. To that end, we created the RubySpec to ensure that we maintained parity with official Ruby. We are currently at a 93% RubySpec pass rate and growing everyday. For now Rubinius is targeting MRI 1.8.7 (1.9 is on the post 1.0 list). Most Gems, Rails plugins and C-Extensions work right out of the box. If you find a bug, let us know and we'll get on top of it."
    - tedunangst 15y ago
      The dynamic features of ruby are not so different than those of python, lua, or javascript. Actually in terms of reasonable memory consumption I think rubinius is quite a bit better than pypy.
lliiffee 15y ago
Can I take advantage of this thread to ask the HN crowd a technical question? Some time ago, I implemented an automatic differentiation tool. Using operator overloading on a special "autodouble" type the tool would trace the execution of a block of numerical code. Then, some calculus would automatically happen, and it would output and compile fast c-code that would compute the original function and derivatives in pure c. This was great, except the c-code that was output was freaking gigantic (like hundreds or thousands of megabytes) albeit very simple, and so the c compiler would take forever to run. Sigh. My question is: could I leverage pypy somehow to avoid this? Can I output RPython? Can I output whatever RPython is compiled down to instead? Can I do this with no more than, say, a 3x penalty compared to c? (I apologize for asking a question only marginally related to the particular article here...)
- carterschonwald 15y ago
  Off hand and guessing about the problem: You might want to look at approaches that use dual numbers. Likewise, in instead of inlining the procedures, generate a differentiated version of each procedure with a new name. If those don't cover your problems, perhaps look at how other autodiff tools for C do it?
  - lliiffee 15y ago
    For being offhand, those are very good guesses! Dual numbers won't be efficient, as I want reverse-mode autodiff. As to the multiple procedures: Well, as I was doing it, even a single procedure can be many hundreds of megabytes large. Even for very very simple code, however, I noticed that GCC was superlinear in code size. I suppose I could somewhat arbitrarily break up the code into arbitrary functions. I wonder if that would speed things up? Other autodiff tools: Well, they basically trace execution, but then run an interpreter instead of trying to actual generate compiled code. I wanted to both be faster than that and have the wonderful experience of writing pure python...
    - carterschonwald 15y ago
      it sounds like you're getting lots of code duplication. 1) try running a common subexpression elimination process on your code before doing the autodiffing, and create a procedure for each shared expression 2) for prim ops, again, have a procedure created for the diffed version instead of inlining, and sub in the procedure instead. perhaps something like these ideas would help If you want an example of a nice high level Auto diff lib, a nice one that works via operator overloading is http://hackage.haskell.org/package/ad http://hackage.haskell.org/package/ad , which seems quite nice though I've not had the opportunity to use it myself. yes, optimizing compilers such as gcc use algorithms that are superlinear in code size when they're optimizing. Perhaps you should instead try out the operator overloading approach (and see if you can )? gl :-) Aside: When I hear the phrase execution trace in the context of program analysis, i think abstract interpretation, though I'm not sure if thats relevant for you. cheers!
socratic 15y ago
Are there production users of PyPy? I feel like PyPy has always been the most academically interesting Python implementation. But has it taken away mindshare from CPython?
- vgnet 15y ago
  Yes, there are. Recently Quora announced it was running on PyPy[1]. Some other disclosures were made (a Django project[2], LWN internal processing[3], tweets about speedups in production, etc.), but the PyPy team is thinking about officially asking for success stories in the near future[4]. [1]: http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-now-running-on-PyPy http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-now-... [2]: https://convore.com/python/whos-using-pypy-in-production/ https://convore.com/python/whos-using-pypy-in-production/ [3]: http://lwn.net/Articles/442268/ http://lwn.net/Articles/442268/ [4]: https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/success_stories.rst https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/succe...
  - socratic 15y ago
    To me, this list seems to suggest the opposite actually. [1] is the company where one of the main developers of PyPy works. [2] appears to be a discussion where they are looking for anyone using PyPy in production. [3] seems to be an article about someone experimenting with PyPy for git processing. [4] is the developers looking for non-toy examples of production use so that they can get more funding.
    - vgnet 15y ago
      It suggests that there aren't many production users, true. But it does prove that there are some.
      - socratic 15y ago
        Indeed. Though two-ish production users after 7 years of development hardly seems like a success. By way of contrast, (1) in the Ruby world, YARV went from an alternative implementation to the official implementation within two years, and (2) in the JavaScript world, node.js is similarly being used in production in tons of places after only two years. (Though I realize that those examples aren't exact parallels.) This isn't meant to criticize your response, rather, I find it interesting that the Python world seems to have so many alternative implementations (PyPy, IronPython, Jython, Cython) despite what appears to be really minority mindshare compared to CPython.
sho_hn 15y ago
I'm still bummed at being stuck with the dilemma of having to chose between CPython 3.x and PyPy. PyPy with Py3k support would rock.

244 points by jparise 15y ago

voyvf 15y ago

> has beta level support for loading CPython C extensions. Is this via ctypes, or "real" support in much the same as how CPython would behave? I ask because this is one of the features that I've been waiting (impatiently) for - I've run some Flask projects using PyPy and gunicorn, and love how fast it goes, but really want to be able to use the rest of my codebase, which unfortunately does rely on some C (and Cython) extensions. (:

vgnet 15y ago
It's "real" support, using your criteria. However, it's slower in PyPy than in CPython and than what the same thing based on ctypes would be on PyPy. If there's a pure-Python version of the C extension, it might be faster on PyPy than the C extension support that cpyext (PyPy C-API compatibility module) provides. Cython-based code is currently incompatible (it goes well beyond the public C-API), but a GSoC project to generate ctypes-based pure-Python code from Cython is (was?) going on.
- voyvf 15y ago
  > It's "real" support, using your criteria. However, it's slower in PyPy than in CPython and than what the same thing based on ctypes would be on PyPy. > If there's a pure-Python version of the C extension, it might be faster on PyPy than the C extension support that cpyext (PyPy C-API compatibility module) provides. That's fine, the mere ability to run them will be nice. (: > Cython-based code is currently incompatible (it goes well beyond the public C-API), but a GSoC project to generate ctypes-based pure-Python code from Cython is (was?) going on. I remember that GSoC! Any idea as to what happened to it, if it was scrapped, or whatever?
  - vgnet 15y ago
    > I remember that GSoC! Any idea as to what happened to it, > if it was scrapped, or whatever? I know it made a lot of progress in Cython-land (updates at http://rguillebert.blogspot.com/ http://rguillebert.blogspot.com/) but no real overview or high level status update has been given yet (AFAIK). Since it's from GSoC 2011, it's about time for that to happen... and for other people to start contributing code <hint, hint> :).
pjenvey 15y ago
Not via ctypes, via CPyExt, a layer emulating the CPython C extension API (it actually implements a reference counting GC and everything)

sylvinus 15y ago

I'm always blown away by the consistent performance gains they reach with each new version. Congrats!

kristofferR 15y ago

In general I'm very happy with my choice of Ruby/Rails instead of Python/Django, but PyPy is one of the few things I envy Python developers for. I wish something similar could be developed for Ruby.

postfuturist 15y ago
http://rubini.us/ http://rubini.us/
lobo_tuerto 15y ago
Maybe rubinius will fill this need? http://rubini.us/ http://rubini.us/ Also, this could be an interesting read: http://www.engineyard.com/blog/2010/making-ruby-fast-the-rubinius-jit/ http://www.engineyard.com/blog/2010/making-ruby-fast-the-rub...
- bryanwb 15y ago
  but what is the status of rubinius? is it nearly as far along as pypy? I would think that the metaprogramming features of ruby which make it so much fun would also make it n times harder to build a JIT for.
  - lobo_tuerto 15y ago
    I think the second link I posted in my previous comment could help you get a grasp about what they are doing in respect to the dynamic nature of the language and JIT. Here is the current status towards 1.9 implementation: http://status.rubini.us/ http://status.rubini.us/ From the frontpage: "How compatible is Rubinius? From the start, compatibility has been critical to us. To that end, we created the RubySpec to ensure that we maintained parity with official Ruby. We are currently at a 93% RubySpec pass rate and growing everyday. For now Rubinius is targeting MRI 1.8.7 (1.9 is on the post 1.0 list). Most Gems, Rails plugins and C-Extensions work right out of the box. If you find a bug, let us know and we'll get on top of it."
  - tedunangst 15y ago
    The dynamic features of ruby are not so different than those of python, lua, or javascript. Actually in terms of reasonable memory consumption I think rubinius is quite a bit better than pypy.

lliiffee 15y ago

Can I take advantage of this thread to ask the HN crowd a technical question? Some time ago, I implemented an automatic differentiation tool. Using operator overloading on a special "autodouble" type the tool would trace the execution of a block of numerical code. Then, some calculus would automatically happen, and it would output and compile fast c-code that would compute the original function and derivatives in pure c. This was great, except the c-code that was output was freaking gigantic (like hundreds or thousands of megabytes) albeit very simple, and so the c compiler would take forever to run. Sigh. My question is: could I leverage pypy somehow to avoid this? Can I output RPython? Can I output whatever RPython is compiled down to instead? Can I do this with no more than, say, a 3x penalty compared to c? (I apologize for asking a question only marginally related to the particular article here...)

carterschonwald 15y ago
Off hand and guessing about the problem: You might want to look at approaches that use dual numbers. Likewise, in instead of inlining the procedures, generate a differentiated version of each procedure with a new name. If those don't cover your problems, perhaps look at how other autodiff tools for C do it?
- lliiffee 15y ago
  For being offhand, those are very good guesses! Dual numbers won't be efficient, as I want reverse-mode autodiff. As to the multiple procedures: Well, as I was doing it, even a single procedure can be many hundreds of megabytes large. Even for very very simple code, however, I noticed that GCC was superlinear in code size. I suppose I could somewhat arbitrarily break up the code into arbitrary functions. I wonder if that would speed things up? Other autodiff tools: Well, they basically trace execution, but then run an interpreter instead of trying to actual generate compiled code. I wanted to both be faster than that and have the wonderful experience of writing pure python...
  - carterschonwald 15y ago
    it sounds like you're getting lots of code duplication. 1) try running a common subexpression elimination process on your code before doing the autodiffing, and create a procedure for each shared expression 2) for prim ops, again, have a procedure created for the diffed version instead of inlining, and sub in the procedure instead. perhaps something like these ideas would help If you want an example of a nice high level Auto diff lib, a nice one that works via operator overloading is http://hackage.haskell.org/package/ad http://hackage.haskell.org/package/ad , which seems quite nice though I've not had the opportunity to use it myself. yes, optimizing compilers such as gcc use algorithms that are superlinear in code size when they're optimizing. Perhaps you should instead try out the operator overloading approach (and see if you can )? gl :-) Aside: When I hear the phrase execution trace in the context of program analysis, i think abstract interpretation, though I'm not sure if thats relevant for you. cheers!

socratic 15y ago

Are there production users of PyPy? I feel like PyPy has always been the most academically interesting Python implementation. But has it taken away mindshare from CPython?

vgnet 15y ago
Yes, there are. Recently Quora announced it was running on PyPy[1]. Some other disclosures were made (a Django project[2], LWN internal processing[3], tweets about speedups in production, etc.), but the PyPy team is thinking about officially asking for success stories in the near future[4]. [1]: http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-now-running-on-PyPy http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-now-... [2]: https://convore.com/python/whos-using-pypy-in-production/ https://convore.com/python/whos-using-pypy-in-production/ [3]: http://lwn.net/Articles/442268/ http://lwn.net/Articles/442268/ [4]: https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/success_stories.rst https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/succe...
- socratic 15y ago
  To me, this list seems to suggest the opposite actually. [1] is the company where one of the main developers of PyPy works. [2] appears to be a discussion where they are looking for anyone using PyPy in production. [3] seems to be an article about someone experimenting with PyPy for git processing. [4] is the developers looking for non-toy examples of production use so that they can get more funding.
  - vgnet 15y ago
    It suggests that there aren't many production users, true. But it does prove that there are some.
    - socratic 15y ago
      Indeed. Though two-ish production users after 7 years of development hardly seems like a success. By way of contrast, (1) in the Ruby world, YARV went from an alternative implementation to the official implementation within two years, and (2) in the JavaScript world, node.js is similarly being used in production in tons of places after only two years. (Though I realize that those examples aren't exact parallels.) This isn't meant to criticize your response, rather, I find it interesting that the Python world seems to have so many alternative implementations (PyPy, IronPython, Jython, Cython) despite what appears to be really minority mindshare compared to CPython.

sho_hn 15y ago

I'm still bummed at being stuck with the dilemma of having to chose between CPython 3.x and PyPy. PyPy with Py3k support would rock.