11 ms·

PyPy 5.1 released

240 points by sandGorgon 11y ago

wyldfire 11y ago
> Numpy: "Fix transpose(arg)...add broadcast..." Does anyone know if there are some milestones in sight regarding packages which use numpy? e.g. "working scipy or sklearn". EDIT: BTW I forgot to add: tremendous thanks to PyPy team for your continued work on this project. I benefit from it and always look forward to the next release!
- mattip 11y ago
  (PyPy dev here) Thanks. As our C-API support improves, we hope to be able to support more of the scientific python stack with minimal upstream changes
  - sevensor 11y ago
    Hugely impressed that you now have broadcast and all ufuncs. As a user it seems like these constitute a big chunk of what makes numpy what it is.
justantheruser 11y ago
Are there plans to add Python 3.5 support?
- maxerickson 11y ago
  I am not involved with PyPy. Observing from the outside, the answer is yes, but it isn't a priority because there is no one paying for it. (They've largely used up the targeted crowdfunding; the unused money isn't enough to make a large amount of progress)
- wbond 11y ago
  Pypy3 is a separate project. From previous comments by the PyPy team, their customers are almost all targeting Python 2, hence Python 3 doesn’t get nearly as much attention. Pypy3’s last release was 2.4.0 in 2014. It targets Python 3.2, which is relatively unsupported compared to Python 3.3. Here is a recent comment by a core member of the team on future Pypy3 work: https://news.ycombinator.com/item?id=11262150 https://news.ycombinator.com/item?id=11262150.
  - skolos 11y ago
    > their customers are almost all targeting Python 2 Odd statement. How can they have any customers who target Python 3 if they don't support it?
    - pritambaral 11y ago
      Could be they never received as many (or many) potential customers. There have been a few services/apps I was looking to use but had to turn down after talking to them because they didn't support Linux (Xamarin, Codename One, etc.) They all said they didn't receive as many requests for Linux as they did for Mac or Windows. I guess this would be similar.
      - fijal 11y ago
        I think you're confusing cause and effect. In open source world, you can't charge for something that exists. It needs to be a need to do so. In order to push PyPy into putting significant effort from a core team - we would need exactly that. A big chunk of money dedicated to just that. We've spent all the money we got and we did get something out of it. But it needs to be more seriously supported in order to kickstart that. Obviously people are free on whatever they want in their free time, but unless there is enough to push the PyPy3 support to a significant postion, it's unlikely to move at a higher pace.
    - Nkdo 11y ago
      The vast majority of python codebases are still in python 2, even though many of the popular framework have been ported to 3, and there's no reason for most of them to bother with the amount of work it can be to port to py3 when they're happy with py2. Whether it be devs like Dropbox or open source software like NVDA, they will stay in the world of python 2 for the years to come. The amount of customers they could get by supporting python 3 is definitely not worth the effort unless they get paid explicitly for it. In terms of libraries, python 3 is "mostly" ready (though there are still issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas, which replicated GWT with Python instead of Java. Or Google AppEngine, which still shows no sign of moving to 3.) but large codebases outside of highly worked on web frameworks haven't moved on and there's not enough gain to justify the effort it takes to port them. The Python 3 debacle caused a lot of devs to also look at other languages for new projects. Old python 2 codebases will stay python 2, but new projects might not necessarily be python at all. The anger with Python 3 is real. See also: https://caremad.io/2015/04/a-year-of-pypi-downloads/ https://caremad.io/2015/04/a-year-of-pypi-downloads/ https://www.reddit.com/r/Python/comments/45sm94/what_are_the_most_recent_python_3_vs_python_2/czzy5wa https://www.reddit.com/r/Python/comments/45sm94/what_are_the... "Looking at these graphs, we can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%" I expect things like AppEngine to deprecate Python rather than port to 3, when/if python 2 reaches EOL and not enough people gather to maintain or even evolve that old branch. Python 3 users are a vocal bunch, but the war they're fighting is a lost one.
BuckRogers 11y ago
Wow. Just wow. Congrats to the PyPy team. It never stops. I stopped calling myself a Python programmer sometime ago and started saying I'm a PyPy programmer because this is the future of Python. The underlying basis for PyPy has big implications for a number of dynamic languages though so we should all be cheering it on and contributing. For those who want better Python3 support, there's only ONE answer so no need to keep begging. Donate more money to the project or donate your talents. Patches welcome!
- kzrdude 11y ago
  How can it be the future if Py 3 support is the side project?
  - sp332 11y ago
    After they get momentum by attracting people with their speed improvements, it will be a lot easier to attract talent and money to tackle Python 3. The reason Python 3 is on the back burner is that there simply isn't a large enough audience right now. That could change if they keep increasing their reach.
    - kzrdude 11y ago
      After they get momentum? But pypy has existed for a long time. They are in a maturity phase now, surely.
      - sp332 11y ago
        There are major bug fixes and compatibility and performance improvements in this release. They're also asking for a large amount of money to start a push to add Py 3 support. And NumPy support is still improving - they'll gain a lot of users as that becomes complete. None of this seems like "maturity phase", especially in terms of userbase or funding.
      - deleted 11y ago
        [deleted]
pdknsk 11y ago
I wish PyPy supported multiprocessing. It does theoretically, but imposes a significant penalty, so it's not in a practically usable state. https://bitbucket.org/pypy/pypy/issues/1538/ https://bitbucket.org/pypy/pypy/issues/1538/
- emidln 11y ago
  That's from almost 3 years ago. Have you ran the attached patch to check the benchmarks with a more current release? I don't know if that's been fixed, but it probably wouldn't take much time to test out.
  - pdknsk 11y ago
    I last tried my own code which uses multiprocessing a few weeks ago. If anything, it gets slower with each release.
    - tachion 11y ago
      Is this the same case with threading (like multiprocessing.dummy)?
canjobear 11y ago
Are there major technical issues that make it hard to do PyPy for Python 3.5, or is it just a matter of wanting to focus 100% on the fixed target of 2.7?
- PeCaN 11y ago
  The main issue is that PyPy's commercial target audience (people who need fast Python, particularly for numerical calculations) doesn't care about Python 3. There's not nearly as much potential funding for Python 3 as for Python 2. Technically, Python 3 is not any harder to implement than Python 2, it's just useless.
  - mangeletti 11y ago
    I think you're partially right about the reasons (the other half of the equation simply being the momentum of Python 2 efforts), but I think your last clause ("it's just useless") is incorrect. I can tell you for 100% certain that, if a project of mine could run on a version of "Python" that was even twice as fast, I would switch immediately, because 6 application servers could become 3. If the gains are even greater, that's tremendous. It's not just scientific computing that benefits, and in fact, a lot of the scientific / numerical analysis done in Python is really done in C / C ++ anyway (think Numpy, et al which delegate to compiled code for much of their expensive operations).
    - Nullabillity 11y ago
      PyPy isn't useless, Python 3 is.
    - srean 11y ago
      I think it comes down to the question how much funding would such folks be able to raise to support that effort. It is my understanding that PyPy is quite resource starved.
- sametmax 11y ago
  It's a matter of resources. I donated, and I invite everyone earning money with Python to do the same (http://pypy.org/py3donate.html http://pypy.org/py3donate.html). More generally, the Python community really lacks money compared to other ones. GO, PHP and JS all have bigs players spending a lot of cash on it. While some companies does invest in Python, they don't spend nearly the same amount on it, and the PSF has a very tigh budget. One of the reason is that Python is "good enough", and so people don't invest on it because they don't need more from it. While JS was so slow that Google spent millions to create the V8. It's sad, but being clean and robust and strongly community driven leads to a lack of funding for Python. I wish we had a Mark shuttlework for the language.
BuckRogers 11y ago
Because the future is whatever you're building software in today. For the vast majority that's still Python2. For those starting new projects that might be Go, Elixir, Python2 or Python3. Just because some entity like the PSF declares Python3 the future doesn't make it so. Even if some people buy into it and believe it. It has to take over from the bottom up and its been floundering since 3.0 landed in 2008.
- andrepd 11y ago
  I don't see why, other than laziness to migrate existing projects.
  - gdulli 11y ago
    It's not laziness, it's basic cost/benefit evaluation. Many feel that 2.7 isn't lacking in such a way that 3.x offers so great a benefit as to outweigh its costs (effort to convert, loss of some portion of the long tail of available packages.) Others disagree. The success of 2.7 is an amazing thing. Most languages have one great major version, some have zero, Python has two.
  - Twirrim 11y ago
    Laziness? It's non-trivial to migrate to Python 3 if you've spent any time building up any number of bespoke libraries. Look at it this way: 1) I can spend 2 days writing, testing, reviewing and pushing code that works perfectly fine in Python 2, or 2) I can spend a couple of weeks+ porting all the libraries I need for the tool to add python 3 support, maintaining backwards compatibility, fixing tests, reviewing etc. etc. etc., then spend 2 days to write the actual code I started out needing to do. Which one do you think I'm going to be able to justify, given there is always a backlog of work? Multiple weeks of developer time is a non-trivial expense. There has to be a significant technical advantage to justify the effort and for a lot of things that Python is used for there just isn't enough of one. I'm not anti-Python 3. I like it, in fact. Where I can use it, I do. It's just a lot of the time I can't because of all of the associated extra work involved.
- baq 11y ago
aidenn0 11y ago
Is IBM involved in PyPy at all? Seeing PPC64 and s390x on a list of supported targets is unusual for a language runtime.
- gsnedders 11y ago
  They funded the PPC and s390x ports, as far as I'm aware.
- DasIch 11y ago
  I don't think they're involved in the project but they have funded the work for supporting s390x[1]. [1]: http://morepypy.blogspot.de/2016/04/pypy-enterprise-edition.html http://morepypy.blogspot.de/2016/04/pypy-enterprise-edition....
haberman 11y ago
Startup time is ok. "Hello, World" runs in something like 0.08s on my computer. Not sure what this would be like for an app that actually has a lot of code. Startup time is a metric I care a lot about, because it speaks to whether it feels snappy to use for command-line applications. But holy crap the binary is enormous. I'm getting 22MB of text, compared with 991kb for CPython.
abc_lisper 11y ago
How compatible are third-party packages with PyPy? Is there a compatibility matrix for all the packages on pypi?
- wolf550e 11y ago
  http://packages.pypy.org http://packages.pypy.org The web server type stuff works. The scientific computing type stuff mostly doesn't work.
deleted 11y ago
[deleted]
sandGorgon 11y ago
Why does PyPy not get support from the big Python based giants - the Dropboxes or the Googles of the world. It seems to me that PyPy is struggling for funding. PyPy has the potential to become the canonical Python - if the entire team had their finances taken care of (and not have to do things like http://baroquesoftware.com/ http://baroquesoftware.com/ ). Is there a particular reason for this ?
- bkeroack 11y ago
  PyPy is in the unfortunate position of being both uninteresting to many of those who rely on Python (numpy, etc and Spark work fine and are fast without PyPy) and irrelevant to application developers who switched off Python because it was too slow (to things like Go, Rust, etc). (This isn't an insult to PyPy, btw--it's an amazing project that has accomplished a lot. I'm just explaining why it doesn't get the support one might expect.)
  - sandGorgon 11y ago
    but assuming the vast majority of python users are using it for web application and so is Dropbox/Google, it seems to me that PyPy has a marketing+PR problem than anything else.
    - pwang 11y ago
      That's not a fair assumption: a huge number of Python users are using it for some level of data processing and data analysis. And that percentage is growing each day, because Python is a very hot technology in data science.
rcarmo 11y ago
Had to happen. New PyPy release, new 2vs3 feud in comments. Can't we all just get along? Anyway, my $0.02: I'm looking forward to test this with Hy (hylang.org) and seeing if some of the module referencing is fixed. Makes for a great JITed LISP already.
davidchua 11y ago
Pardon my ignorance but what exactly makes PyPy different from standard CPython?

240 points by sandGorgon 11y ago

> Numpy: "Fix transpose(arg)...add broadcast..." Does anyone know if there are some milestones in sight regarding packages which use numpy? e.g. "working scipy or sklearn". EDIT: BTW I forgot to add: tremendous thanks to PyPy team for your continued work on this project. I benefit from it and always look forward to the next release!

mattip 11y ago
(PyPy dev here) Thanks. As our C-API support improves, we hope to be able to support more of the scientific python stack with minimal upstream changes
- sevensor 11y ago
  Hugely impressed that you now have broadcast and all ufuncs. As a user it seems like these constitute a big chunk of what makes numpy what it is.

justantheruser 11y ago

Are there plans to add Python 3.5 support?

maxerickson 11y ago
I am not involved with PyPy. Observing from the outside, the answer is yes, but it isn't a priority because there is no one paying for it. (They've largely used up the targeted crowdfunding; the unused money isn't enough to make a large amount of progress)
wbond 11y ago
Pypy3 is a separate project. From previous comments by the PyPy team, their customers are almost all targeting Python 2, hence Python 3 doesn’t get nearly as much attention. Pypy3’s last release was 2.4.0 in 2014. It targets Python 3.2, which is relatively unsupported compared to Python 3.3. Here is a recent comment by a core member of the team on future Pypy3 work: https://news.ycombinator.com/item?id=11262150 https://news.ycombinator.com/item?id=11262150.
- skolos 11y ago
  > their customers are almost all targeting Python 2 Odd statement. How can they have any customers who target Python 3 if they don't support it?
  - pritambaral 11y ago
    Could be they never received as many (or many) potential customers. There have been a few services/apps I was looking to use but had to turn down after talking to them because they didn't support Linux (Xamarin, Codename One, etc.) They all said they didn't receive as many requests for Linux as they did for Mac or Windows. I guess this would be similar.
    - fijal 11y ago
      I think you're confusing cause and effect. In open source world, you can't charge for something that exists. It needs to be a need to do so. In order to push PyPy into putting significant effort from a core team - we would need exactly that. A big chunk of money dedicated to just that. We've spent all the money we got and we did get something out of it. But it needs to be more seriously supported in order to kickstart that. Obviously people are free on whatever they want in their free time, but unless there is enough to push the PyPy3 support to a significant postion, it's unlikely to move at a higher pace.
  - Nkdo 11y ago
    The vast majority of python codebases are still in python 2, even though many of the popular framework have been ported to 3, and there's no reason for most of them to bother with the amount of work it can be to port to py3 when they're happy with py2. Whether it be devs like Dropbox or open source software like NVDA, they will stay in the world of python 2 for the years to come. The amount of customers they could get by supporting python 3 is definitely not worth the effort unless they get paid explicitly for it. In terms of libraries, python 3 is "mostly" ready (though there are still issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas, which replicated GWT with Python instead of Java. Or Google AppEngine, which still shows no sign of moving to 3.) but large codebases outside of highly worked on web frameworks haven't moved on and there's not enough gain to justify the effort it takes to port them. The Python 3 debacle caused a lot of devs to also look at other languages for new projects. Old python 2 codebases will stay python 2, but new projects might not necessarily be python at all. The anger with Python 3 is real. See also: https://caremad.io/2015/04/a-year-of-pypi-downloads/ https://caremad.io/2015/04/a-year-of-pypi-downloads/ https://www.reddit.com/r/Python/comments/45sm94/what_are_the_most_recent_python_3_vs_python_2/czzy5wa https://www.reddit.com/r/Python/comments/45sm94/what_are_the... "Looking at these graphs, we can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%" I expect things like AppEngine to deprecate Python rather than port to 3, when/if python 2 reaches EOL and not enough people gather to maintain or even evolve that old branch. Python 3 users are a vocal bunch, but the war they're fighting is a lost one.

BuckRogers 11y ago

Wow. Just wow. Congrats to the PyPy team. It never stops. I stopped calling myself a Python programmer sometime ago and started saying I'm a PyPy programmer because this is the future of Python. The underlying basis for PyPy has big implications for a number of dynamic languages though so we should all be cheering it on and contributing. For those who want better Python3 support, there's only ONE answer so no need to keep begging. Donate more money to the project or donate your talents. Patches welcome!

kzrdude 11y ago
How can it be the future if Py 3 support is the side project?
- sp332 11y ago
  After they get momentum by attracting people with their speed improvements, it will be a lot easier to attract talent and money to tackle Python 3. The reason Python 3 is on the back burner is that there simply isn't a large enough audience right now. That could change if they keep increasing their reach.
  - kzrdude 11y ago
    After they get momentum? But pypy has existed for a long time. They are in a maturity phase now, surely.
    - sp332 11y ago
      There are major bug fixes and compatibility and performance improvements in this release. They're also asking for a large amount of money to start a push to add Py 3 support. And NumPy support is still improving - they'll gain a lot of users as that becomes complete. None of this seems like "maturity phase", especially in terms of userbase or funding.
    - deleted 11y ago
      [deleted]

pdknsk 11y ago

I wish PyPy supported multiprocessing. It does theoretically, but imposes a significant penalty, so it's not in a practically usable state. https://bitbucket.org/pypy/pypy/issues/1538/ https://bitbucket.org/pypy/pypy/issues/1538/

emidln 11y ago
That's from almost 3 years ago. Have you ran the attached patch to check the benchmarks with a more current release? I don't know if that's been fixed, but it probably wouldn't take much time to test out.
- pdknsk 11y ago
  I last tried my own code which uses multiprocessing a few weeks ago. If anything, it gets slower with each release.
  - tachion 11y ago
    Is this the same case with threading (like multiprocessing.dummy)?

canjobear 11y ago

Are there major technical issues that make it hard to do PyPy for Python 3.5, or is it just a matter of wanting to focus 100% on the fixed target of 2.7?

PeCaN 11y ago
The main issue is that PyPy's commercial target audience (people who need fast Python, particularly for numerical calculations) doesn't care about Python 3. There's not nearly as much potential funding for Python 3 as for Python 2. Technically, Python 3 is not any harder to implement than Python 2, it's just useless.
- mangeletti 11y ago
  I think you're partially right about the reasons (the other half of the equation simply being the momentum of Python 2 efforts), but I think your last clause ("it's just useless") is incorrect. I can tell you for 100% certain that, if a project of mine could run on a version of "Python" that was even twice as fast, I would switch immediately, because 6 application servers could become 3. If the gains are even greater, that's tremendous. It's not just scientific computing that benefits, and in fact, a lot of the scientific / numerical analysis done in Python is really done in C / C ++ anyway (think Numpy, et al which delegate to compiled code for much of their expensive operations).
  - Nullabillity 11y ago
    PyPy isn't useless, Python 3 is.
  - srean 11y ago
    I think it comes down to the question how much funding would such folks be able to raise to support that effort. It is my understanding that PyPy is quite resource starved.
sametmax 11y ago
It's a matter of resources. I donated, and I invite everyone earning money with Python to do the same (http://pypy.org/py3donate.html http://pypy.org/py3donate.html). More generally, the Python community really lacks money compared to other ones. GO, PHP and JS all have bigs players spending a lot of cash on it. While some companies does invest in Python, they don't spend nearly the same amount on it, and the PSF has a very tigh budget. One of the reason is that Python is "good enough", and so people don't invest on it because they don't need more from it. While JS was so slow that Google spent millions to create the V8. It's sad, but being clean and robust and strongly community driven leads to a lack of funding for Python. I wish we had a Mark shuttlework for the language.

BuckRogers 11y ago

Because the future is whatever you're building software in today. For the vast majority that's still Python2. For those starting new projects that might be Go, Elixir, Python2 or Python3. Just because some entity like the PSF declares Python3 the future doesn't make it so. Even if some people buy into it and believe it. It has to take over from the bottom up and its been floundering since 3.0 landed in 2008.

andrepd 11y ago
I don't see why, other than laziness to migrate existing projects.
- gdulli 11y ago
  It's not laziness, it's basic cost/benefit evaluation. Many feel that 2.7 isn't lacking in such a way that 3.x offers so great a benefit as to outweigh its costs (effort to convert, loss of some portion of the long tail of available packages.) Others disagree. The success of 2.7 is an amazing thing. Most languages have one great major version, some have zero, Python has two.
- Twirrim 11y ago
  Laziness? It's non-trivial to migrate to Python 3 if you've spent any time building up any number of bespoke libraries. Look at it this way: 1) I can spend 2 days writing, testing, reviewing and pushing code that works perfectly fine in Python 2, or 2) I can spend a couple of weeks+ porting all the libraries I need for the tool to add python 3 support, maintaining backwards compatibility, fixing tests, reviewing etc. etc. etc., then spend 2 days to write the actual code I started out needing to do. Which one do you think I'm going to be able to justify, given there is always a backlog of work? Multiple weeks of developer time is a non-trivial expense. There has to be a significant technical advantage to justify the effort and for a lot of things that Python is used for there just isn't enough of one. I'm not anti-Python 3. I like it, in fact. Where I can use it, I do. It's just a lot of the time I can't because of all of the associated extra work involved.
baq 11y ago

aidenn0 11y ago

Is IBM involved in PyPy at all? Seeing PPC64 and s390x on a list of supported targets is unusual for a language runtime.

gsnedders 11y ago
They funded the PPC and s390x ports, as far as I'm aware.
DasIch 11y ago
I don't think they're involved in the project but they have funded the work for supporting s390x[1]. [1]: http://morepypy.blogspot.de/2016/04/pypy-enterprise-edition.html http://morepypy.blogspot.de/2016/04/pypy-enterprise-edition....

haberman 11y ago

Startup time is ok. "Hello, World" runs in something like 0.08s on my computer. Not sure what this would be like for an app that actually has a lot of code. Startup time is a metric I care a lot about, because it speaks to whether it feels snappy to use for command-line applications. But holy crap the binary is enormous. I'm getting 22MB of text, compared with 991kb for CPython.

abc_lisper 11y ago

How compatible are third-party packages with PyPy? Is there a compatibility matrix for all the packages on pypi?

wolf550e 11y ago
http://packages.pypy.org http://packages.pypy.org The web server type stuff works. The scientific computing type stuff mostly doesn't work.

deleted 11y ago

[deleted]

sandGorgon 11y ago

Why does PyPy not get support from the big Python based giants - the Dropboxes or the Googles of the world. It seems to me that PyPy is struggling for funding. PyPy has the potential to become the canonical Python - if the entire team had their finances taken care of (and not have to do things like http://baroquesoftware.com/ http://baroquesoftware.com/ ). Is there a particular reason for this ?

bkeroack 11y ago
PyPy is in the unfortunate position of being both uninteresting to many of those who rely on Python (numpy, etc and Spark work fine and are fast without PyPy) and irrelevant to application developers who switched off Python because it was too slow (to things like Go, Rust, etc). (This isn't an insult to PyPy, btw--it's an amazing project that has accomplished a lot. I'm just explaining why it doesn't get the support one might expect.)
- sandGorgon 11y ago
  but assuming the vast majority of python users are using it for web application and so is Dropbox/Google, it seems to me that PyPy has a marketing+PR problem than anything else.
  - pwang 11y ago
    That's not a fair assumption: a huge number of Python users are using it for some level of data processing and data analysis. And that percentage is growing each day, because Python is a very hot technology in data science.

rcarmo 11y ago

Had to happen. New PyPy release, new 2vs3 feud in comments. Can't we all just get along? Anyway, my $0.02: I'm looking forward to test this with Hy (hylang.org) and seeing if some of the module referencing is fixed. Makes for a great JITed LISP already.

davidchua 11y ago

Pardon my ignorance but what exactly makes PyPy different from standard CPython?