6 ms·

PyPy funded to begin support for Python 3 and Numpy

178 points by synparb 15y ago

ColdAsIce 15y ago
I like pypy and its ambitions, last time I tested it, about a month ago it was very speedy and startup time considerable faster than Cpython. However the regex exercises I wanted to do couldnt be done. Pypy seemingly didnt have a good regexp engine. If I remember correctly, something with groups and backwards-reference... Has that changed?
- bitcracker 15y ago
  The fundamental question is: How compatible will PyPy ever be? Which kind of applications can be run with PyPy? http://pypy.org/compat.html http://pypy.org/compat.html mentions compatiblity according to the standard library. This is fine for (web) servers and command line applications. But what about desktop applications? Can I (someday) take a PyQt or PyGTK code and compile it with PyPy without modifications?
  - dripton 15y ago
    Maybe. cpyext is a hack to let PyPy run Python/C API modules. It works for some, but not all, and it's slow. I'm sure it'll keep getting better, but not sure if it'll ever be good enough to run PyGTK or PyQt. PyPy has good support for ctypes, so ctypes bindings are a good option. There are projects out there like pygir-ctypes and ctypes-gtk. One of them just needs to become complete enough to be a good choice for GTK programming. Compatibility with new PyGObject is more likely than compatibility with legacy PyGTK, though. PyQt is harder because it's C++.
  - DasIch 15y ago
    I believe that C extensions, which have always been and still are very CPython specific, will have to be replaced with ctypes and possibly Cython versions eventually. I expect to see quite a lot of improvement in that area as a side effect of the work on numpy.
- kingkilr 15y ago
  No, not much has changed about our regex engine in the last month. However, our `re` module is fully compatible with CPython's so I'm bit confused, are you saying it didn't run something, or was slow?
- ak217 15y ago
  Hi, could you give some details about those regexes that you were trying to run? Are you sure these are problems with the PyPy implementation, not with the Python regexp spec itself? I've used grouping and back-references with pypy with no problems so far, so I wonder which case you're talking about.
  - ColdAsIce 15y ago
    Yes I am sure it was the pypy implementation since the same regex would do fine on python2.7. Unfortunately I dont have the regex at hand, just remembering it was something with groups and backtracking.
- deleted 15y ago
  [deleted]
- andrewcooke 15y ago
  i have a pure python regexp implementation that you can use in pypy (and it passes almost every test used by the cpython re package - only the LOCALE flag is not supported). however, it's around 100x[1] slower that the re package (on simple matches; it's not backtracking so it can be similar in speed for pathological cases). also, when i looked at pypy the re implementation was in c and looked so similar to cpython's i assumed it was the same or closely related. but i didn't pay much attention, so i may be mistaken. [1] roughly. it actually runs 6x faster on recent pypy that cpython.
EdM 15y ago
This post made me think of this article: http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-p... For those who don't know, Travis Oliphant is the creator of NumPy and(?) SciPy. It's a good read and it puts some of the issues with a port into perspective.
- fijal 15y ago
  Hey. While I generally grossly disagree with Travis in regard to how far PyPy can go I wonder what kind of perspective are you talking about?
  - EdM 15y ago
    I think this statement sums up the perspective I'm talking about: "NumPy is just the beginning (SciPy, matplotlib, scikits, and 100s of other packages and legacy C/C++ and Fortran code are all very important)" I'm not that familiar with matplotlib and not familiar at all with scikits. But, the point is that there is a lot of other C/Fortran code that users of NumPy rely on. How much do you gain by porting NumPy to PyPy? (Not a rhetorical question... I'm genuinely curious why the PyPy folks have chosen this as a goal?) PyPy team, if you're out there, please don't take my question as criticism -- it's not. I'm just genuinely curious. Congrats on getting the funding and keep doing what you love!
    - fijal 15y ago
      It's discussed to death in the comments on that blog and others, but reading it might be very boring, so I'll repeat my stance on it (I'm a guy implementing numpy on pypy): NumPy that's faster is already very interesting for many people, because you don't have to go to great lenghts to shift code to C or Cython to experiment. Besides it integrates seamlessly with your current stack that might be in python. Regarding low-level API: Calling C/fortran from PyPy's numpy should be dead easy, over say ctypes. You should be able to call to whatever C libraries you wish. Matplotlib, SciPy and scikits should be relatively easy to get working to some extend using hacks like this - http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html http://morepypy.blogspot.com/2011/12/plotting-using-matplotl... As for other stuff - well if it depends too much on CPython C API PORT IT. It's not that hard and once you have a respectable Python runtime, you can do it, it has been done. Just because we won't support all possible users from day one does not mean we should not try. There are very valid usecases where people shy away from Python because as soon as you try to write a loop in Python, stuff gets to such a crawl that you can't even run experiments. I personally believe Cython is not an answer here and you actually need full python to do most, especially for unexperienced users, so we're primarily targeting the niche that can't be possibly attacked by any solution that's based on CPython. As for other stuff - numpy even if you vectorize stuff is nowhere near the speed of C. We try to attack that as well and even surpass C eventually. This is pretty much it, feel free to ask more questions.

178 points by synparb 15y ago

I like pypy and its ambitions, last time I tested it, about a month ago it was very speedy and startup time considerable faster than Cpython. However the regex exercises I wanted to do couldnt be done. Pypy seemingly didnt have a good regexp engine. If I remember correctly, something with groups and backwards-reference... Has that changed?

bitcracker 15y ago
The fundamental question is: How compatible will PyPy ever be? Which kind of applications can be run with PyPy? http://pypy.org/compat.html http://pypy.org/compat.html mentions compatiblity according to the standard library. This is fine for (web) servers and command line applications. But what about desktop applications? Can I (someday) take a PyQt or PyGTK code and compile it with PyPy without modifications?
- dripton 15y ago
  Maybe. cpyext is a hack to let PyPy run Python/C API modules. It works for some, but not all, and it's slow. I'm sure it'll keep getting better, but not sure if it'll ever be good enough to run PyGTK or PyQt. PyPy has good support for ctypes, so ctypes bindings are a good option. There are projects out there like pygir-ctypes and ctypes-gtk. One of them just needs to become complete enough to be a good choice for GTK programming. Compatibility with new PyGObject is more likely than compatibility with legacy PyGTK, though. PyQt is harder because it's C++.
- DasIch 15y ago
  I believe that C extensions, which have always been and still are very CPython specific, will have to be replaced with ctypes and possibly Cython versions eventually. I expect to see quite a lot of improvement in that area as a side effect of the work on numpy.
kingkilr 15y ago
No, not much has changed about our regex engine in the last month. However, our `re` module is fully compatible with CPython's so I'm bit confused, are you saying it didn't run something, or was slow?
ak217 15y ago
Hi, could you give some details about those regexes that you were trying to run? Are you sure these are problems with the PyPy implementation, not with the Python regexp spec itself? I've used grouping and back-references with pypy with no problems so far, so I wonder which case you're talking about.
- ColdAsIce 15y ago
  Yes I am sure it was the pypy implementation since the same regex would do fine on python2.7. Unfortunately I dont have the regex at hand, just remembering it was something with groups and backtracking.
deleted 15y ago
[deleted]
andrewcooke 15y ago
i have a pure python regexp implementation that you can use in pypy (and it passes almost every test used by the cpython re package - only the LOCALE flag is not supported). however, it's around 100x[1] slower that the re package (on simple matches; it's not backtracking so it can be similar in speed for pathological cases). also, when i looked at pypy the re implementation was in c and looked so similar to cpython's i assumed it was the same or closely related. but i didn't pay much attention, so i may be mistaken. [1] roughly. it actually runs 6x faster on recent pypy that cpython.

EdM 15y ago

This post made me think of this article: http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-p... For those who don't know, Travis Oliphant is the creator of NumPy and(?) SciPy. It's a good read and it puts some of the issues with a port into perspective.

fijal 15y ago
Hey. While I generally grossly disagree with Travis in regard to how far PyPy can go I wonder what kind of perspective are you talking about?
- EdM 15y ago
  I think this statement sums up the perspective I'm talking about: "NumPy is just the beginning (SciPy, matplotlib, scikits, and 100s of other packages and legacy C/C++ and Fortran code are all very important)" I'm not that familiar with matplotlib and not familiar at all with scikits. But, the point is that there is a lot of other C/Fortran code that users of NumPy rely on. How much do you gain by porting NumPy to PyPy? (Not a rhetorical question... I'm genuinely curious why the PyPy folks have chosen this as a goal?) PyPy team, if you're out there, please don't take my question as criticism -- it's not. I'm just genuinely curious. Congrats on getting the funding and keep doing what you love!
  - fijal 15y ago
    It's discussed to death in the comments on that blog and others, but reading it might be very boring, so I'll repeat my stance on it (I'm a guy implementing numpy on pypy): NumPy that's faster is already very interesting for many people, because you don't have to go to great lenghts to shift code to C or Cython to experiment. Besides it integrates seamlessly with your current stack that might be in python. Regarding low-level API: Calling C/fortran from PyPy's numpy should be dead easy, over say ctypes. You should be able to call to whatever C libraries you wish. Matplotlib, SciPy and scikits should be relatively easy to get working to some extend using hacks like this - http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html http://morepypy.blogspot.com/2011/12/plotting-using-matplotl... As for other stuff - well if it depends too much on CPython C API PORT IT. It's not that hard and once you have a respectable Python runtime, you can do it, it has been done. Just because we won't support all possible users from day one does not mean we should not try. There are very valid usecases where people shy away from Python because as soon as you try to write a loop in Python, stuff gets to such a crawl that you can't even run experiments. I personally believe Cython is not an answer here and you actually need full python to do most, especially for unexperienced users, so we're primarily targeting the niche that can't be possibly attacked by any solution that's based on CPython. As for other stuff - numpy even if you vectorize stuff is nowhere near the speed of C. We try to attack that as well and even surpass C eventually. This is pretty much it, feel free to ask more questions.