3 ms·

This post made me think of this article: http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html http://technicaldiscovery.blogspo

by EdM 15y ago

This post made me think of this article: http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-p... For those who don't know, Travis Oliphant is the creator of NumPy and(?) SciPy. It's a good read and it puts some of the issues with a port into perspective.

fijal 15y ago
Hey. While I generally grossly disagree with Travis in regard to how far PyPy can go I wonder what kind of perspective are you talking about?
- EdM 15y ago
  I think this statement sums up the perspective I'm talking about: "NumPy is just the beginning (SciPy, matplotlib, scikits, and 100s of other packages and legacy C/C++ and Fortran code are all very important)" I'm not that familiar with matplotlib and not familiar at all with scikits. But, the point is that there is a lot of other C/Fortran code that users of NumPy rely on. How much do you gain by porting NumPy to PyPy? (Not a rhetorical question... I'm genuinely curious why the PyPy folks have chosen this as a goal?) PyPy team, if you're out there, please don't take my question as criticism -- it's not. I'm just genuinely curious. Congrats on getting the funding and keep doing what you love!
  - fijal 15y ago
    It's discussed to death in the comments on that blog and others, but reading it might be very boring, so I'll repeat my stance on it (I'm a guy implementing numpy on pypy): NumPy that's faster is already very interesting for many people, because you don't have to go to great lenghts to shift code to C or Cython to experiment. Besides it integrates seamlessly with your current stack that might be in python. Regarding low-level API: Calling C/fortran from PyPy's numpy should be dead easy, over say ctypes. You should be able to call to whatever C libraries you wish. Matplotlib, SciPy and scikits should be relatively easy to get working to some extend using hacks like this - http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html http://morepypy.blogspot.com/2011/12/plotting-using-matplotl... As for other stuff - well if it depends too much on CPython C API PORT IT. It's not that hard and once you have a respectable Python runtime, you can do it, it has been done. Just because we won't support all possible users from day one does not mean we should not try. There are very valid usecases where people shy away from Python because as soon as you try to write a loop in Python, stuff gets to such a crawl that you can't even run experiments. I personally believe Cython is not an answer here and you actually need full python to do most, especially for unexperienced users, so we're primarily targeting the niche that can't be possibly attacked by any solution that's based on CPython. As for other stuff - numpy even if you vectorize stuff is nowhere near the speed of C. We try to attack that as well and even surpass C eventually. This is pretty much it, feel free to ask more questions.
    - teoliphant 15y ago
      Your statement that NumPy is "nowhere near the speed of C" is false and misleading. For some people "NumPy" is actually just a front-end to the vendor-optimized libraries that actually do the work (and your C-coded loops are going to be much slower than those). For most operations (with large-enough vectors) NumPy is only 2x slower than a specific crafted C-loop. Yes, there are generic operations in NumPy that you can speed -up with specific code in C (or any-other compiled language). In addition, there is much low-hanging fruit to optimize in NumPy as well (which we at Continuum are working on as I write this). Fijal, I know you are enthusiastic about PyPy and you should be --- it's a cool system. But, please don't spread mis-information. There are a lot of people who don't understand enough about the details of what you are talking about, and you are just going to alienate them once they realize that you don't have all your facts about NumPy clear. For people who only make occasional use of NumPy, PyPy and it's version of numpy will likely be fine. But, those people should be well-aware that they are intentionally remaining outside the larger Python/NumPy ecosystem (Matplotlib, SciPy, scikits, etc.) and it will be a long-haul to build the features in PyPy to enable that ecosystem to migrate (and that assumes the individual projects decide that it's even worthwhile to do so).
      - fijal 15y ago
        I think I disagree pretty much about every single point you make. First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. As your list of operations increase, your need of intermediates grow and your speed decreases, but let's not go to details. Obviously if you just call a vendor-optimized library, you can use whatever you feel like and it'll be equally good, be it PyPy, be it numpy, be it matlab. You consistently spread rumor that we intend to reimplement all of scipy/matplotlib/scikits etc in RPython and this is plain false. I think those projects are completely reusable using one hack or another, for example the blog post I posted where within a day I was able to draw basic stuff using matplotlib on PyPy. We seriously want to reuse as much code as possible from the entire ecosystem, but also a part of the project is to provide people with a really fast python that can perform numeric computations. Also, which facts about numpy I didn't get clear?
- teoliphant 15y ago
  Fijal, I'm not sure you even understand my perspective. None of your comments in response have given me confidence that you do. I have no disagreement with you about how far "PyPy could go". Obviously, we could re-create the entire Python ecosystem including the scientific stack under its run-time. I just question your understanding of how expensive and time-consuming that would be. My official position is that I think it's possible to get the benefits of PyPy (i.e. fast Python loops) in different ways that don't also toss out years of extension modules in the process and whose answer to current users about the features they rely on in CPython not working under PyPy is "just port it" or "just use ctypes".
  - fijal 15y ago
    Well. Fast Python loops in CPython has been tried before and failed. I seriously don't see a way of getting a working JIT that really optimizes a lot of code out there and native support for CPython extensions. There will be some side that suffers. Also, numpy is fairly special as it does have a good potential to be optimized by the JIT in ways that are not quite possible using C or Cython.
  - synparb 15y ago
    From having read the various replies to this and similar threads, I think a basic problem (as Travis pointed out) is that Fijal has been (1) highly dismissive of numpy and particularly cython, which a lot of really smart people have put a lot of time and effort into and created an amazing scientific community within python, and (2) confrontational with people who are trying to lend an alternative opinion which has been informed by a lot of experience in the scientific python world. PyPy might eventually be a wholly superior alternative for scientific computing with Python, but it would be good to acknowledge the insight of those who are 'in the trenches' even if you go a different route, because part of the success of scientific python has been the community. I know very little about the development aspect of PyPy or Numpy, but I know that at this moment in time Numpy/Scipy/Cython have revolutionized how I do research on a day to day basis. It seems unfortunate that there seems to be such animosity surrounding this issue.
    - 15y ago

by EdM 15y ago

fijal 15y ago

Hey. While I generally grossly disagree with Travis in regard to how far PyPy can go I wonder what kind of perspective are you talking about?

EdM 15y ago
I think this statement sums up the perspective I'm talking about: "NumPy is just the beginning (SciPy, matplotlib, scikits, and 100s of other packages and legacy C/C++ and Fortran code are all very important)" I'm not that familiar with matplotlib and not familiar at all with scikits. But, the point is that there is a lot of other C/Fortran code that users of NumPy rely on. How much do you gain by porting NumPy to PyPy? (Not a rhetorical question... I'm genuinely curious why the PyPy folks have chosen this as a goal?) PyPy team, if you're out there, please don't take my question as criticism -- it's not. I'm just genuinely curious. Congrats on getting the funding and keep doing what you love!
- fijal 15y ago
  It's discussed to death in the comments on that blog and others, but reading it might be very boring, so I'll repeat my stance on it (I'm a guy implementing numpy on pypy): NumPy that's faster is already very interesting for many people, because you don't have to go to great lenghts to shift code to C or Cython to experiment. Besides it integrates seamlessly with your current stack that might be in python. Regarding low-level API: Calling C/fortran from PyPy's numpy should be dead easy, over say ctypes. You should be able to call to whatever C libraries you wish. Matplotlib, SciPy and scikits should be relatively easy to get working to some extend using hacks like this - http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html http://morepypy.blogspot.com/2011/12/plotting-using-matplotl... As for other stuff - well if it depends too much on CPython C API PORT IT. It's not that hard and once you have a respectable Python runtime, you can do it, it has been done. Just because we won't support all possible users from day one does not mean we should not try. There are very valid usecases where people shy away from Python because as soon as you try to write a loop in Python, stuff gets to such a crawl that you can't even run experiments. I personally believe Cython is not an answer here and you actually need full python to do most, especially for unexperienced users, so we're primarily targeting the niche that can't be possibly attacked by any solution that's based on CPython. As for other stuff - numpy even if you vectorize stuff is nowhere near the speed of C. We try to attack that as well and even surpass C eventually. This is pretty much it, feel free to ask more questions.
  - teoliphant 15y ago
    Your statement that NumPy is "nowhere near the speed of C" is false and misleading. For some people "NumPy" is actually just a front-end to the vendor-optimized libraries that actually do the work (and your C-coded loops are going to be much slower than those). For most operations (with large-enough vectors) NumPy is only 2x slower than a specific crafted C-loop. Yes, there are generic operations in NumPy that you can speed -up with specific code in C (or any-other compiled language). In addition, there is much low-hanging fruit to optimize in NumPy as well (which we at Continuum are working on as I write this). Fijal, I know you are enthusiastic about PyPy and you should be --- it's a cool system. But, please don't spread mis-information. There are a lot of people who don't understand enough about the details of what you are talking about, and you are just going to alienate them once they realize that you don't have all your facts about NumPy clear. For people who only make occasional use of NumPy, PyPy and it's version of numpy will likely be fine. But, those people should be well-aware that they are intentionally remaining outside the larger Python/NumPy ecosystem (Matplotlib, SciPy, scikits, etc.) and it will be a long-haul to build the features in PyPy to enable that ecosystem to migrate (and that assumes the individual projects decide that it's even worthwhile to do so).
    - fijal 15y ago
      I think I disagree pretty much about every single point you make. First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. As your list of operations increase, your need of intermediates grow and your speed decreases, but let's not go to details. Obviously if you just call a vendor-optimized library, you can use whatever you feel like and it'll be equally good, be it PyPy, be it numpy, be it matlab. You consistently spread rumor that we intend to reimplement all of scipy/matplotlib/scikits etc in RPython and this is plain false. I think those projects are completely reusable using one hack or another, for example the blog post I posted where within a day I was able to draw basic stuff using matplotlib on PyPy. We seriously want to reuse as much code as possible from the entire ecosystem, but also a part of the project is to provide people with a really fast python that can perform numeric computations. Also, which facts about numpy I didn't get clear?
teoliphant 15y ago
Fijal, I'm not sure you even understand my perspective. None of your comments in response have given me confidence that you do. I have no disagreement with you about how far "PyPy could go". Obviously, we could re-create the entire Python ecosystem including the scientific stack under its run-time. I just question your understanding of how expensive and time-consuming that would be. My official position is that I think it's possible to get the benefits of PyPy (i.e. fast Python loops) in different ways that don't also toss out years of extension modules in the process and whose answer to current users about the features they rely on in CPython not working under PyPy is "just port it" or "just use ctypes".
- fijal 15y ago
  Well. Fast Python loops in CPython has been tried before and failed. I seriously don't see a way of getting a working JIT that really optimizes a lot of code out there and native support for CPython extensions. There will be some side that suffers. Also, numpy is fairly special as it does have a good potential to be optimized by the JIT in ways that are not quite possible using C or Cython.
- synparb 15y ago
  From having read the various replies to this and similar threads, I think a basic problem (as Travis pointed out) is that Fijal has been (1) highly dismissive of numpy and particularly cython, which a lot of really smart people have put a lot of time and effort into and created an amazing scientific community within python, and (2) confrontational with people who are trying to lend an alternative opinion which has been informed by a lot of experience in the scientific python world. PyPy might eventually be a wholly superior alternative for scientific computing with Python, but it would be good to acknowledge the insight of those who are 'in the trenches' even if you go a different route, because part of the success of scientific python has been the community. I know very little about the development aspect of PyPy or Numpy, but I know that at this moment in time Numpy/Scipy/Cython have revolutionized how I do research on a day to day basis. It seems unfortunate that there seems to be such animosity surrounding this issue.
  - 15y ago