6 ms·

Thoughts on porting NumPy to PyPy

150 points by johndcook 15y ago

philipn 15y ago
I love it when someone writes an articulate, non-inflamatory blog post with reasonable suggestions on how to improve things. It's sad how rare this is around here these days.
- hartror 15y ago
  I think they are always around but just don't make it to the front page as the link bait articles get the attention.
- fuzzyman 15y ago
  I missed the specific suggestions on how the pypy guys could improve things - other than "do a lot of work on numpy itself as well so that it better supports multiple implementations" (paraphrase obviously). That really needs to be the side of the equation that the numpy team take hold of.
  - cdavid 15y ago
    I don't think Travis is suggesting to the pypy people to do all the work, but rather contribute to the discussion to help the numpy community as a whole for a way forward that makes it easier for both pypy and numpy own needs.
wesm 15y ago
I couldn't agree more. This is related to another recent HN posting (http://news.ycombinator.com/item?id=3104598 http://news.ycombinator.com/item?id=3104598) and discussions at PyCodeConf in Miami. There's a general feeling (which I've experienced in many of my interactions with non-scientific Python programmers) that the folks working on pure Python don't really "get" the scientific Python community. We would all understand each other better if the PyPy folks or the Python core developers spent a year, say, working at Enthought on scientific Python consulting projects or working on a core project that uses NumPy/SciPy like scikit-learn, matplotlib, statsmodels, theano, pandas, or many, many others. A small annoyance but having a matrix multiplication infix operator would actually be a huge help but the idea has met a great deal of resistance from core Python as being "too domain specific". It strikes me as very short-sighted as I think Python is well-poised to make waves in the data analysis, statistics, and high performance computing ecosystem. Having used Python to build large systems for financial applications, I am acutely aware of how Python is being used in that industry and some of the ways that it needs to adapt to be more relevant. So bravo, Travis, for speaking up! Also, his point that Cython (http://cython.org http://cython.org) tends to be ignored in the broader discussion about performance computing in Python is especially flagrant when you consider how it's revolutionized the way that scipythonistas (myself included) speed up their code over the last 2-3 years.
- fijal 15y ago
  Note that as PyPy we deliberately don't take part in the language design discussions. We won't implement the array infix operator precisely because other people make those decisions.
  - kragen 15y ago
    Why not? Don't you think the language design discussion would be improved by your knowledge?
- IanOzsvald 15y ago
  I spoke with Travis on Friday at Enthought and he opened my eyes to some of the possible problems of the pypy-numpy effort (which I've started discussing: http://mail.python.org/pipermail/pypy-dev/2011-October/008601.html http://mail.python.org/pipermail/pypy-dev/2011-October/00860... ). I hadn't realised that the port might exclude use of the rest of SciPy, this strikes me as a massive missed opportunity if it comes to pass. Perhaps a few other knowledgeable folk could post in the pypy-dev thread so we can reasonably understand the limitations and possibilities of the various approaches?
  - fuzzyman 15y ago
    I don't really know what you mean by supporting numpy on pypy "excludes" scipy (other than in the short term). If you mean you assumed that the pypy team would port all of scipy as well as numpy, that seems like an unrealistic expectation for an initial port! An initial port seems like the only way forward from the point of view of the pypy team. I think it is unrealistic to expect the pypy team to take on the work of changing numpy so that it is more friendly to alternative implementations. I would certainly expect them to be involved in the discussion though. The article also seems to miss that there is work ongoing to bring pypy support to Cython.
    - wesm 15y ago
      Well, the big problem to solve with SciPy is the fact that there is more, respectively, of C, C++, and Fortran in the SciPy codebase than Python (http://www.ohloh.net/p/scipy http://www.ohloh.net/p/scipy). Part of why Python has succeeded in scientific computing is integration of legacy codebases (via f2py or C extensions or...). There are probably man-years of work involved in devising a solution-- which by the time it's complete may be basically irrelevant or, worse, fragment the community. I personally think that going down this rabbit hole (porting 10 years of scientific Python libraries to PyPy) would amount to an exercise in vanity rather than producing the kinds of revolutionary changes to array-oriented computing that need to happen soon to deal with the large-scale data processing challenges of the present and future. Having recently used GPUs to speed up statistical inference algorithms by a factor of 50 or more, I am not that motivated by a JIT beating C in some cases (as Travis wrote: "C speed is the wrong target"). Many in the SciPy community are convinced that NumPy will not provide the computational foundation that we need going forward, and they are going to step up and start building the next generation NumPy (or whatever it's going to be called). We'd rather have more of the smartest computer scientists in the Python community focused on this problem (building more sophisticated data processing pipelines for use in Python) than on speeding up the Python code that by my estimation doesn't matter that much.
juiceandjuice 15y ago
> There were a lot of young people there who understood a lot more about making web-pages and working at cool web start-ups than solving partial differential equations with arrays. I can especially relate to this, even though I'm 26. Admittedly, I work in a microcosm though, where I'm the youngest, least educated and least experienced in my group, despite having a BS in Physics and 4 years research in particle astrophysics. Luckily we do a little less with matrix operations, but I agree that Python, specifically PyPy, has so much potential to be the scientific computing standard and that the python community should really push for it. In a world with less python, there's so much pain involved on a regular basis setting up software like ROOT, switching between libraries of FFTs or plot libraries, installing Octave or Matlab to work nice with some bash script and dealing with OS discrepancies in getting someone else's code to run. It sucks. If PyPy can displace that with near-C performance, the world would be a much better place.
pwang 15y ago
I was going to do a Cython lightning talk, but the organizers indicated that we ran out of time slots, so I could only give one talk. Half of the audience at PyCodeConf indicated they were aware of the Cython project, although perhaps it would have been useful regardless...
eastwest 15y ago
I just don't see adding a special matrix infix operator to Python happening. It is too specialized, and yet matrix is not special enough that it must have its own operator. Isn't this what operator overloading is for? I suspect the real problem is NumPy's type system implementation, since data-types are not visible as different Python types.
- wesm 15y ago
  If you spent a day doing some serious linear algebra in Python you might change your tune. * (multiplication) between NumPy arrays by default does element-wise multiplication (potentially with broadcasting), which is the desired default behavior. In R, for example, you can define custom infix operators so that a * b is elementwise but a %* % b is matrix multiplication. If you write down a complicated linear algebra expression, something like A.T ! (B.T ! C ! B).T ! A would be a lot friendlier to scientists than the current: dot(A.T, dot(dot(B.T, dot(C, B)).T, A)) You might just say "well suck it up" but I've got to say that doing linear algebra in Matlab is a lot easier because the linear algebra that I do with pen and paper looks pretty much exactly the same as the corresponding code. On the other hand, Matlab is super clunky compared with NumPy at doing APL-style array processing with broadcasting operations, etc. In my work I tend to do more of the latter and less of the former but whenever I implement something with a lot of matrix multiplications it takes me a lot longer in Python to get things right. Anyway, the point is: non-scientific Python folks need to take a walk in our shoes to gain an understanding of the challenges we face on a ongoing basis. I'm having a hard time understanding your last statement. NumPy data types (dtypes) simply tell the ndarray how to interpret the block of data associated with it (the # of bytes per item, shape, and strides).
  - illumen 15y ago
    I guess it would be possible to implement with a with-hack... with numpy.doing_matrix: matrixy_multiply_goodness = m1 * m2 array_multiply_goodness = a1 * a2
  - scott_s 15y ago
    I see your point, and it is why I tend to like operator overloading in languages. However, Python is a general purpose language, and scientific computing is a single domain. It is used in many different domains. Consider that there are many changes that individual communities would like, and if Python granted all of those requests, the language would be a mess. That's the challenge in designing a general purpose programming language.
lucian1900 15y ago
If numpy-c were to be called over ctypes, it'd be slow.The reason PyPy's numpy reimplementation is so insanely fast is the integration with the JIT. A more plausible scenario would be both CPython and PyPy using numpy-py to call into numpy-c and numpy-pypy respectively.
ZeroGravitas 15y ago
Off-topic but anyone else seeing every instance of "fi" in the text replaced with a slightly overlapping AV? (I tried to cut and paste it but it pasted as just "fi"). I'm on Ubuntu Ocelot & Firefox 7. nevermind: I'm seeing it on various websites that use ligatures. Possibly because I'm half-way through an Ubuntu upgrade.

150 points by johndcook 15y ago

I love it when someone writes an articulate, non-inflamatory blog post with reasonable suggestions on how to improve things. It's sad how rare this is around here these days.

hartror 15y ago
I think they are always around but just don't make it to the front page as the link bait articles get the attention.
fuzzyman 15y ago
I missed the specific suggestions on how the pypy guys could improve things - other than "do a lot of work on numpy itself as well so that it better supports multiple implementations" (paraphrase obviously). That really needs to be the side of the equation that the numpy team take hold of.
- cdavid 15y ago
  I don't think Travis is suggesting to the pypy people to do all the work, but rather contribute to the discussion to help the numpy community as a whole for a way forward that makes it easier for both pypy and numpy own needs.

wesm 15y ago

I couldn't agree more. This is related to another recent HN posting (http://news.ycombinator.com/item?id=3104598 http://news.ycombinator.com/item?id=3104598) and discussions at PyCodeConf in Miami. There's a general feeling (which I've experienced in many of my interactions with non-scientific Python programmers) that the folks working on pure Python don't really "get" the scientific Python community. We would all understand each other better if the PyPy folks or the Python core developers spent a year, say, working at Enthought on scientific Python consulting projects or working on a core project that uses NumPy/SciPy like scikit-learn, matplotlib, statsmodels, theano, pandas, or many, many others. A small annoyance but having a matrix multiplication infix operator would actually be a huge help but the idea has met a great deal of resistance from core Python as being "too domain specific". It strikes me as very short-sighted as I think Python is well-poised to make waves in the data analysis, statistics, and high performance computing ecosystem. Having used Python to build large systems for financial applications, I am acutely aware of how Python is being used in that industry and some of the ways that it needs to adapt to be more relevant. So bravo, Travis, for speaking up! Also, his point that Cython (http://cython.org http://cython.org) tends to be ignored in the broader discussion about performance computing in Python is especially flagrant when you consider how it's revolutionized the way that scipythonistas (myself included) speed up their code over the last 2-3 years.

fijal 15y ago
Note that as PyPy we deliberately don't take part in the language design discussions. We won't implement the array infix operator precisely because other people make those decisions.
- kragen 15y ago
  Why not? Don't you think the language design discussion would be improved by your knowledge?
IanOzsvald 15y ago
I spoke with Travis on Friday at Enthought and he opened my eyes to some of the possible problems of the pypy-numpy effort (which I've started discussing: http://mail.python.org/pipermail/pypy-dev/2011-October/008601.html http://mail.python.org/pipermail/pypy-dev/2011-October/00860... ). I hadn't realised that the port might exclude use of the rest of SciPy, this strikes me as a massive missed opportunity if it comes to pass. Perhaps a few other knowledgeable folk could post in the pypy-dev thread so we can reasonably understand the limitations and possibilities of the various approaches?
- fuzzyman 15y ago
  I don't really know what you mean by supporting numpy on pypy "excludes" scipy (other than in the short term). If you mean you assumed that the pypy team would port all of scipy as well as numpy, that seems like an unrealistic expectation for an initial port! An initial port seems like the only way forward from the point of view of the pypy team. I think it is unrealistic to expect the pypy team to take on the work of changing numpy so that it is more friendly to alternative implementations. I would certainly expect them to be involved in the discussion though. The article also seems to miss that there is work ongoing to bring pypy support to Cython.
  - wesm 15y ago
    Well, the big problem to solve with SciPy is the fact that there is more, respectively, of C, C++, and Fortran in the SciPy codebase than Python (http://www.ohloh.net/p/scipy http://www.ohloh.net/p/scipy). Part of why Python has succeeded in scientific computing is integration of legacy codebases (via f2py or C extensions or...). There are probably man-years of work involved in devising a solution-- which by the time it's complete may be basically irrelevant or, worse, fragment the community. I personally think that going down this rabbit hole (porting 10 years of scientific Python libraries to PyPy) would amount to an exercise in vanity rather than producing the kinds of revolutionary changes to array-oriented computing that need to happen soon to deal with the large-scale data processing challenges of the present and future. Having recently used GPUs to speed up statistical inference algorithms by a factor of 50 or more, I am not that motivated by a JIT beating C in some cases (as Travis wrote: "C speed is the wrong target"). Many in the SciPy community are convinced that NumPy will not provide the computational foundation that we need going forward, and they are going to step up and start building the next generation NumPy (or whatever it's going to be called). We'd rather have more of the smartest computer scientists in the Python community focused on this problem (building more sophisticated data processing pipelines for use in Python) than on speeding up the Python code that by my estimation doesn't matter that much.

juiceandjuice 15y ago

> There were a lot of young people there who understood a lot more about making web-pages and working at cool web start-ups than solving partial differential equations with arrays. I can especially relate to this, even though I'm 26. Admittedly, I work in a microcosm though, where I'm the youngest, least educated and least experienced in my group, despite having a BS in Physics and 4 years research in particle astrophysics. Luckily we do a little less with matrix operations, but I agree that Python, specifically PyPy, has so much potential to be the scientific computing standard and that the python community should really push for it. In a world with less python, there's so much pain involved on a regular basis setting up software like ROOT, switching between libraries of FFTs or plot libraries, installing Octave or Matlab to work nice with some bash script and dealing with OS discrepancies in getting someone else's code to run. It sucks. If PyPy can displace that with near-C performance, the world would be a much better place.

pwang 15y ago

I was going to do a Cython lightning talk, but the organizers indicated that we ran out of time slots, so I could only give one talk. Half of the audience at PyCodeConf indicated they were aware of the Cython project, although perhaps it would have been useful regardless...

eastwest 15y ago

I just don't see adding a special matrix infix operator to Python happening. It is too specialized, and yet matrix is not special enough that it must have its own operator. Isn't this what operator overloading is for? I suspect the real problem is NumPy's type system implementation, since data-types are not visible as different Python types.

wesm 15y ago
If you spent a day doing some serious linear algebra in Python you might change your tune. * (multiplication) between NumPy arrays by default does element-wise multiplication (potentially with broadcasting), which is the desired default behavior. In R, for example, you can define custom infix operators so that a * b is elementwise but a %* % b is matrix multiplication. If you write down a complicated linear algebra expression, something like A.T ! (B.T ! C ! B).T ! A would be a lot friendlier to scientists than the current: dot(A.T, dot(dot(B.T, dot(C, B)).T, A)) You might just say "well suck it up" but I've got to say that doing linear algebra in Matlab is a lot easier because the linear algebra that I do with pen and paper looks pretty much exactly the same as the corresponding code. On the other hand, Matlab is super clunky compared with NumPy at doing APL-style array processing with broadcasting operations, etc. In my work I tend to do more of the latter and less of the former but whenever I implement something with a lot of matrix multiplications it takes me a lot longer in Python to get things right. Anyway, the point is: non-scientific Python folks need to take a walk in our shoes to gain an understanding of the challenges we face on a ongoing basis. I'm having a hard time understanding your last statement. NumPy data types (dtypes) simply tell the ndarray how to interpret the block of data associated with it (the # of bytes per item, shape, and strides).
- illumen 15y ago
  I guess it would be possible to implement with a with-hack... with numpy.doing_matrix: matrixy_multiply_goodness = m1 * m2 array_multiply_goodness = a1 * a2
- scott_s 15y ago
  I see your point, and it is why I tend to like operator overloading in languages. However, Python is a general purpose language, and scientific computing is a single domain. It is used in many different domains. Consider that there are many changes that individual communities would like, and if Python granted all of those requests, the language would be a mess. That's the challenge in designing a general purpose programming language.

lucian1900 15y ago

If numpy-c were to be called over ctypes, it'd be slow.The reason PyPy's numpy reimplementation is so insanely fast is the integration with the JIT. A more plausible scenario would be both CPython and PyPy using numpy-py to call into numpy-c and numpy-pypy respectively.

ZeroGravitas 15y ago

Off-topic but anyone else seeing every instance of "fi" in the text replaced with a slightly overlapping AV? (I tried to cut and paste it but it pasted as just "fi"). I'm on Ubuntu Ocelot & Firefox 7. nevermind: I'm seeing it on various websites that use ligatures. Possibly because I'm half-way through an Ubuntu upgrade.