3 ms·

Your statement that NumPy is "nowhere near the speed of C" is false and misleading. For some people "NumPy" is actually just a front-end to the vendor-optimize

Your statement that NumPy is "nowhere near the speed of C" is false and misleading. For some people "NumPy" is actually just a front-end to the vendor-optimized libraries that actually do the work (and your C-coded loops are going to be much slower than those). For most operations (with large-enough vectors) NumPy is only 2x slower than a specific crafted C-loop. Yes, there are generic operations in NumPy that you can speed -up with specific code in C (or any-other compiled language). In addition, there is much low-hanging fruit to optimize in NumPy as well (which we at Continuum are working on as I write this). Fijal, I know you are enthusiastic about PyPy and you should be --- it's a cool system. But, please don't spread mis-information. There are a lot of people who don't understand enough about the details of what you are talking about, and you are just going to alienate them once they realize that you don't have all your facts about NumPy clear. For people who only make occasional use of NumPy, PyPy and it's version of numpy will likely be fine. But, those people should be well-aware that they are intentionally remaining outside the larger Python/NumPy ecosystem (Matplotlib, SciPy, scikits, etc.) and it will be a long-haul to build the features in PyPy to enable that ecosystem to migrate (and that assumes the individual projects decide that it's even worthwhile to do so).

fijal 15y ago
I think I disagree pretty much about every single point you make. First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. As your list of operations increase, your need of intermediates grow and your speed decreases, but let's not go to details. Obviously if you just call a vendor-optimized library, you can use whatever you feel like and it'll be equally good, be it PyPy, be it numpy, be it matlab. You consistently spread rumor that we intend to reimplement all of scipy/matplotlib/scikits etc in RPython and this is plain false. I think those projects are completely reusable using one hack or another, for example the blog post I posted where within a day I was able to draw basic stuff using matplotlib on PyPy. We seriously want to reuse as much code as possible from the entire ecosystem, but also a part of the project is to provide people with a really fast python that can perform numeric computations. Also, which facts about numpy I didn't get clear?
- MichaelSalib 15y ago
  First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. Isn't numexpr a good solution for that problem? numexpr is much much simpler than PyPy, so if it can reduce the performance overhead of complex vector operations with loop fusion, that seems a huge win.
  - pwang 15y ago
    Yes; numexpr and weave are pretty reasonable solutions, although a little "weird" because they take opaque strings. One bit of context that is missing from this discussion (although Travis did allude to it earlier) is that we are actively working on building robust deferred computation support into Numpy, and to make these run much faster than what hand-tuned C can provide, via a variety of mechanisms. (Disclaimer: I also work with Travis at http://continuum.io http://continuum.io, along with the author of Numexpr and PyTables. :-)
  - kingkilr 15y ago
    [citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.
    - MichaelSalib 15y ago
      numexpr is much simpler than pypy. I never said that it was much simpler than pypy's incomplete numpy support.
    - pwang 15y ago
      Um, that's not a dumb measure of simplicity, it's just a measure of code size. You might as well look at the average number of characters in the names of the authors of the two projects; it would be equally irrelevant. Here is a real measure of simplicity: how long does it take to explain to a Numpy user how to wrap an array expression in a string, versus explaining how a JITting compiler compiler works and how to interface its runtime to their existing Python installation and how to build it and what the limitations of RPython are. Heck, I'm an actual developer (not a scientific programmer) and it took me a little while to understand what PyPy does.

fijal 15y ago

I think I disagree pretty much about every single point you make. First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. As your list of operations increase, your need of intermediates grow and your speed decreases, but let's not go to details. Obviously if you just call a vendor-optimized library, you can use whatever you feel like and it'll be equally good, be it PyPy, be it numpy, be it matlab. You consistently spread rumor that we intend to reimplement all of scipy/matplotlib/scikits etc in RPython and this is plain false. I think those projects are completely reusable using one hack or another, for example the blog post I posted where within a day I was able to draw basic stuff using matplotlib on PyPy. We seriously want to reuse as much code as possible from the entire ecosystem, but also a part of the project is to provide people with a really fast python that can perform numeric computations. Also, which facts about numpy I didn't get clear?

MichaelSalib 15y ago
First for something as simple as laplace equation solver numpy vectorized loop is 35ms per loop vs 6.3ms for C. Isn't numexpr a good solution for that problem? numexpr is much much simpler than PyPy, so if it can reduce the performance overhead of complex vector operations with loop fusion, that seems a huge win.
- pwang 15y ago
  Yes; numexpr and weave are pretty reasonable solutions, although a little "weird" because they take opaque strings. One bit of context that is missing from this discussion (although Travis did allude to it earlier) is that we are actively working on building robust deferred computation support into Numpy, and to make these run much faster than what hand-tuned C can provide, via a variety of mechanisms. (Disclaimer: I also work with Travis at http://continuum.io http://continuum.io, along with the author of Numexpr and PyTables. :-)
- kingkilr 15y ago
  [citation needed] as for numexpr being simpler than PyPy. Using the dumbest measure of simplicity, total code size, it fails: PyPy's NumPy is 167KB of code, NumExpr is 181KB.
  - MichaelSalib 15y ago
    numexpr is much simpler than pypy. I never said that it was much simpler than pypy's incomplete numpy support.
  - pwang 15y ago
    Um, that's not a dumb measure of simplicity, it's just a measure of code size. You might as well look at the average number of characters in the names of the authors of the two projects; it would be equally irrelevant. Here is a real measure of simplicity: how long does it take to explain to a Numpy user how to wrap an array expression in a string, versus explaining how a JITting compiler compiler works and how to interface its runtime to their existing Python installation and how to build it and what the limitations of RPython are. Heck, I'm an actual developer (not a scientific programmer) and it took me a little while to understand what PyPy does.