6 ms·

The NumPy array: a structure for efficient numerical computation (2011) [pdf]

126 points by kercker 10y ago

jostmey 10y ago
How about an efficient structure for storing "actual" arrays in python. Numpy is my friend.
- Animats 10y ago
  Does PyPy have enough type inference smarts to figure out when it can use a homogeneous array? In CPython, everything, including every array entry, is a CObject. (Except strings and now, bytearrays. They're special.) That's why Python arrays of numbers are so slow. NumPy's representation is very close to that of FORTRAN. This problem was mostly solved in the 1950s and 1960s, when FORTRAN DO-loop optimizations for arrays got good.
  - ludamad 10y ago
    http://doc.pypy.org/en/latest/release-1.8.0.html http://doc.pypy.org/en/latest/release-1.8.0.html It does this since 1.8 it seems (mentioned under 'list specialization'). Probably 'type inference' isn't the right way to describe it. It will optimistically assume it's a homogeneous array and will have to recreate this array if the assumption is broken. This will be useful in the vast majority of cases, as mixed type arrays are not usually used in performance-sensitive code.
    - rlayton2 10y ago
      Further, there is almost always a better data structure for any mixed-type list. So if you know your code will go through pypy, you can just use a different structure.
  - tgb 10y ago
    What kind of DO-loop optimizations were there?
    - Animats 10y ago
      The same ones we have now. Advancing through an array with pointers instead of computing the subscript each time. Advancing down a higher dimension of an array with a stride. Backus had those working in a FORTRAN compiler in the 1950s. If FORTRAN had been slower than assembly, nobody would have used it; machines were too slow and too expensive.
- DasIch 10y ago
  You mean like the array[1] that's in the standard library? [1]: https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html
jdlyga 10y ago
If only it was also available for C++. Armadillo is nice but it has a lot of important differences, and doesn't seem to support more than 3 dimensional matrices.
- keypusher 10y ago
  There was some effort toward getting it in Boost. https://github.com/ndarray/Boost.NumPy https://github.com/ndarray/Boost.NumPy
- chombier 10y ago
  You might want to check out the Eigen library: http://eigen.tuxfamily.org/ http://eigen.tuxfamily.org/
- srean 10y ago
  If you want multidimensional arrays in C++, then blitz++ is your friend. There is Eigen but it wont do multidimensional stuff. Although there might be a few projects out there that wrap the multidimensional functionality around Eigen arrays.
  - kersny 10y ago
    One such multidimensional extension of Eigen is from Google and I believe was extracted from TensorFlow: https://bitbucket.org/eigen/eigen/src/039efd86b75ce1b72befb7e0a5bd069baacb17b7/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default https://bitbucket.org/eigen/eigen/src/039efd86b75ce1b72befb7...
- lloda 10y ago
  Here is my Blitz++-inspired hobby project, supports arbitrary rank: https://github.com/lloda/ra-ra https://github.com/lloda/ra-ra I'm sure you can find more polished efforts if you look around.
- naveen99 10y ago
  Arrayfire is c++ with Opencl, cuda, and cpu back ends. Limited to 4d. http://arrayfire.com/indexing-with-arrayfire/ http://arrayfire.com/indexing-with-arrayfire/
agibsonccc 10y ago
The numpy array is a great data structure. I know it's not python but I wanted to plug our attempt at this for the jvm, nd4j: http://nd4j.org/tensor http://nd4j.org/tensor We have a java wrapper: https://github.com/deeplearning4j/nd4j https://github.com/deeplearning4j/nd4j Scala: https://github.com/deeplearning4j/nd4s https://github.com/deeplearning4j/nd4s And underlying c++: https://github.com/deeplearning4j/libnd4j https://github.com/deeplearning4j/libnd4j We heavily dog food it in our deep learning library but we're hoping this can be more broadly useful for people.
- hatmatrix 10y ago
  That's great but what about nd4j's data structures resemble numpy arrays? Without making connection to the original most this comment just reads like typical spam.
  - jnbiche 10y ago
    In my experience, it's perfectly acceptable on HN to plug a related project in a comment, as long as it's not done repeatedly for totally unrelated projects (in 5+ years here, I can only think of 1 or 2 examples of this actually happening on a frequent basis). Given that these are open source and not even commercial projects, and that this is presumably an enthusiastic developer for those projects just sharing his/her work with a potentially interested audience, I hardly think the "spam" label is appropriate. And to answer your question specifically, based on what I've just read, ND4J offers an optimized n-dimensional array intended for numeric/scientific computing, very much in the manner of numpy. This is presumably what the comment's "our attempt at this for the jvm" refers to.
    - agibsonccc 10y ago
      Yes it's a port of the associated concepts to the jvm. FWIW: we actually do use this commercially ;). It's apache though so do what you want with it etc. Apologies if it came off like that but the main intent was to show an alternative implementation. If you actually look at the nd4j.org link you'll also notice a simpler/more concise description of the same concepts. Eg: linear buffers,shapes, strides, offsets, dimensions
noobermin 10y ago
This type is my bread and butter. I breathe numpy arrays in my research.
- jeroenjanssens 10y ago
  What an interesting juxtaposition of idioms :)
hatmatrix 10y ago
It's a shame that there is no consolidated way to provide index labels to both columns and rows without wrapping this in another class (e.g., your own, or pandas). R's matrices and arrays are very convenient as they provide this functionality out-of-the-box.
- cgearhart 10y ago
  I'm not terribly familiar with R, but I've used Pandas a bit. Could you accomplish what you're looking for with structured arrays or record arrays in Numpy? http://docs.scipy.org/doc/numpy/user/basics.rec.html http://docs.scipy.org/doc/numpy/user/basics.rec.html
  - hatmatrix 10y ago
    Record arrays only provide indexing by name along columns, unfortunately. So if you want index labels along both dimensions (in 2-D), Pandas is necessary. There is no good option for the N-D case that I can find (R's native array has this capability by default though).
nl 10y ago
If you like NumPy, it might be worth taking a look at Dask[1]. Very similar API, but designed to support arrays larger than memory, as well as operations across a network cluster. [1] http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/
denfromufa 10y ago
dynd is next generation numpy for language-agnostic usage: https://github.com/libdynd/libdynd https://github.com/libdynd/libdynd

126 points by kercker 10y ago

jostmey 10y ago

How about an efficient structure for storing "actual" arrays in python. Numpy is my friend.

Animats 10y ago
Does PyPy have enough type inference smarts to figure out when it can use a homogeneous array? In CPython, everything, including every array entry, is a CObject. (Except strings and now, bytearrays. They're special.) That's why Python arrays of numbers are so slow. NumPy's representation is very close to that of FORTRAN. This problem was mostly solved in the 1950s and 1960s, when FORTRAN DO-loop optimizations for arrays got good.
- ludamad 10y ago
  http://doc.pypy.org/en/latest/release-1.8.0.html http://doc.pypy.org/en/latest/release-1.8.0.html It does this since 1.8 it seems (mentioned under 'list specialization'). Probably 'type inference' isn't the right way to describe it. It will optimistically assume it's a homogeneous array and will have to recreate this array if the assumption is broken. This will be useful in the vast majority of cases, as mixed type arrays are not usually used in performance-sensitive code.
  - rlayton2 10y ago
    Further, there is almost always a better data structure for any mixed-type list. So if you know your code will go through pypy, you can just use a different structure.
- tgb 10y ago
  What kind of DO-loop optimizations were there?
  - Animats 10y ago
    The same ones we have now. Advancing through an array with pointers instead of computing the subscript each time. Advancing down a higher dimension of an array with a stride. Backus had those working in a FORTRAN compiler in the 1950s. If FORTRAN had been slower than assembly, nobody would have used it; machines were too slow and too expensive.
DasIch 10y ago
You mean like the array[1] that's in the standard library? [1]: https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html

jdlyga 10y ago

If only it was also available for C++. Armadillo is nice but it has a lot of important differences, and doesn't seem to support more than 3 dimensional matrices.

keypusher 10y ago
There was some effort toward getting it in Boost. https://github.com/ndarray/Boost.NumPy https://github.com/ndarray/Boost.NumPy
chombier 10y ago
You might want to check out the Eigen library: http://eigen.tuxfamily.org/ http://eigen.tuxfamily.org/
srean 10y ago
If you want multidimensional arrays in C++, then blitz++ is your friend. There is Eigen but it wont do multidimensional stuff. Although there might be a few projects out there that wrap the multidimensional functionality around Eigen arrays.
- kersny 10y ago
  One such multidimensional extension of Eigen is from Google and I believe was extracted from TensorFlow: https://bitbucket.org/eigen/eigen/src/039efd86b75ce1b72befb7e0a5bd069baacb17b7/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default https://bitbucket.org/eigen/eigen/src/039efd86b75ce1b72befb7...
lloda 10y ago
Here is my Blitz++-inspired hobby project, supports arbitrary rank: https://github.com/lloda/ra-ra https://github.com/lloda/ra-ra I'm sure you can find more polished efforts if you look around.
naveen99 10y ago
Arrayfire is c++ with Opencl, cuda, and cpu back ends. Limited to 4d. http://arrayfire.com/indexing-with-arrayfire/ http://arrayfire.com/indexing-with-arrayfire/

agibsonccc 10y ago

The numpy array is a great data structure. I know it's not python but I wanted to plug our attempt at this for the jvm, nd4j: http://nd4j.org/tensor http://nd4j.org/tensor We have a java wrapper: https://github.com/deeplearning4j/nd4j https://github.com/deeplearning4j/nd4j Scala: https://github.com/deeplearning4j/nd4s https://github.com/deeplearning4j/nd4s And underlying c++: https://github.com/deeplearning4j/libnd4j https://github.com/deeplearning4j/libnd4j We heavily dog food it in our deep learning library but we're hoping this can be more broadly useful for people.

hatmatrix 10y ago
That's great but what about nd4j's data structures resemble numpy arrays? Without making connection to the original most this comment just reads like typical spam.
- jnbiche 10y ago
  In my experience, it's perfectly acceptable on HN to plug a related project in a comment, as long as it's not done repeatedly for totally unrelated projects (in 5+ years here, I can only think of 1 or 2 examples of this actually happening on a frequent basis). Given that these are open source and not even commercial projects, and that this is presumably an enthusiastic developer for those projects just sharing his/her work with a potentially interested audience, I hardly think the "spam" label is appropriate. And to answer your question specifically, based on what I've just read, ND4J offers an optimized n-dimensional array intended for numeric/scientific computing, very much in the manner of numpy. This is presumably what the comment's "our attempt at this for the jvm" refers to.
  - agibsonccc 10y ago
    Yes it's a port of the associated concepts to the jvm. FWIW: we actually do use this commercially ;). It's apache though so do what you want with it etc. Apologies if it came off like that but the main intent was to show an alternative implementation. If you actually look at the nd4j.org link you'll also notice a simpler/more concise description of the same concepts. Eg: linear buffers,shapes, strides, offsets, dimensions

noobermin 10y ago

This type is my bread and butter. I breathe numpy arrays in my research.

jeroenjanssens 10y ago
What an interesting juxtaposition of idioms :)

hatmatrix 10y ago

It's a shame that there is no consolidated way to provide index labels to both columns and rows without wrapping this in another class (e.g., your own, or pandas). R's matrices and arrays are very convenient as they provide this functionality out-of-the-box.

cgearhart 10y ago
I'm not terribly familiar with R, but I've used Pandas a bit. Could you accomplish what you're looking for with structured arrays or record arrays in Numpy? http://docs.scipy.org/doc/numpy/user/basics.rec.html http://docs.scipy.org/doc/numpy/user/basics.rec.html
- hatmatrix 10y ago
  Record arrays only provide indexing by name along columns, unfortunately. So if you want index labels along both dimensions (in 2-D), Pandas is necessary. There is no good option for the N-D case that I can find (R's native array has this capability by default though).

nl 10y ago

If you like NumPy, it might be worth taking a look at Dask[1]. Very similar API, but designed to support arrays larger than memory, as well as operations across a network cluster. [1] http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/

denfromufa 10y ago

dynd is next generation numpy for language-agnostic usage: https://github.com/libdynd/libdynd https://github.com/libdynd/libdynd