5 ms·

I’m a newbie in NN topic and feel surprised to hear that noone uses Numpy in actual NN implementations, although it’s written in C++ and highly optimized. Why i

I’m a newbie in NN topic and feel surprised to hear that noone uses Numpy in actual NN implementations, although it’s written in C++ and highly optimized. Why is that ? And, how about Gonum (Go equivalent) ? Finally, I’m currently going through the deeplearning.ai program. I got one week left, and will experiment with building some apps. Which technical stack should I choose ?

jimmy_dean 7y ago
The main reason numpy isn't used in NN implementations is that it does not, natively speaking, have GPU support. Tensor structures in PyTorch and TensorFlow have the most solid backend support for GPUs (TPUs) and have a good amount of numpy's ndarray capabilities. There is recent work to put numpy on the same footing for deep learning. Check https://github.com/google/jax https://github.com/google/jax
csande17 7y ago
Most software (including Python and Numpy and Go and pretty much every Rust program) runs on your computer's CPU. The CPU is good at running programs with a lot of different instructions and if-statements and loops and stuff. But for neural networks, people often prefer to use special hardware like graphics cards, since graphics cards are really good at doing relatively simple math on many pieces of data at once. So they create special libraries like TensorFlow that can send commands to the graphics card instead of doing the math on the CPU. (And they don't use Numpy because even though it's highly optimized, it's highly optimized for CPUs, and graphics cards are a lot faster than CPUs at running neural networks.)
PeterisP 7y ago
Numpy is at a too low level for applied NN implementations. If I'm not doing research on new methods but want to build a model for a particular problem using well-known best practices, then all the custom code that my app needs and what I need to write is about the transformation and representation and structure of my particular dataset and task; but things like, for example, optimized backpropagation for a stack of bidirectional LSTM layers are not custom for my app, they're generic - why would I need or want to reimplement them except as a learning exercise? That'd be like reinventing a bicycle, for generic things like that I'd want to call a library where that code is well-tested and well-optimized (including for GPU usage) by someone else, and that library isn't numpy. Numpy works at the granularity of matrix multiplication ops, but applied ML works at the granularity of whole layers such as self-attention or LSTM or CNN; which perhaps are not that complex conceptually, but do require some attention to implement properly in an optimized way; you can implement them in numpy but you probably shouldn't (unless as a learning exercise).
chillee 7y ago
There's 3 fundamental gaps between Numpy and NN libraries (other commenters have jointly mentioned 2/3) 1. Numpy doesn't run on GPU. 2. Numpy isn't high level enough for NN building. 3. Numpy doesn't have auto differentiation. Other options solve some of these - Autograd solves 3, Jax solves 1 and 3, etc. But if you want all 3 then you want to use Pytorch.

jimmy_dean 7y ago

The main reason numpy isn't used in NN implementations is that it does not, natively speaking, have GPU support. Tensor structures in PyTorch and TensorFlow have the most solid backend support for GPUs (TPUs) and have a good amount of numpy's ndarray capabilities. There is recent work to put numpy on the same footing for deep learning. Check https://github.com/google/jax https://github.com/google/jax

csande17 7y ago

Most software (including Python and Numpy and Go and pretty much every Rust program) runs on your computer's CPU. The CPU is good at running programs with a lot of different instructions and if-statements and loops and stuff. But for neural networks, people often prefer to use special hardware like graphics cards, since graphics cards are really good at doing relatively simple math on many pieces of data at once. So they create special libraries like TensorFlow that can send commands to the graphics card instead of doing the math on the CPU. (And they don't use Numpy because even though it's highly optimized, it's highly optimized for CPUs, and graphics cards are a lot faster than CPUs at running neural networks.)

PeterisP 7y ago

Numpy is at a too low level for applied NN implementations. If I'm not doing research on new methods but want to build a model for a particular problem using well-known best practices, then all the custom code that my app needs and what I need to write is about the transformation and representation and structure of my particular dataset and task; but things like, for example, optimized backpropagation for a stack of bidirectional LSTM layers are not custom for my app, they're generic - why would I need or want to reimplement them except as a learning exercise? That'd be like reinventing a bicycle, for generic things like that I'd want to call a library where that code is well-tested and well-optimized (including for GPU usage) by someone else, and that library isn't numpy. Numpy works at the granularity of matrix multiplication ops, but applied ML works at the granularity of whole layers such as self-attention or LSTM or CNN; which perhaps are not that complex conceptually, but do require some attention to implement properly in an optimized way; you can implement them in numpy but you probably shouldn't (unless as a learning exercise).

chillee 7y ago

There's 3 fundamental gaps between Numpy and NN libraries (other commenters have jointly mentioned 2/3) 1. Numpy doesn't run on GPU. 2. Numpy isn't high level enough for NN building. 3. Numpy doesn't have auto differentiation. Other options solve some of these - Autograd solves 3, Jax solves 1 and 3, etc. But if you want all 3 then you want to use Pytorch.