4 ms·
“as_strided” and “sum” are all you need
- hikerclimber1 5y agoEverything is subjective. Especially laws.
- Zababa 5y ago"...to implement the non-pointwise operations in a neural network."
- shoo 5y agoi wonder how this compares with einsum https://ajcr.net/Basic-guide-to-einsum/ https://ajcr.net/Basic-guide-to-einsum/ > The einsum function is one of NumPy’s jewels. It can often outperform familiar array functions in terms of speed and memory efficiency, thanks to its expressive power and smart loops it seems like einsum has been ported to torch & tensorflow https://numpy.org/doc/stable/reference/generated/numpy.einsum.html https://numpy.org/doc/stable/reference/generated/numpy.einsu... ; https://pytorch.org/docs/stable/generated/torch.einsum.html https://pytorch.org/docs/stable/generated/torch.einsum.html ; https://www.tensorflow.org/api_docs/python/tf/einsum https://www.tensorflow.org/api_docs/python/tf/einsum
- brrrrrm 5y agoGreat question! einsum is almost exactly broadcast + sum. as_strided lets you do “funky” strides to enable all the various convolutions (sliding windows aren’t easily represented with einsum).
- dataflow 5y agoSeveral years ago I used tensordot and from memory it seemed to perform slightly better than einsum. Not sure if that's still the case but I remember really enjoying using it.
- ssivark 5y agoSome short answers to this and following questions: 1. Einsum doesn’t AFAIK allow custom strides. So Einsum does only the mapreduce part (and that too only for multiplication & summation combo). To get a taste for the absolute brilliance of a slightly more generic interface, check out Tullio in Julia. 2. Tensordot etc use BLAS under the hood IIRC and can therefore parallelize automatically. The Einsum backend is good old fashioned single threaded C code, so it’s fast (and inflexible) but single core.
- yobbo 5y agoEinsum is a generalization of matrix multiplications to "tensor multiplications". It can take as_strided objects as input. I don't think there exists a cuda-accelerated einsum, so the equivalent matmul is always faster? With cuda, it might be implemented by "exploding" the inputs to matrices, which means it can quickly run out of memory?
- gerdesj 5y agoA neural network is not the useful default answer. Try soap instead.
- ailun 5y agoThis is one of the best explanations I’ve seen of as_strided. Could have used this a few months ago when we had to use it to implement a CNN in class!
- Noumenon72 5y ago> as_strided is trickier to explain. It allows users to reinterpet input tensors by specifying custom sizes, strides and offsets for the dimensions of the output. This doesn't work as an explanation without defining what a "stride" is. You can't explain addition just by showing some examples of "before addition" and "after addition". Here's what I found at https://ipython-books.github.io/46-using-stride-tricks-with-numpy/ https://ipython-books.github.io/46-using-stride-tricks-with-...: > Strides describe, in any dimension, how many bytes we need to jump over in the data buffer to go from one item to the next. So the stride in a 1-D array of float64 is 8 because you need to go 8 bytes forward to go from one item to the next. Change it to 32 and you have a way to access every fourth item without defining a new array.
- brrrrrm 5y agothanks for adding this context! One caveat is that the PyTorch strides used in the post are defined by elements rather than byte-wise.