3 ms·

i wonder how this compares with einsum https://ajcr.net/Basic-guide-to-einsum/ https://ajcr.net/Basic-guide-to-einsum/ > The einsum function is one of NumPy’s

by shoo 5y ago

i wonder how this compares with einsum https://ajcr.net/Basic-guide-to-einsum/ https://ajcr.net/Basic-guide-to-einsum/ > The einsum function is one of NumPy’s jewels. It can often outperform familiar array functions in terms of speed and memory efficiency, thanks to its expressive power and smart loops it seems like einsum has been ported to torch & tensorflow https://numpy.org/doc/stable/reference/generated/numpy.einsum.html https://numpy.org/doc/stable/reference/generated/numpy.einsu... ; https://pytorch.org/docs/stable/generated/torch.einsum.html https://pytorch.org/docs/stable/generated/torch.einsum.html ; https://www.tensorflow.org/api_docs/python/tf/einsum https://www.tensorflow.org/api_docs/python/tf/einsum

brrrrrm 5y ago
Great question! einsum is almost exactly broadcast + sum. as_strided lets you do “funky” strides to enable all the various convolutions (sliding windows aren’t easily represented with einsum).
dataflow 5y ago
Several years ago I used tensordot and from memory it seemed to perform slightly better than einsum. Not sure if that's still the case but I remember really enjoying using it.
ssivark 5y ago
Some short answers to this and following questions: 1. Einsum doesn’t AFAIK allow custom strides. So Einsum does only the mapreduce part (and that too only for multiplication & summation combo). To get a taste for the absolute brilliance of a slightly more generic interface, check out Tullio in Julia. 2. Tensordot etc use BLAS under the hood IIRC and can therefore parallelize automatically. The Einsum backend is good old fashioned single threaded C code, so it’s fast (and inflexible) but single core.
yobbo 5y ago
Einsum is a generalization of matrix multiplications to "tensor multiplications". It can take as_strided objects as input. I don't think there exists a cuda-accelerated einsum, so the equivalent matmul is always faster? With cuda, it might be implemented by "exploding" the inputs to matrices, which means it can quickly run out of memory?

by shoo 5y ago

brrrrrm 5y ago

Great question! einsum is almost exactly broadcast + sum. as_strided lets you do “funky” strides to enable all the various convolutions (sliding windows aren’t easily represented with einsum).

dataflow 5y ago

Several years ago I used tensordot and from memory it seemed to perform slightly better than einsum. Not sure if that's still the case but I remember really enjoying using it.

ssivark 5y ago

Some short answers to this and following questions: 1. Einsum doesn’t AFAIK allow custom strides. So Einsum does only the mapreduce part (and that too only for multiplication & summation combo). To get a taste for the absolute brilliance of a slightly more generic interface, check out Tullio in Julia. 2. Tensordot etc use BLAS under the hood IIRC and can therefore parallelize automatically. The Einsum backend is good old fashioned single threaded C code, so it’s fast (and inflexible) but single core.

yobbo 5y ago

Einsum is a generalization of matrix multiplications to "tensor multiplications". It can take as_strided objects as input. I don't think there exists a cuda-accelerated einsum, so the equivalent matmul is always faster? With cuda, it might be implemented by "exploding" the inputs to matrices, which means it can quickly run out of memory?