3 ms·

The tensor diagrams are not quite standard (yet). That's why I also include more "classical" neural network diagrams next to them. I've recently been working o

The tensor diagrams are not quite standard (yet). That's why I also include more "classical" neural network diagrams next to them. I've recently been working on a library for doing automatic manipulation and differentiation of tensor diagrams (https://github.com/thomasahle/tensorgrad https://github.com/thomasahle/tensorgrad), and to me they are clearly a cleaner notation. For a beautiful introduction to tensor networks, see also Jordan Taylor's blog post (https://www.lesswrong.com/posts/BQKKQiBmc63fwjDrj/graphical-tensor-notation-for-interpretability https://www.lesswrong.com/posts/BQKKQiBmc63fwjDrj/graphical-...)

programjames 2y ago
These remind me of interaction combinators [1], which are being used in the Bend programming language [1]. I think it'd be good for the standard to also be a valid interaction net. [1]: https://core.ac.uk/download/pdf/81113716.pdf https://core.ac.uk/download/pdf/81113716.pdf [2]: https://news.ycombinator.com/item?id=40390287 https://news.ycombinator.com/item?id=40390287
- thomasahle 2y ago
  This stuff is super cool! It basically generalizes tensor diagrams to general computational graphs. However, when thinking about ML architectures, I actually like that classical tensor diagrams make it harder to express non-associative architectures. E.g. RNNs are much harder to write than Transformers.
cgadski 2y ago
After learning about tensor diagrams a few months ago, they're my default notation for tensors. I liked your chart and also Jordan Taylor's diagram for multi-head attention. Some notes for other readers seeing this for the first time: My favorite property of that these diagrams is that they make it easy to re-interpret a multilinear expression as a multilinear function of any of its variables. For example, in standard matrix notation you'd write x^T A x to get a quadratic form with respect to the variable x. I think most people read this either left to right or right to left: take a matrix-vector product, and then take an inner product between vectors. Tensor notation is more like prolog: the diagram x - A - x involves these two indices/variables (the lines) "bound" by three tensors/relations (A and two copies of x.) That framing makes it easier to think about the expression as a function of A: it's just a "Frobenius inner product" between -A- and the tensor product -x x-. The same thing happens with the inner product between a signal and a convolution of two other signals. In standard notation it might take a little thought to remember how to differentiate <x, y * z> with respect to y (<x, y * z> = <y, x * z'> where x' is a time-reversal), but thinking with a tensor diagram reminds you to focus on the relation x = y + z (a 3-dimensional tensor) constraining the indices x, y and z of your three signals. All of this becomes increasingly critical when you have more indices involved. For example, how can you write the flattened matrix vec(AX + XB) as a matrix-vector product of vec(X) so we can solve the equation AX + XB = C? (Example stolen from your book.) I still have to get a hold of all the rules for dealing with non-linearities ("bubbles") though. I'll have to take a look at your tensor cookbook :) I'm also sad that I can't write tensor diagrams easily in my digital notes. Tensor diagrams are algebraically the same thing as factor graphs in probability theory. (Tensors correspond to factors and indices correspond to variables.) The only difference is that factors in probability theory need to be non-negative. You can define a contraction over indices for tensors taking values in any semiring though. The max-plus semiring gives you maximum log-likelihood problems, and so on.
- thomasahle 2y ago
  I'm really glad you've found my "book" useful! Makes me want to continue writing it :)
cshimmin 2y ago
I'm familiar with almost all of these architectures, but not the tensor diagram notation. I can't figure out what "B" is? I thought maybe it's a bias vector, but then why does it only appear on the input data, and not on subsequent fc layers?
- cgadski 2y ago
  B is the number of data vectors going on. You can erase the line labeled by B without much loss. (You just get the diagram for the feed-forward of a single vector.)

programjames 2y ago

These remind me of interaction combinators [1], which are being used in the Bend programming language [1]. I think it'd be good for the standard to also be a valid interaction net. [1]: https://core.ac.uk/download/pdf/81113716.pdf https://core.ac.uk/download/pdf/81113716.pdf [2]: https://news.ycombinator.com/item?id=40390287 https://news.ycombinator.com/item?id=40390287

thomasahle 2y ago
This stuff is super cool! It basically generalizes tensor diagrams to general computational graphs. However, when thinking about ML architectures, I actually like that classical tensor diagrams make it harder to express non-associative architectures. E.g. RNNs are much harder to write than Transformers.

cgadski 2y ago

After learning about tensor diagrams a few months ago, they're my default notation for tensors. I liked your chart and also Jordan Taylor's diagram for multi-head attention. Some notes for other readers seeing this for the first time: My favorite property of that these diagrams is that they make it easy to re-interpret a multilinear expression as a multilinear function of any of its variables. For example, in standard matrix notation you'd write x^T A x to get a quadratic form with respect to the variable x. I think most people read this either left to right or right to left: take a matrix-vector product, and then take an inner product between vectors. Tensor notation is more like prolog: the diagram x - A - x involves these two indices/variables (the lines) "bound" by three tensors/relations (A and two copies of x.) That framing makes it easier to think about the expression as a function of A: it's just a "Frobenius inner product" between -A- and the tensor product -x x-. The same thing happens with the inner product between a signal and a convolution of two other signals. In standard notation it might take a little thought to remember how to differentiate <x, y * z> with respect to y (<x, y * z> = <y, x * z'> where x' is a time-reversal), but thinking with a tensor diagram reminds you to focus on the relation x = y + z (a 3-dimensional tensor) constraining the indices x, y and z of your three signals. All of this becomes increasingly critical when you have more indices involved. For example, how can you write the flattened matrix vec(AX + XB) as a matrix-vector product of vec(X) so we can solve the equation AX + XB = C? (Example stolen from your book.) I still have to get a hold of all the rules for dealing with non-linearities ("bubbles") though. I'll have to take a look at your tensor cookbook :) I'm also sad that I can't write tensor diagrams easily in my digital notes. Tensor diagrams are algebraically the same thing as factor graphs in probability theory. (Tensors correspond to factors and indices correspond to variables.) The only difference is that factors in probability theory need to be non-negative. You can define a contraction over indices for tensors taking values in any semiring though. The max-plus semiring gives you maximum log-likelihood problems, and so on.

thomasahle 2y ago
I'm really glad you've found my "book" useful! Makes me want to continue writing it :)

cshimmin 2y ago

I'm familiar with almost all of these architectures, but not the tensor diagram notation. I can't figure out what "B" is? I thought maybe it's a bias vector, but then why does it only appear on the input data, and not on subsequent fc layers?

cgadski 2y ago
B is the number of data vectors going on. You can erase the line labeled by B without much loss. (You just get the diagram for the feed-forward of a single vector.)