8 ms·
The Tensor Cookbook (2024)
- hnarayanan 2y agoWhat even is this notation??
- abecedarius 2y agoSee the link at bottom right to "Penrose graphical notation". (I have nothing to do with the site.)
- hnarayanan 2y agoSorry. I was just making an exasperated statement that roughly boils down to “what is wrong with mainstream notation and who is going to learn or care about this completely different thing?”
- abecedarius 2y agoAnd I was just trying to help. :) Easy to miss that link. Unfortunately Wikipedia generally isn't great for motivation on math topics. Tensor ops in e.g. pytorch are pretty opaque to me, too much implicit on shapes which can change as you go down a pipeline. Maybe I'll come to appreciate it better.
- hnarayanan 2y agoAnd thank you. (I come from an old generation of people trained in linear algebra and differential geometry, well before the current ML era so I am holding onto my "get off my lawn" grumpiness.)
- robblbobbl 2y agoGreat job!
- thomasahle 2y agoThank you! Please share any feedback you have!
- keithalewis 2y agoThis is incomplete, incorrect, and irrelevant. Standard notation already exists. I'm sure it is fun to draw squiggly lines and some people enjoy reinventing the wheel. Spend some time learning what others have taught us before striking out on your own lonely path.
- llm_trw 2y agoThis is standard notation that's been used for decades. https://arxiv.org/abs/2402.01790v1 https://arxiv.org/abs/2402.01790v1
- chriskanan 2y agoThis paper motivates and explains concepts much better than the Tensor Cookbook.
- llm_trw 2y agoThe cookbook is a work in progress by the looks of it.
- thomasahle 2y agoI'm hoping the Tensor Cookbook can become as engaging to read for others as Jordan Taylor's paper was to me. If you have any thoughts on where I lose people, please share!
- keithalewis 2y ago"This book aims to standardize the notation for tensor diagrams..." https://youtu.be/zELbzXAmcUA?t=73 https://youtu.be/zELbzXAmcUA?t=73
- thomasahle 2y agoTensor diagrams are standard, but some notation is missing. My goal was to be able to handle the entire Matrix Cookbook. For this I needed a good notation for functions applied to specific dimensions and broadcasting over the rest. Like softmax in a transformer. The function chapter is still under development in the book though. So if you have any good references for how it's been done graphically in the past, that I might have missed, feel free to share them.
- gweinberg 2y agoWhy is counting lines easier than just using numbers? A vector is a 1 tensor, a matrix is a 2 tensor, a 3 tensor is a 3 tensor, and a scalar is a zero tensor!
- absolutelastone 2y agoIt's easier to see where the contractions are, since the same line connects two tensors, versus hunting for repeated indices in a mess of many indices among many tensors multiplied together.
- chriskanan 2y agoThe Matrix Cookbook was one of the most useful texts I had in the late 2000s during my PhD for computing update rules for loss functions. With autograd now widely employed, I don't think it is as fundamental as it was. Given that, what additional value is The Tensor Cookbook providing such that it is worth learning an entirely new notation (for me)? It would probably require long term usage to really benefit from these visual depictions.
- kengoa 2y agoI encountered it during my masters in 2019 and it still remains my favourite textbook to this day.
- thomasahle 2y agoThe Matrix Cookbook has a lot of useful formulas, but they don't have much explanation. In the Tensor Cookbook I aim to show the same formulas using tensor diagrams, in a way that (hopefully) make them seem so obvious you don't even need the book afterwards.
- chriskanan 2y agoThat's a good point regarding the Matrix Cookbook. I didn't start using it until I'd already learned a lot of linear algebra and how to do matrix calculus by hand, so I didn't really need much of an explanation when I first discovered it in the 2000s. In contrast, the Tensor Cookbook was my first introduction to tensor diagrams, so I didn't have any prior experience with them to lean on. It certainly looks like a useful and powerful technique, but it seems like something that warrants almost a crash course in the topic with some exercises rather than just jumping in.
- thomasahle 2y ago> It certainly looks like a useful and powerful technique, but it seems like something that warrants almost a crash course in the topic with some exercises rather than just jumping in. I guess my personal crash course was writing a book on the topic. And a software library... Ironically this makes it harder for me to appreciate the level of explanation needed for others. Thus I rely on people like you to tell me where the chain jumps off, so I can expand the sections. Please let me know what sections were too quickly skipped through!
- galaxyLogic 2y agoMy ideal preference would be a notation I can type on a standard keyboard.
- fluorinerocket 2y agoWriting code with einsum, and knowing what each index represents like time, space, element is so much more intuitive then trying to reason about just how need to transpose things so that everything works to do the tensor contraction you want. I will be studying this
- thomasahle 2y agoPlease send me any feedback you have! I'm still figuring out what presentation and content is most interesting to people!
- physicsguy 2y agoNot saying this notation isn’t useful, but the big advantage with index notation is that a large, large number of people are already familiar with it because it’s a core part of many Physics or Maths degrees, so will be using it naturally. If you have to work with such people then it’s useful to “speak a common language”
- thomasahle 2y agoYou can always draw the diagrams while you're thinking, and then "compile down" to index notation when you share it with others. But luckily tensor diagrams is quite standard in many fields.
- mufasachan 2y agoI do not know the people's background of a lot of comments here. They might have much more experiences than me with tensors. But, in my deep learning code and works, when I need to design an operation that involves a mix as little as 3 tensors with 4+ dimensions, I always struggle. I need to draft some slices to understand which slice should be contracted etc.. Many times the shape of the output is not even clear in my mind. Plus, add some padding maskS on the tensors and it confuses me quite a lot. I really like this notation, the last example of 1.1 is readable in its sum formulation, but the diagram formulation is much more "alive" in my mind. I am really lost here if I have missed something about indices notations with tensors or some visualization techniques. Or maybe the confusion of a tensor operation depends of the field? Or maybe I just miss practices and experiences with indices notations...
- thomasahle 2y agoStandard tensor notation in pytorch and other libraries is very indirect, and often shapes are only documented in comments. I definitely find it helps to draw parts of your architecture as a tensor diagram. Or perhaps use a library like tensorgrad which makes eveything explicit.