4 ms·
Mathematically tensors are multilinear maps, not data structures. Typically tensors in deep learning represent multilinear mappings forwards and backwards. Sca
by nabla9 8y ago
Mathematically tensors are multilinear maps, not data structures. Typically tensors in deep learning represent multilinear mappings forwards and backwards. Scalars, vectors and matrices can also be tensors when they represent mapping.
If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.
- thanatropism 8y agoWe've got a winner. Tensors "do" something; they're functions. In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however. I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.
- twtw 8y agoIs it accurate to say that rank-n tensor is to n-D array of numbers as linear transformation is to matrix?
- joshuamorton 8y ago>We've got a winner. Tensors "do" something; they're functions. And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video). >neural networks as ODEs You're a few months late :P [0] [0]: https://arxiv.org/abs/1806.07366 https://arxiv.org/abs/1806.07366
- Koshkin 8y ago> they're functions Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.
- thanatropism 8y agoEhhh. Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but... You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.
- amelius 8y ago> Mathematically tensors are multilinear maps, not data structures. Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.