5 ms·

Even though I don't know what a Tensor is, I had a suspicion that TensorFlow was really just "MatrixFlow". I felt validated after reading myth 1, but I'm still

Even though I don't know what a Tensor is, I had a suspicion that TensorFlow was really just "MatrixFlow". I felt validated after reading myth 1, but I'm still trying to wrap my head around the difference between Tensors & Matrices. I have a feeling that I missing out on something beautiful, like Fourier Transforms, and when I finally get it a deep smile will spread across my face.

deleted 8y ago
[deleted]
stilley2 8y ago
This is probably wrong, but I always think of tensor's as n-dimensional generalizations of matrices with units attached. Edit: After some wikipediaing, "bases" might be a better word than "units."
- ummonk 8y ago
  Well intuitively an n-dimensional generalization of matrices would just be a big multi-dimensional table. But a tensor is different in that you have some number of dimensions which are covariant and some number which are contravariant. Additionally, you've sort of got it backwards. A matrix with units (and a set of basis vectors) attached is one representation of a rank (1, 1) tensor. But it's not really a unique representation of the tensor - you could choose a different set of basis vectors and come up with a different matrix representation of the exact same tensor. The tensor is an entity, while the matrix is a representation of an entity within a given coordinate system.
artwr 8y ago
Tensors are in effect a generalization of matrices in higher dimensions. A tensor of dimension 3 is a similar step up to the step from a linear array to a matrix. They arise in all sorts of places though fluids is where I met them first. Operations on tensors are also a bit more challenging than operations on matrices.
mr_toad 8y ago
https://math.stackexchange.com/questions/412423/what-are-the-differences-between-a-matrix-and-a-tensor https://math.stackexchange.com/questions/412423/what-are-the...
- max_likelihood 8y ago
  Thanks for linking. My tldr from the top-rated answer: the components of a Tensor can be written in "matrix" form (i.e. a 2D array of numbers), but the Tensor is not that matrix. Ultimately, "a Tensor is what transforms like a Tensor".
antognini 8y ago
The way "tensor" is typically used in machine learning it really is just an n-dimensional generalization of a matrix. In physics, however, a tensor has a more specific meaning. In this context, certain 2-dimensional tensors can be represented as matrices, but a matrix is a distinct concept. A bit more precisely, in physics a tensor is an object that transforms a particular way during coordinate transformations. Intuitively this means that a tensor must be some physical "thing". A classical example of a tensor is the moment of inertia tensor. Every 3-d object has a moment of inertia tensor. This tells you how the torque relates to angular acceleration, and it will in general be different across different axes of the object. Now, you can choose any three (non-collinear) directions you want and write down a matrix which represents the tensor in that basis, but this representation is fundamentally coordinate dependent. The moment of inertia tensor, by contrast is a coordinate-independent entity. Just like a vector, it will have certain values in certain reference frames, but the vector itself transcends any coordinate system. (Though this is a bit of tautology since a vector is a 1-dimensional tensor.)
- selestify 8y ago
  How do you represent a coordinate-independent tensor? Don't you still need a basis?
  - antognini 8y ago
    You just call it something like T. If you want any numbers you need a coordinate basis. For those interested, the first chapter of Kip Thorne's book has a good, though idiosyncratic, explanation of tensors: http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K.pdf http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K....
    - kgwgk 8y ago
      The first lectures here https://theoreticalminimum.com/courses/general-relativity/2012/fall https://theoreticalminimum.com/courses/general-relativity/20... are mostly an introduction to tensor calculus.
- 8y ago
obastani 8y ago
In this case I don't think what the article says is exactly right. Naively, tensors are just n-dimensional arrays, which TensorFlow supports. The paper linked in the article appears more to be talking about how derivatives of tensors are represented in TensorFlow. The difference doesn't seem to matter unless you are taking higher-order derivatives. This makes sense, since TensorFlow is focused on first-order derivatives needed for gradient descent, but traditional machine learning algorithms also rely on second-order derivatives to make use of more powerful optimization algorithms based on Newton's method. I'm not sure exactly where the difference comes from, but it comes from a convenient notation for tensors used in physics, known as Einstein notation (Einstein invented this notation to make his life easier when deriving general relativity). In this notation, tensors are represented by a single scalar variable. For example, matrix multiplication y = A x is expressed as y_i = A_ij x_j. If I understand correctly, the paper points out that an algorithm for computing derivatives based on this notation is faster for taking higher-order derivatives compared to using TensorFlow. Mathematically, tensors are more complicated objects. Basically, they are what you get when you take higher-order derivatives of a function. In particular, the first-order derivative of a function f: R^n -> R^m at a point x \in R^n is the best linear function A_x \in R^{m X n} that approximates the original function, i.e., f(x + dx) ~= f(x) + A_x dx. A linear function is represented by a matrix, so a first-order derivative is a matrix. If I take the second-order derivative, I get a more complicated object B_x, which represents the quadratic term in the Taylor expansion: f(x + dx) ~= f(x) + A_x dx + B_x(dx, dx) where B_x(a, b) is a linear function (or more precisely, a "multilinear" function) of two vectors a, b (which are the same in the above formula). That is, whereas A_x is a (linear) function R^n -> R^m, B_x is a (multilinear) function R^n X R^n -> R^m. This mathematical object B_x is an example of a tensor. In R^n and R^m, tensors are pretty boring, but they become more interesting when dealing with functions on manifolds.
- max_likelihood 8y ago
  I have often seen Tensors introduced in the context of Einstein's General Relativity. I read this article on HN: https://news.ycombinator.com/item?id=19055994 https://news.ycombinator.com/item?id=19055994 a few weeks back and found it really helpful.
- 8y ago
amelius 8y ago
It's not even MatrixFlow, but more ArrayFlow.

deleted 8y ago

[deleted]

stilley2 8y ago

This is probably wrong, but I always think of tensor's as n-dimensional generalizations of matrices with units attached. Edit: After some wikipediaing, "bases" might be a better word than "units."

ummonk 8y ago
Well intuitively an n-dimensional generalization of matrices would just be a big multi-dimensional table. But a tensor is different in that you have some number of dimensions which are covariant and some number which are contravariant. Additionally, you've sort of got it backwards. A matrix with units (and a set of basis vectors) attached is one representation of a rank (1, 1) tensor. But it's not really a unique representation of the tensor - you could choose a different set of basis vectors and come up with a different matrix representation of the exact same tensor. The tensor is an entity, while the matrix is a representation of an entity within a given coordinate system.

artwr 8y ago

Tensors are in effect a generalization of matrices in higher dimensions. A tensor of dimension 3 is a similar step up to the step from a linear array to a matrix. They arise in all sorts of places though fluids is where I met them first. Operations on tensors are also a bit more challenging than operations on matrices.

mr_toad 8y ago

https://math.stackexchange.com/questions/412423/what-are-the-differences-between-a-matrix-and-a-tensor https://math.stackexchange.com/questions/412423/what-are-the...

max_likelihood 8y ago
Thanks for linking. My tldr from the top-rated answer: the components of a Tensor can be written in "matrix" form (i.e. a 2D array of numbers), but the Tensor is not that matrix. Ultimately, "a Tensor is what transforms like a Tensor".

antognini 8y ago

The way "tensor" is typically used in machine learning it really is just an n-dimensional generalization of a matrix. In physics, however, a tensor has a more specific meaning. In this context, certain 2-dimensional tensors can be represented as matrices, but a matrix is a distinct concept. A bit more precisely, in physics a tensor is an object that transforms a particular way during coordinate transformations. Intuitively this means that a tensor must be some physical "thing". A classical example of a tensor is the moment of inertia tensor. Every 3-d object has a moment of inertia tensor. This tells you how the torque relates to angular acceleration, and it will in general be different across different axes of the object. Now, you can choose any three (non-collinear) directions you want and write down a matrix which represents the tensor in that basis, but this representation is fundamentally coordinate dependent. The moment of inertia tensor, by contrast is a coordinate-independent entity. Just like a vector, it will have certain values in certain reference frames, but the vector itself transcends any coordinate system. (Though this is a bit of tautology since a vector is a 1-dimensional tensor.)

selestify 8y ago
How do you represent a coordinate-independent tensor? Don't you still need a basis?
- antognini 8y ago
  You just call it something like T. If you want any numbers you need a coordinate basis. For those interested, the first chapter of Kip Thorne's book has a good, though idiosyncratic, explanation of tensors: http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K.pdf http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K....
  - kgwgk 8y ago
    The first lectures here https://theoreticalminimum.com/courses/general-relativity/2012/fall https://theoreticalminimum.com/courses/general-relativity/20... are mostly an introduction to tensor calculus.
8y ago

obastani 8y ago

In this case I don't think what the article says is exactly right. Naively, tensors are just n-dimensional arrays, which TensorFlow supports. The paper linked in the article appears more to be talking about how derivatives of tensors are represented in TensorFlow. The difference doesn't seem to matter unless you are taking higher-order derivatives. This makes sense, since TensorFlow is focused on first-order derivatives needed for gradient descent, but traditional machine learning algorithms also rely on second-order derivatives to make use of more powerful optimization algorithms based on Newton's method. I'm not sure exactly where the difference comes from, but it comes from a convenient notation for tensors used in physics, known as Einstein notation (Einstein invented this notation to make his life easier when deriving general relativity). In this notation, tensors are represented by a single scalar variable. For example, matrix multiplication y = A x is expressed as y_i = A_ij x_j. If I understand correctly, the paper points out that an algorithm for computing derivatives based on this notation is faster for taking higher-order derivatives compared to using TensorFlow. Mathematically, tensors are more complicated objects. Basically, they are what you get when you take higher-order derivatives of a function. In particular, the first-order derivative of a function f: R^n -> R^m at a point x \in R^n is the best linear function A_x \in R^{m X n} that approximates the original function, i.e., f(x + dx) ~= f(x) + A_x dx. A linear function is represented by a matrix, so a first-order derivative is a matrix. If I take the second-order derivative, I get a more complicated object B_x, which represents the quadratic term in the Taylor expansion: f(x + dx) ~= f(x) + A_x dx + B_x(dx, dx) where B_x(a, b) is a linear function (or more precisely, a "multilinear" function) of two vectors a, b (which are the same in the above formula). That is, whereas A_x is a (linear) function R^n -> R^m, B_x is a (multilinear) function R^n X R^n -> R^m. This mathematical object B_x is an example of a tensor. In R^n and R^m, tensors are pretty boring, but they become more interesting when dealing with functions on manifolds.

max_likelihood 8y ago
I have often seen Tensors introduced in the context of Einstein's General Relativity. I read this article on HN: https://news.ycombinator.com/item?id=19055994 https://news.ycombinator.com/item?id=19055994 a few weeks back and found it really helpful.
8y ago

amelius 8y ago

It's not even MatrixFlow, but more ArrayFlow.