5 ms·

The way "tensor" is typically used in machine learning it really is just an n-dimensional generalization of a matrix. In physics, however, a tensor has a more

The way "tensor" is typically used in machine learning it really is just an n-dimensional generalization of a matrix. In physics, however, a tensor has a more specific meaning. In this context, certain 2-dimensional tensors can be represented as matrices, but a matrix is a distinct concept. A bit more precisely, in physics a tensor is an object that transforms a particular way during coordinate transformations. Intuitively this means that a tensor must be some physical "thing". A classical example of a tensor is the moment of inertia tensor. Every 3-d object has a moment of inertia tensor. This tells you how the torque relates to angular acceleration, and it will in general be different across different axes of the object. Now, you can choose any three (non-collinear) directions you want and write down a matrix which represents the tensor in that basis, but this representation is fundamentally coordinate dependent. The moment of inertia tensor, by contrast is a coordinate-independent entity. Just like a vector, it will have certain values in certain reference frames, but the vector itself transcends any coordinate system. (Though this is a bit of tautology since a vector is a 1-dimensional tensor.)

selestify 8y ago
How do you represent a coordinate-independent tensor? Don't you still need a basis?
- antognini 8y ago
  You just call it something like T. If you want any numbers you need a coordinate basis. For those interested, the first chapter of Kip Thorne's book has a good, though idiosyncratic, explanation of tensors: http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K.pdf http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K....
  - kgwgk 8y ago
    The first lectures here https://theoreticalminimum.com/courses/general-relativity/2012/fall https://theoreticalminimum.com/courses/general-relativity/20... are mostly an introduction to tensor calculus.
cochne 8y ago
I think Tensor in machine learning is more akin to array, just an n-dimensional collection of numbers. It faces the same confusion with a real tensor, as "2-d array" does with "matrix". While a matrix can be represented as a 2-d array, and a tensor can be represented as a n-d array, they have different mathematical connotations. A matrix should be viewed as a 2-d array representing a linear map from on vector space (say R^n) to another (say R^m). We can say this because it is a theorem that any linear map from one (finite dimensional) vector space to another can be represented as a matrix and computed via matrix multiplication. Tensors are the same thing, except the input/output is no longer limited to vector spaces. Consider you have a vector and a bunch (let's say q) of matrices, and you take the matrix product of the vector with all those matrices. You will get q vectors, which you can stick together to form a matrix. This act of multiplying a vector by a bunch of matrices is clearly linear with respect to the input: If we multiply the input by X, every vector will be multiplied by X, so the resulting matrix will be multiplied by X. Suppose you do this operation, T on v to get vectors T1(v), T2(v) ... Tq(v) and on w to get T1(w), T2(w), ... Tq(w). Since all T's are matrix products (linear) then if we do the operation on v + w we will get T1(v) + T1(w), T2(v) + T2(w) ... Tq(v) + Tq(w). Which is essentially T(v) + T(w). So now we know T is linear with respect to the input. Now, this all took a long time to describe, so let's simplify it: How about instead of a group of matrices, we just call this thing a 3-d tensor? we can let i and j index the regular matrix dimensions and make up a new dimension for the matrix we're on, call it k. Now at any coordinate we get a value so it's basically a 3-d array, but it represents something much more specific than that. You can guess how this might generalize to mapping matrices x matrices to 3d tensors or 3d matrices x 3d tensors to 4d tensors and so on. So now the question is, does TensorFlow conflate these? I think it does - somewhat. A convolution can be viewed as a tensor (a single filter maps matrices (images) x 3d-tensor (kernel) to matrices (another image)) so I'd call that a Tensor operation. But consider the input image itself. Is this truly a tensor? If we consider a simple situation, say we have some data vector and we're doing a matrix multiply to get the output of a linear model. Is the input a matrix? I would say no, because we don't think of it as acting on the model, we thing of the model as acting on it, even though what we are doing is really equivalent to multiplying two matrices. Equivalently, I would not call the input image, or any activation in a neural network a true tensor, even though it is numerically equivalent. There are true tensors in TensorFlow, but if you're using high level functions (dense, conv2d) they are usually hidden from the user.
soVeryTired 8y ago
> A bit more precisely, in physics a tensor is an object that transforms a particular way during coordinate transformations No offence, but that's a hideous definition :) For me a (real) tensor is a function that takes an ordered set of N row vectors and M column vectors as arguments, and spits back a real number as a result. It has to be linear in its arguments. That's all folks! By this token a matrix A is a tensor: it takes one row vector x, and one column vector y, and returns a real number xAy. Similarly, a row vector x is a tensor: feed it a column vector y and you get the real number xy. You can dress all this up in the language of linear functionals or n-forms, but at core that's what's going on.
- antognini 8y ago
  Yes, that definition is fine for machine learning, but it's not quite complete for physics. To extend your definition for physics, a tensor is a function that takes an ordered set of N row vectors and M column vectors as arguments and spits back a real, coordinate-invariant number as a result.
  - soVeryTired 8y ago
    I think you get coordinate invariance for free if you think of a vector as an object in its own right, rather than as a tuple in a coordinate system. But then I guess it's more accurate to speak of vectors and covectors than row vectors and column vectors.

selestify 8y ago

How do you represent a coordinate-independent tensor? Don't you still need a basis?

antognini 8y ago
You just call it something like T. If you want any numbers you need a coordinate basis. For those interested, the first chapter of Kip Thorne's book has a good, though idiosyncratic, explanation of tensors: http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K.pdf http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/1201.1.K....
- kgwgk 8y ago
  The first lectures here https://theoreticalminimum.com/courses/general-relativity/2012/fall https://theoreticalminimum.com/courses/general-relativity/20... are mostly an introduction to tensor calculus.

cochne 8y ago

I think Tensor in machine learning is more akin to array, just an n-dimensional collection of numbers. It faces the same confusion with a real tensor, as "2-d array" does with "matrix". While a matrix can be represented as a 2-d array, and a tensor can be represented as a n-d array, they have different mathematical connotations. A matrix should be viewed as a 2-d array representing a linear map from on vector space (say R^n) to another (say R^m). We can say this because it is a theorem that any linear map from one (finite dimensional) vector space to another can be represented as a matrix and computed via matrix multiplication. Tensors are the same thing, except the input/output is no longer limited to vector spaces. Consider you have a vector and a bunch (let's say q) of matrices, and you take the matrix product of the vector with all those matrices. You will get q vectors, which you can stick together to form a matrix. This act of multiplying a vector by a bunch of matrices is clearly linear with respect to the input: If we multiply the input by X, every vector will be multiplied by X, so the resulting matrix will be multiplied by X. Suppose you do this operation, T on v to get vectors T1(v), T2(v) ... Tq(v) and on w to get T1(w), T2(w), ... Tq(w). Since all T's are matrix products (linear) then if we do the operation on v + w we will get T1(v) + T1(w), T2(v) + T2(w) ... Tq(v) + Tq(w). Which is essentially T(v) + T(w). So now we know T is linear with respect to the input. Now, this all took a long time to describe, so let's simplify it: How about instead of a group of matrices, we just call this thing a 3-d tensor? we can let i and j index the regular matrix dimensions and make up a new dimension for the matrix we're on, call it k. Now at any coordinate we get a value so it's basically a 3-d array, but it represents something much more specific than that. You can guess how this might generalize to mapping matrices x matrices to 3d tensors or 3d matrices x 3d tensors to 4d tensors and so on. So now the question is, does TensorFlow conflate these? I think it does - somewhat. A convolution can be viewed as a tensor (a single filter maps matrices (images) x 3d-tensor (kernel) to matrices (another image)) so I'd call that a Tensor operation. But consider the input image itself. Is this truly a tensor? If we consider a simple situation, say we have some data vector and we're doing a matrix multiply to get the output of a linear model. Is the input a matrix? I would say no, because we don't think of it as acting on the model, we thing of the model as acting on it, even though what we are doing is really equivalent to multiplying two matrices. Equivalently, I would not call the input image, or any activation in a neural network a true tensor, even though it is numerically equivalent. There are true tensors in TensorFlow, but if you're using high level functions (dense, conv2d) they are usually hidden from the user.

soVeryTired 8y ago

> A bit more precisely, in physics a tensor is an object that transforms a particular way during coordinate transformations No offence, but that's a hideous definition :) For me a (real) tensor is a function that takes an ordered set of N row vectors and M column vectors as arguments, and spits back a real number as a result. It has to be linear in its arguments. That's all folks! By this token a matrix A is a tensor: it takes one row vector x, and one column vector y, and returns a real number xAy. Similarly, a row vector x is a tensor: feed it a column vector y and you get the real number xy. You can dress all this up in the language of linear functionals or n-forms, but at core that's what's going on.

antognini 8y ago
Yes, that definition is fine for machine learning, but it's not quite complete for physics. To extend your definition for physics, a tensor is a function that takes an ordered set of N row vectors and M column vectors as arguments and spits back a real, coordinate-invariant number as a result.
- soVeryTired 8y ago
  I think you get coordinate invariance for free if you think of a vector as an object in its own right, rather than as a tuple in a coordinate system. But then I guess it's more accurate to speak of vectors and covectors than row vectors and column vectors.