I think Tensor in machine learning is more akin to array, just an n-dimensional collection of numbers. It faces the same confusion with a real tensor, as "2-d

I think Tensor in machine learning is more akin to array, just an n-dimensional collection of numbers. It faces the same confusion with a real tensor, as "2-d array" does with "matrix". While a matrix can be represented as a 2-d array, and a tensor can be represented as a n-d array, they have different mathematical connotations. A matrix should be viewed as a 2-d array representing a linear map from on vector space (say R^n) to another (say R^m). We can say this because it is a theorem that any linear map from one (finite dimensional) vector space to another can be represented as a matrix and computed via matrix multiplication. Tensors are the same thing, except the input/output is no longer limited to vector spaces. Consider you have a vector and a bunch (let's say q) of matrices, and you take the matrix product of the vector with all those matrices. You will get q vectors, which you can stick together to form a matrix. This act of multiplying a vector by a bunch of matrices is clearly linear with respect to the input: If we multiply the input by X, every vector will be multiplied by X, so the resulting matrix will be multiplied by X. Suppose you do this operation, T on v to get vectors T1(v), T2(v) ... Tq(v) and on w to get T1(w), T2(w), ... Tq(w). Since all T's are matrix products (linear) then if we do the operation on v + w we will get T1(v) + T1(w), T2(v) + T2(w) ... Tq(v) + Tq(w). Which is essentially T(v) + T(w). So now we know T is linear with respect to the input. Now, this all took a long time to describe, so let's simplify it: How about instead of a group of matrices, we just call this thing a 3-d tensor? we can let i and j index the regular matrix dimensions and make up a new dimension for the matrix we're on, call it k. Now at any coordinate we get a value so it's basically a 3-d array, but it represents something much more specific than that. You can guess how this might generalize to mapping matrices x matrices to 3d tensors or 3d matrices x 3d tensors to 4d tensors and so on. So now the question is, does TensorFlow conflate these? I think it does - somewhat. A convolution can be viewed as a tensor (a single filter maps matrices (images) x 3d-tensor (kernel) to matrices (another image)) so I'd call that a Tensor operation. But consider the input image itself. Is this truly a tensor? If we consider a simple situation, say we have some data vector and we're doing a matrix multiply to get the output of a linear model. Is the input a matrix? I would say no, because we don't think of it as acting on the model, we thing of the model as acting on it, even though what we are doing is really equivalent to multiplying two matrices. Equivalently, I would not call the input image, or any activation in a neural network a true tensor, even though it is numerically equivalent. There are true tensors in TensorFlow, but if you're using high level functions (dense, conv2d) they are usually hidden from the user.