5 ms·

I think one of the issues that makes tensors difficult to understand is that different disciplines use the word in different ways. I'm going to generalize here

I think one of the issues that makes tensors difficult to understand is that different disciplines use the word in different ways. I'm going to generalize here a bit about how each group uses the term, but I suspect not everyone will agree with me. Apologies in advance! For machine-learning types, a tensor is a multidimensaional array of numbers that admits certain operations (addition, subtraction, hadamard product, etc) [0] For mathematicians, a tensor is a function F. We pass the function an ordered tuple of N vectors and M covectors (or, to keep things simple, N column vectors and M row vectors), and the function returns a scalar. The function F needs to be linear in each of the N vectors and M covectors. In this view, a matrix is a tensor with N=1, M=1. The operations used by machine learning types arise naturally once you crank the mathematical handle a little bit. For physicists, a tensor is what mathematicians would refer to as a tensor field. So they take a space X (think R^N), and for each point in the space they associate a mathematician's tensor F as defined above. The properties of the function F are permitted to change from point to point. The Riemannian metric tensor is a classic example of 'tensor' according to this usage. [0] https://machinelearningmastery.com/introduction-to-tensors-for-machine-learning/ https://machinelearningmastery.com/introduction-to-tensors-f...

ktpsns 5y ago
This is so true! For machine-learning and data science, tensors are largely what is called n-dimensional array. Numpy is a popular python library built basically around this data type. Mathematical tensors in a particular base (sorry for the sloppy language, theoretical physicist here) can be displayed with such a data type, depsite the n-dimensional array data type (as available in numpy) lacks the (co)vector properties. Funny enough, in theoretical physics there are two groups of people: The one prefering to write out complicated objects (i.e. tensors) with indices and the ones who do not (typically called abstract tensor notation in general relativity). The latter are much more close to mathematical physics and prefer to understand tensors as mappings from (co)vector spaces to scalars, similar to as you already wrote. Most general relativity computer codes are written by the index people, not by the mathematical physicists ;-)
JadeNB 5y ago
> For mathematicians, a tensor is a function F. We pass the function an ordered tuple of N vectors and M covectors (or, to keep things simple, N column vectors and M row vectors), and the function returns a scalar. The function F needs to be linear in each of the N vectors and M covectors. In this view, a matrix is a tensor with N=1, M=1. The operations used by machine learning types arise naturally once you crank the mathematical handle a little bit. As with almost all non-technical statements one can make about math (including my amendment—recursive nerd-sniping away), this is neither quite true nor quite false. A tensor is an element of a tensor product—that's it. (But what tensor product? There are a lot of notions. I'll use the bare tensor product of non-topologised vector spaces.) One kind of thing you can tensor is copies of a single vector space, and/or its dual. This is the sort of tensor product you likely have in mind. Tensoring copies of the dual allows you to feed in elements of the original vector space. Tensoring copies of the vector space allows you to feed in elements of the dual vector space (there is mathematically no intrinsic difference between a vector and a co-vector; having a favourite vector space in mind, you can talk about the elements of that vector space or of its dual, and—which is where the vector vs. co-vector terminology in physics comes from—about how coefficient vectors change when you change the basis of V), although, importantly, if your original vector space is infinite dimensional then the natural embedding into its double dual is not an isomorphism. Which is to say, yes, the objects you describe all arise as mathematical tensors; but mathematical tensors can also describe much more.
- soVeryTired 5y ago
  > if your original vector space is infinite dimensional then the natural embedding into its double dual is not an isomorphism Ok, so this feels like the crux of the matter. So how does the fact that V** is not isomorphic to V make the tensor product construction a more general concept than the linear function construction?
  - JadeNB 5y ago
    > Ok, so this feels like the crux of the matter. So how does the fact that V* is not isomorphic to V make the tensor product construction a more general concept than the linear function construction? This doesn't really have anything to do with tensor products per se; it can already be seen with tensor products involving only a single factor, which are just vector spaces. Thinking of tensors only as functions means that, for example, one can never think of the original vector space V itself, only of its image in the double dual V^{**} (a fancy way of saying that you can evaluate a vector v \in V on an element v^* of the dual vector space V^* by evaluating v^* at v: in confusing but suggestive notation, v(v^*) = v^*(v)). It's certainly true that you can do this, and, given the axiom of choice, you don't lose any information; you know everything about a vector v \in V by knowing its value on elements of V^* (which is to say, by knowing the values of elements of V^* on V). However, if V is infinite dimensional, then you are forcing yourself to carry around extra, possibly unwanted information: if you are taking bare algebraic duals, not topological duals, then V^{**} is inconceivably larger than V, which is to say that there are way more linear functionals on V^* than just those coming from evaluation at a fixed element of V. You can fix some of this inconceivable largeness by knowing a little more structure carried by V, and by forcing your dual to reflect that structure—usually you know the topology, and ask that the dual consist of continuous functionals; and once there's topology, you start asking things of the tensor product, too. (For example, you probably don't want to take the vector-space tensor product of Hilbert spaces, but rather its completion in some suitable sense.) But, even with a more refined notion of duality, it's still only the nice spaces V that are identified with their double duals via the canonical map V \to V^{**}; the terminology is 'reflexive'. (I think I caught all the asterisks that the Markdown parser ate the first time through.)
joppy 5y ago
I think that tensors have a much more broad meaning to mathematicians. At least in pure mathematics and algebra, there is much more of a focus on the tensor product operation, a way of taking two (or more) vector spaces and producing a new vector space: their tensor product. Tensors are elements of this new space. In particular they don’t need to be multilinear functions on a product of vector spaces (for finite dimensional spaces there is not much difference, but for infinite-dimensional spaces there is a real difference between these two things). Tensor products in this generality don’t need to have an (n, m) rank in the sense you’re describing, since they might be put together from completely different spaces. For example it’s perfectly fine to form the tensor product of a 2-dimensional space with a 5-dimensional space, yielding a 10-dimensional space.
- soVeryTired 5y ago
  That's a fair point about the tensor product being more central in algebra. But I think the 'friendliest' introduction to tensors is the multilinear function viewpoint. IMO it motivates the tensor product. I'm not familiar with the infinite-dimensional case. Do you have an example of a tensor that isn't a multilinear function on a product of vector spaces? I'd be interested to refine my understanding here.
  - joppy 5y ago
    The easiest way to see it is cardinality: the space of linear functions on a countably-infinite dimensional vector space is uncountably-infinite dimensional. This is the reason why you can’t necessarily swap out vectors with linear functions on an infinite-dimensional vector space (if you restrict to functions with finite support or something, it’s ok). A familiar example of a tensor product of countably infinite dimensional vector spaces would be polynomials in multiple variables. Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables. In the above it’s important to note that a polynomial has finitely many terms, so a power series like 1 + x + x^2 + … is not a polynomial. The space of power series is isomorphic to the space of linear functions on F[x], and is uncountably-infinite dimensional.

ktpsns 5y ago

This is so true! For machine-learning and data science, tensors are largely what is called n-dimensional array. Numpy is a popular python library built basically around this data type. Mathematical tensors in a particular base (sorry for the sloppy language, theoretical physicist here) can be displayed with such a data type, depsite the n-dimensional array data type (as available in numpy) lacks the (co)vector properties. Funny enough, in theoretical physics there are two groups of people: The one prefering to write out complicated objects (i.e. tensors) with indices and the ones who do not (typically called abstract tensor notation in general relativity). The latter are much more close to mathematical physics and prefer to understand tensors as mappings from (co)vector spaces to scalars, similar to as you already wrote. Most general relativity computer codes are written by the index people, not by the mathematical physicists ;-)

JadeNB 5y ago

> For mathematicians, a tensor is a function F. We pass the function an ordered tuple of N vectors and M covectors (or, to keep things simple, N column vectors and M row vectors), and the function returns a scalar. The function F needs to be linear in each of the N vectors and M covectors. In this view, a matrix is a tensor with N=1, M=1. The operations used by machine learning types arise naturally once you crank the mathematical handle a little bit. As with almost all non-technical statements one can make about math (including my amendment—recursive nerd-sniping away), this is neither quite true nor quite false. A tensor is an element of a tensor product—that's it. (But what tensor product? There are a lot of notions. I'll use the bare tensor product of non-topologised vector spaces.) One kind of thing you can tensor is copies of a single vector space, and/or its dual. This is the sort of tensor product you likely have in mind. Tensoring copies of the dual allows you to feed in elements of the original vector space. Tensoring copies of the vector space allows you to feed in elements of the dual vector space (there is mathematically no intrinsic difference between a vector and a co-vector; having a favourite vector space in mind, you can talk about the elements of that vector space or of its dual, and—which is where the vector vs. co-vector terminology in physics comes from—about how coefficient vectors change when you change the basis of V), although, importantly, if your original vector space is infinite dimensional then the natural embedding into its double dual is not an isomorphism. Which is to say, yes, the objects you describe all arise as mathematical tensors; but mathematical tensors can also describe much more.

soVeryTired 5y ago
> if your original vector space is infinite dimensional then the natural embedding into its double dual is not an isomorphism Ok, so this feels like the crux of the matter. So how does the fact that V** is not isomorphic to V make the tensor product construction a more general concept than the linear function construction?
- JadeNB 5y ago
  > Ok, so this feels like the crux of the matter. So how does the fact that V* is not isomorphic to V make the tensor product construction a more general concept than the linear function construction? This doesn't really have anything to do with tensor products per se; it can already be seen with tensor products involving only a single factor, which are just vector spaces. Thinking of tensors only as functions means that, for example, one can never think of the original vector space V itself, only of its image in the double dual V^{**} (a fancy way of saying that you can evaluate a vector v \in V on an element v^* of the dual vector space V^* by evaluating v^* at v: in confusing but suggestive notation, v(v^*) = v^*(v)). It's certainly true that you can do this, and, given the axiom of choice, you don't lose any information; you know everything about a vector v \in V by knowing its value on elements of V^* (which is to say, by knowing the values of elements of V^* on V). However, if V is infinite dimensional, then you are forcing yourself to carry around extra, possibly unwanted information: if you are taking bare algebraic duals, not topological duals, then V^{**} is inconceivably larger than V, which is to say that there are way more linear functionals on V^* than just those coming from evaluation at a fixed element of V. You can fix some of this inconceivable largeness by knowing a little more structure carried by V, and by forcing your dual to reflect that structure—usually you know the topology, and ask that the dual consist of continuous functionals; and once there's topology, you start asking things of the tensor product, too. (For example, you probably don't want to take the vector-space tensor product of Hilbert spaces, but rather its completion in some suitable sense.) But, even with a more refined notion of duality, it's still only the nice spaces V that are identified with their double duals via the canonical map V \to V^{**}; the terminology is 'reflexive'. (I think I caught all the asterisks that the Markdown parser ate the first time through.)

joppy 5y ago

I think that tensors have a much more broad meaning to mathematicians. At least in pure mathematics and algebra, there is much more of a focus on the tensor product operation, a way of taking two (or more) vector spaces and producing a new vector space: their tensor product. Tensors are elements of this new space. In particular they don’t need to be multilinear functions on a product of vector spaces (for finite dimensional spaces there is not much difference, but for infinite-dimensional spaces there is a real difference between these two things). Tensor products in this generality don’t need to have an (n, m) rank in the sense you’re describing, since they might be put together from completely different spaces. For example it’s perfectly fine to form the tensor product of a 2-dimensional space with a 5-dimensional space, yielding a 10-dimensional space.

soVeryTired 5y ago
That's a fair point about the tensor product being more central in algebra. But I think the 'friendliest' introduction to tensors is the multilinear function viewpoint. IMO it motivates the tensor product. I'm not familiar with the infinite-dimensional case. Do you have an example of a tensor that isn't a multilinear function on a product of vector spaces? I'd be interested to refine my understanding here.
- joppy 5y ago
  The easiest way to see it is cardinality: the space of linear functions on a countably-infinite dimensional vector space is uncountably-infinite dimensional. This is the reason why you can’t necessarily swap out vectors with linear functions on an infinite-dimensional vector space (if you restrict to functions with finite support or something, it’s ok). A familiar example of a tensor product of countably infinite dimensional vector spaces would be polynomials in multiple variables. Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables. In the above it’s important to note that a polynomial has finitely many terms, so a power series like 1 + x + x^2 + … is not a polynomial. The space of power series is isomorphic to the space of linear functions on F[x], and is uncountably-infinite dimensional.