17 ms·

Tensors, the geometric tool that solved Einstein's relativity problem

86 points by Luc 2y ago

bollu 2y ago
I've written about [this explanation of tensors](https://pixel-druid.com/articles/tensor-is-a-thing-that-transforms-like-a-tensor.html https://pixel-druid.com/articles/tensor-is-a-thing-that-tran...) before, and it seems worthwhile to write it down again: There are two ways of using linear maps in the context of physics. One is as a thing that acts on the space . The other is a thing that acts on the coordinates . So when we talk about transformations in tensor analysis, we're talking about coordinate transformatios , not space transformations . Suppose I implement a double ended queue using two pointers: ``` struct Queue {int memory, start, end; } void queue_init(int size) { memory = malloc(sizeof(int) size); start = end = memory + (size - 1) / 2; } void queue_push_start(int x) { start = x; start--; } void queue_push_end(int x) { end++; end = x; } int queue_head() { return start; } int queue_tail() { return end; } void queue_deque_head() { start++; } void queue_deque_tail() { tail--; } ``` See that the state of the queue is technically three numbers, { memory, start, end } (Pointers are just numbers after all). But this is coordinate dependent , as start and end are relative to the location of memory. Now suppose I have a procedure to reallocate the queue size: ``` void queue_realloc(Queue q, int new_size) { int start_offset = q->memory - q->start; int end_offset = q->memory - q->end; int oldmem = q->memory; q->memory = realloc(q->memory, new_size); memcpy(q->memory, oldmem + q->start, sizeof(int) * (end_offset - start_offset); q->start = q->memory + start_offset; q->end = q->memory - end_offset; } ``` Notice that when I do this, the values of start and end can be completely different! However, see that the length of the queue, given by (end - start) is invariant : It hasn't changed! --- In the exact same way, a "tensor" is a collection of numbers that describes something physical with respect to a particular coordinate system (the pointers start and end with respect to the memory coordinate system). "tensor calculus" is a bunch of rules that tell you how the numbers change when one changes coordinate systems (ie, how the pointers start and end change when the pointer memory changes). Some quantities that are computed from tensors are "physical", like the length of the queue, as they are invariant under transformations. Tensor calculus gives a principled way to make sure that the final answers we calculate are "invariant" / "physical" / "real". The actual locations of start and end don't matter, as (end - start) will always be the length of the list! --- Physicists (and people who write memory allocators) need such elaborate tracking, to keep track of what is "real" and what is "coordinate dependent", since a lot of physics involves crazy coordinate systems , and having ways to know what things are real and what are artefacts of one's coordinate system is invaluable. For a real example, consider the case of singularities of the Schwarzschild solution to GR, where we initially thought there were two singularities, but it later turned out there was only one "real" singularity, and the other singularity was due to a poor choice of coordinate system: Although there was general consensus that the singularity at r = 0 was a 'genuine' physical singularity, the nature of the singularity at r = rs remained unclear. In 1921 Paul Painlevé and in 1922 Allvar Gullstrand independently produced a metric, a spherically symmetric solution of Einstein's equations, which we now know is coordinate transformation of the Schwarzschild metric, Gullstrand–Painlevé coordinates, in which there was no singularity at r = rs. They, however, did not recognize that their solutions were just coordinate transform
senderista 2y ago
If you have any linear algebra background, then the definition of a tensor is straightforward: given a vector space V over a field K (in physics, K = R or C), a tensor T is a multilinear (i.e. linear in each argument) function from vectors and dual vectors in V to numbers in K. That's it! A type (p, q) tensor T takes p vectors and q dual vectors as arguments (p+q is often called the rank of T but is ambiguous compared to the type). (If you're unfamiliar with the definition of dual vector, it's even simpler: it's just a linear function from V to K.)
- DemocracyFTW2 2y ago
  A monad is just a monoid in the category of endofunctors, what’s the problem?
- will-burner 2y ago
  The definition may be simple, but it's not very concrete and I'd argue that makes it not strait forward. While examples of vector spaces can be very concrete (think R, R^2, R^30), I struggle to think of a concrete example of a multilinear function from vectors and dual vectors in V to numbers in K. On top of that when working with tensors, you don't usually use the definition os a multilinear function at least as far as I remember.
  - tel 2y ago
    Not really to push back as I do agree that this is a bit trickier to get an intuition for than the OP suggests, but the most trivial concrete example of a (1, 1) tensor would just be the evaluation function (v, f) |-> f(v), which, given a metric, corresponds to the inner product.
  - cshimmin 2y ago
    A simple example of a multilinear function is the inner (a.k.a dot) product <a, b>: it takes a vector (b), and a dual vector (a^T), and returns a number. In tensor notation it's typically written δ_ij. It's multilinear because it's linear in each of its arguments separately: <ca, b> = c<a,b> and <a, cb> = c<a,b>. Another simple but less obvious example is a rotation (orthogonal) matrix. It takes a vector as an input, and returns a vector. But a vector itself can be thought of as a linear function that takes a dual vector and returns a number (via the inner product, above!). So, applying the rotation matrix to a vector is a sort of "currying" on the multilinear map, while the matrix alone can be considered a function that takes a vector and a dual vector, and returns a number. In functional notation, you can consider your rotation matrix to be a function (V x V*) -> K, which can in turn be considered a function V -> (V* -> K), where V* is the dual space of V.
max_likelihood 2y ago
I've always thought the use of "Tensor" in the "TensorFlow" library is a misnomer. I'm not too familiar with ML/theory, is there a deeper geometric meaning to the multi-dimensional array of numbers we are multiplying or is "MatrixFlow" a more appropriate name?
- itishappy 2y ago
  The tensors in tensorflow are often higher dimensional. Is a 3d block of numbers (say 1920x1080x3) still a matrix? I would argue it's not. Are there transformation rules for matrices? You're totally correct that the tensors in tensorflow do drop the geometric meaning, but there's precedence there from how CS vs math folk use vectors.
- andrewla 2y ago
  Matrices are strictly two-dimensional arrays (together with some other properties, but for a computer scientist that's it). Tensors are the generalization to higher dimensional arrays.
- MathMonkeyMan 2y ago
  The joke I learned in a Physics course is "a vector is something that transforms like a vector," and "a tensor is something that transforms like a tensor." It's true, though. The physicist's tensor is a matrix of functions of coordinates that transform in a prescribed way when the coordinates are transformed. It's a particular application of the chain rule from calculus. I don't know why the word "tensor" is used in other contexts. Google says that the etymology of the word is: > early 18th century: modern Latin, from Latin tendere ‘to stretch’. So maybe the different senses of the word share the analogy of scaling matrices.
  - ogogmad 2y ago
    The mathematical definition is 99% equivalent to the physical one. I find that the physical one helps to motivate the mathematical one by illustrating the numerical difference between the basis-change transformation for (1,0)- and (0,1)-tensors. The mathematical one is then simpler and more conceptual once you've understood that motivation. The concept of a tensor really belongs to linear algebra, but occurs mostly in differential geometry. There is still a "1% difference" in meaning though. This difference allows a physicist to say "the Christoffel symbols are not a tensor", while a mathematician would say this is a conflation of terms. TensorFlow's terminology is based on the rule of thumb that a "vector" is really a 1D array (think column vector), a "matrix" is really a 2D array, and a "tensor" is then an nD array. That's it. This is offensive to physicists especially, but ¯\_(ツ)_/¯
openrisk 2y ago
> Talk to a computer scientist, and they might tell you that a tensor is an array of numbers that stores important data The conflicting definitions of tensors have precedent in lower dimensions: vectors were already being used in computer science to mean something different than in mathematics / physics, long before the current tensormania. Its not clear if that ambiguity will ever be a practical problem though. For as long as such structures are containers of numerical data with no implied transformation properties we are really talking about two different universes. Things might get interesting though in the overlap between information technology and geometry [1] :-) [1] https://en.wikipedia.org/wiki/Information_geometry https://en.wikipedia.org/wiki/Information_geometry
nyrikki 2y ago
I would argue that today, geometric algebra/Clifford calculus and space time algebra are more intuitive and useful. Gibbs/Heavysides vectors were more popular at the time. At least for me.
- eigenket 2y ago
  Nah, at least in the context of General Relativity tensors and tensor fields are everything. Geometric algebra doesn't add much.
- Koshkin 2y ago
  Ah, been waiting for this. Now the discussion is complete. (At least for me.)
ijidak 2y ago
Here is a video series on tensors I've enjoyed: https://youtube.com/playlist?list=PLJHszsWbB6hrkmmq57lX8BV-o-YIOFsiG&feature=shared https://youtube.com/playlist?list=PLJHszsWbB6hrkmmq57lX8BV-o... And this series by Dialect: https://youtube.com/playlist?list=PL__fY7tXwodmfntSAAyBDxZ4_eE3ZwbFE&feature=shared https://youtube.com/playlist?list=PL__fY7tXwodmfntSAAyBDxZ4_...
- eigenheart 2y ago
  https://news.ycombinator.com/item?id=17817655 https://news.ycombinator.com/item?id=17817655 Hackernews user saivan started notes on eigenchris's tensor series videos.
mvaliente2001 2y ago
The idea of tensors as "a matrix of numbers" or the example of a cube with vectors on every face never clicked for me. It was this (NASA paper)[https://www.grc.nasa.gov/www/k-12/Numbers/Math/documents/Tensors_TM2002211716.pd https://www.grc.nasa.gov/www/k-12/Numbers/Math/documents/Ten...] what finally brought me clarity. The main idea, as others already commented, is that a tensor or rank n is a function that can be applied up to n vector, reducing its rank by one for each vector it consumes.
- cryptonector 2y ago
  > a tensor or rank n is a function that can be applied up to n vector There seems to be a grammar problem here.
- qsdf38100 2y ago
  In your cube example you are using the word "vector" to refer to faces of the cube. Did you mean matrix? My understanding is that the cube is a rank 3 tensor, the faces (or rather slices) of the cube are rank 2 tensors (aka matrices), and the edges (slices) of the matrices are rank 1 tensors (aka vectors).
- Koshkin 2y ago
  But, in practice, how often do we ask a tensor to consume a vector?
wrycoder 2y ago
This is a surprisingly low quality article for Quanta. The discussion here is far beyond it.
Koshkin 2y ago
One better resource to learn about tensors: https://grinfeld.org/books/An-Introduction-To-Tensor-Calculus/ https://grinfeld.org/books/An-Introduction-To-Tensor-Calculu...

86 points by Luc 2y ago

I've written about [this explanation of tensors](https://pixel-druid.com/articles/tensor-is-a-thing-that-transforms-like-a-tensor.html https://pixel-druid.com/articles/tensor-is-a-thing-that-tran...) before, and it seems worthwhile to write it down again: There are two ways of using linear maps in the context of physics. One is as a thing that acts on the space . The other is a thing that acts on the coordinates . So when we talk about transformations in tensor analysis, we're talking about coordinate transformatios , not space transformations . Suppose I implement a double ended queue using two pointers: ``` struct Queue {int memory, start, end; } void queue_init(int size) { memory = malloc(sizeof(int) size); start = end = memory + (size - 1) / 2; } void queue_push_start(int x) { start = x; start--; } void queue_push_end(int x) { end++; end = x; } int queue_head() { return start; } int queue_tail() { return end; } void queue_deque_head() { start++; } void queue_deque_tail() { tail--; } ``` See that the state of the queue is technically three numbers, { memory, start, end } (Pointers are just numbers after all). But this is coordinate dependent , as start and end are relative to the location of memory. Now suppose I have a procedure to reallocate the queue size: ``` void queue_realloc(Queue q, int new_size) { int start_offset = q->memory - q->start; int end_offset = q->memory - q->end; int oldmem = q->memory; q->memory = realloc(q->memory, new_size); memcpy(q->memory, oldmem + q->start, sizeof(int) * (end_offset - start_offset); q->start = q->memory + start_offset; q->end = q->memory - end_offset; } ``` Notice that when I do this, the values of start and end can be completely different! However, see that the length of the queue, given by (end - start) is invariant : It hasn't changed! --- In the exact same way, a "tensor" is a collection of numbers that describes something physical with respect to a particular coordinate system (the pointers start and end with respect to the memory coordinate system). "tensor calculus" is a bunch of rules that tell you how the numbers change when one changes coordinate systems (ie, how the pointers start and end change when the pointer memory changes). Some quantities that are computed from tensors are "physical", like the length of the queue, as they are invariant under transformations. Tensor calculus gives a principled way to make sure that the final answers we calculate are "invariant" / "physical" / "real". The actual locations of start and end don't matter, as (end - start) will always be the length of the list! --- Physicists (and people who write memory allocators) need such elaborate tracking, to keep track of what is "real" and what is "coordinate dependent", since a lot of physics involves crazy coordinate systems , and having ways to know what things are real and what are artefacts of one's coordinate system is invaluable. For a real example, consider the case of singularities of the Schwarzschild solution to GR, where we initially thought there were two singularities, but it later turned out there was only one "real" singularity, and the other singularity was due to a poor choice of coordinate system: Although there was general consensus that the singularity at r = 0 was a 'genuine' physical singularity, the nature of the singularity at r = rs remained unclear. In 1921 Paul Painlevé and in 1922 Allvar Gullstrand independently produced a metric, a spherically symmetric solution of Einstein's equations, which we now know is coordinate transformation of the Schwarzschild metric, Gullstrand–Painlevé coordinates, in which there was no singularity at r = rs. They, however, did not recognize that their solutions were just coordinate transform

senderista 2y ago

If you have any linear algebra background, then the definition of a tensor is straightforward: given a vector space V over a field K (in physics, K = R or C), a tensor T is a multilinear (i.e. linear in each argument) function from vectors and dual vectors in V to numbers in K. That's it! A type (p, q) tensor T takes p vectors and q dual vectors as arguments (p+q is often called the rank of T but is ambiguous compared to the type). (If you're unfamiliar with the definition of dual vector, it's even simpler: it's just a linear function from V to K.)

DemocracyFTW2 2y ago
A monad is just a monoid in the category of endofunctors, what’s the problem?
will-burner 2y ago
The definition may be simple, but it's not very concrete and I'd argue that makes it not strait forward. While examples of vector spaces can be very concrete (think R, R^2, R^30), I struggle to think of a concrete example of a multilinear function from vectors and dual vectors in V to numbers in K. On top of that when working with tensors, you don't usually use the definition os a multilinear function at least as far as I remember.
- tel 2y ago
  Not really to push back as I do agree that this is a bit trickier to get an intuition for than the OP suggests, but the most trivial concrete example of a (1, 1) tensor would just be the evaluation function (v, f) |-> f(v), which, given a metric, corresponds to the inner product.
- cshimmin 2y ago
  A simple example of a multilinear function is the inner (a.k.a dot) product <a, b>: it takes a vector (b), and a dual vector (a^T), and returns a number. In tensor notation it's typically written δ_ij. It's multilinear because it's linear in each of its arguments separately: <ca, b> = c<a,b> and <a, cb> = c<a,b>. Another simple but less obvious example is a rotation (orthogonal) matrix. It takes a vector as an input, and returns a vector. But a vector itself can be thought of as a linear function that takes a dual vector and returns a number (via the inner product, above!). So, applying the rotation matrix to a vector is a sort of "currying" on the multilinear map, while the matrix alone can be considered a function that takes a vector and a dual vector, and returns a number. In functional notation, you can consider your rotation matrix to be a function (V x V*) -> K, which can in turn be considered a function V -> (V* -> K), where V* is the dual space of V.

max_likelihood 2y ago

I've always thought the use of "Tensor" in the "TensorFlow" library is a misnomer. I'm not too familiar with ML/theory, is there a deeper geometric meaning to the multi-dimensional array of numbers we are multiplying or is "MatrixFlow" a more appropriate name?

itishappy 2y ago
The tensors in tensorflow are often higher dimensional. Is a 3d block of numbers (say 1920x1080x3) still a matrix? I would argue it's not. Are there transformation rules for matrices? You're totally correct that the tensors in tensorflow do drop the geometric meaning, but there's precedence there from how CS vs math folk use vectors.
andrewla 2y ago
Matrices are strictly two-dimensional arrays (together with some other properties, but for a computer scientist that's it). Tensors are the generalization to higher dimensional arrays.
MathMonkeyMan 2y ago
The joke I learned in a Physics course is "a vector is something that transforms like a vector," and "a tensor is something that transforms like a tensor." It's true, though. The physicist's tensor is a matrix of functions of coordinates that transform in a prescribed way when the coordinates are transformed. It's a particular application of the chain rule from calculus. I don't know why the word "tensor" is used in other contexts. Google says that the etymology of the word is: > early 18th century: modern Latin, from Latin tendere ‘to stretch’. So maybe the different senses of the word share the analogy of scaling matrices.
- ogogmad 2y ago
  The mathematical definition is 99% equivalent to the physical one. I find that the physical one helps to motivate the mathematical one by illustrating the numerical difference between the basis-change transformation for (1,0)- and (0,1)-tensors. The mathematical one is then simpler and more conceptual once you've understood that motivation. The concept of a tensor really belongs to linear algebra, but occurs mostly in differential geometry. There is still a "1% difference" in meaning though. This difference allows a physicist to say "the Christoffel symbols are not a tensor", while a mathematician would say this is a conflation of terms. TensorFlow's terminology is based on the rule of thumb that a "vector" is really a 1D array (think column vector), a "matrix" is really a 2D array, and a "tensor" is then an nD array. That's it. This is offensive to physicists especially, but ¯\_(ツ)_/¯

openrisk 2y ago

> Talk to a computer scientist, and they might tell you that a tensor is an array of numbers that stores important data The conflicting definitions of tensors have precedent in lower dimensions: vectors were already being used in computer science to mean something different than in mathematics / physics, long before the current tensormania. Its not clear if that ambiguity will ever be a practical problem though. For as long as such structures are containers of numerical data with no implied transformation properties we are really talking about two different universes. Things might get interesting though in the overlap between information technology and geometry [1] :-) [1] https://en.wikipedia.org/wiki/Information_geometry https://en.wikipedia.org/wiki/Information_geometry

nyrikki 2y ago

I would argue that today, geometric algebra/Clifford calculus and space time algebra are more intuitive and useful. Gibbs/Heavysides vectors were more popular at the time. At least for me.

eigenket 2y ago
Nah, at least in the context of General Relativity tensors and tensor fields are everything. Geometric algebra doesn't add much.
Koshkin 2y ago
Ah, been waiting for this. Now the discussion is complete. (At least for me.)

ijidak 2y ago

Here is a video series on tensors I've enjoyed: https://youtube.com/playlist?list=PLJHszsWbB6hrkmmq57lX8BV-o-YIOFsiG&feature=shared https://youtube.com/playlist?list=PLJHszsWbB6hrkmmq57lX8BV-o... And this series by Dialect: https://youtube.com/playlist?list=PL__fY7tXwodmfntSAAyBDxZ4_eE3ZwbFE&feature=shared https://youtube.com/playlist?list=PL__fY7tXwodmfntSAAyBDxZ4_...

eigenheart 2y ago
https://news.ycombinator.com/item?id=17817655 https://news.ycombinator.com/item?id=17817655 Hackernews user saivan started notes on eigenchris's tensor series videos.

mvaliente2001 2y ago

The idea of tensors as "a matrix of numbers" or the example of a cube with vectors on every face never clicked for me. It was this (NASA paper)[https://www.grc.nasa.gov/www/k-12/Numbers/Math/documents/Tensors_TM2002211716.pd https://www.grc.nasa.gov/www/k-12/Numbers/Math/documents/Ten...] what finally brought me clarity. The main idea, as others already commented, is that a tensor or rank n is a function that can be applied up to n vector, reducing its rank by one for each vector it consumes.

cryptonector 2y ago
> a tensor or rank n is a function that can be applied up to n vector There seems to be a grammar problem here.
qsdf38100 2y ago
In your cube example you are using the word "vector" to refer to faces of the cube. Did you mean matrix? My understanding is that the cube is a rank 3 tensor, the faces (or rather slices) of the cube are rank 2 tensors (aka matrices), and the edges (slices) of the matrices are rank 1 tensors (aka vectors).
Koshkin 2y ago
But, in practice, how often do we ask a tensor to consume a vector?

wrycoder 2y ago

This is a surprisingly low quality article for Quanta. The discussion here is far beyond it.

Koshkin 2y ago

One better resource to learn about tensors: https://grinfeld.org/books/An-Introduction-To-Tensor-Calculus/ https://grinfeld.org/books/An-Introduction-To-Tensor-Calculu...