4 ms·

I've been waiting for this for 6th months. Thanks to Sheldon Axler for making it available for free. This is intended to be a second book on Linear Algebra. Fo

I've been waiting for this for 6th months. Thanks to Sheldon Axler for making it available for free. This is intended to be a second book on Linear Algebra. For a first book I suggest "Linear Algebra: Theory, Intuition, Code" by Mike X Cohen. It's a bit different than a typical math textbook, it has more focus on conversational explanations using words, although the book does have plenty of proofs as well. The book also has a lot of code examples, which I didn't do, but I did appreciate the discussions related to computing; for example, the book explains that several calculations that can be done by hand are numerically unstable when done on computers (those darn floats are tricky). For the HN crowd, this is the right focus, math for the sake of computing, rather than math for the sake of math. One insight I gained from the book was the 4 different perspectives of matrix multiplication. I had never encountered this, not even in the oft-suggested "Essence of Linear Algebra" YouTube series. Everything I had seen explained only one of the 4 views, and then I'd encounter a calculation that was better understood by another view and would be confused. It still bends my mind to think all these different perspectives describe the same calculation, they're just different ways of interpreting it. At the risk of spamming a bit, I'll put my notes here, because this is something I've never seen written down elsewhere. The book has more explanation, these are just my condensed notes. 4 perspectives on matrix multiplication ======================================= 1 Element perspective (all possible dot / inner products) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (row count × row length) × (column length × column count) In AB, every element is the dot product of the corresponding row of A and column of B. The rows in A are the same length as the columns in B and thus have dot products. 2 Layer perspective (sum of outer product layers) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (column length × column count) × (row count × row length) AB is the sum of every outer product between the corresponding columns in A and rows in B. The column count in A is the same as the row count in B, thus the columns and rows pair up exactly for the outer product operation. The outer product does not require vectors to be the same length. 3 Column perspective (weighted sums / linear combinations) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (column length × column count) × (column length × column count) In AB, every column is a weighted sum of the columns in A; the weights come from the columns in B. The weight count in the columns of B must match the column count in A. 4 Row perspective (weighted sums / linear combinations) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (row count × row length) × (row count × row length) In AB, every row is a weighted sum of the rows in B; the weights come from the rows in A. The weight count in the rows of A must match the row count in B.

ndriscoll 3y ago
The most important interpretation IMO is that a matrix is a specification for a linear map. A linear map is determined by what it does to a basis, and the columns of a matrix are just the list of outputs for each basis element (e.g. the first column is `f(b_1)`. The nth column is `f(b_n)`). If A is the matrix for f and B the matrix for g (for some chosen bases), then BA is the matrix for the composition x -> g(f(x)). i.e. the nth column is `g(f(b_n))`. The codomain of f has to match the domain of g for composition to make sense, which means dimensions have to match (i.e. row count of A must be column count of B).
- Buttons840 3y ago
  It's debatable what the "most important" perspective is. For example, if I need a bunch of dot products between two sets of vectors, that doesn't seem like a linear map or a change of basis (not to me at least), and yet it's exactly what matrix multiplication is, just calculating a bunch of dot products between two sets of vectors. Or when I think about the singular value decomposition, I'm not thinking about linear maps and change of basis, but I am thinking about a sum of many outer product layers.
  - ndriscoll 3y ago
    If you don't have a linear map in mind, why do you write your dot products with one set of column vectors and another set of row vectors? Computationally, the best way to do dot products would be to walk all of your arrays in contiguous memory order, so the row/column thing is an unnecessary complication. And if you have more than 2 matrices to multiply/"steps of dot products to do in a pipeline", there's almost certainly a relevant interpretation as linear maps lurking. Outer products are one way to define a "simple" linear map. What SVD tells you is that every (finite dimensional) linear map is a sum of outer products; there are no other possibilities.
senderista 3y ago
This is great! I eventually figured out all these interpretations as well, but it took forever. In particular, the "sum of outer products" interpretation is crucial for understanding the SVD.

ndriscoll 3y ago

The most important interpretation IMO is that a matrix is a specification for a linear map. A linear map is determined by what it does to a basis, and the columns of a matrix are just the list of outputs for each basis element (e.g. the first column is `f(b_1)`. The nth column is `f(b_n)`). If A is the matrix for f and B the matrix for g (for some chosen bases), then BA is the matrix for the composition x -> g(f(x)). i.e. the nth column is `g(f(b_n))`. The codomain of f has to match the domain of g for composition to make sense, which means dimensions have to match (i.e. row count of A must be column count of B).

Buttons840 3y ago
It's debatable what the "most important" perspective is. For example, if I need a bunch of dot products between two sets of vectors, that doesn't seem like a linear map or a change of basis (not to me at least), and yet it's exactly what matrix multiplication is, just calculating a bunch of dot products between two sets of vectors. Or when I think about the singular value decomposition, I'm not thinking about linear maps and change of basis, but I am thinking about a sum of many outer product layers.
- ndriscoll 3y ago
  If you don't have a linear map in mind, why do you write your dot products with one set of column vectors and another set of row vectors? Computationally, the best way to do dot products would be to walk all of your arrays in contiguous memory order, so the row/column thing is an unnecessary complication. And if you have more than 2 matrices to multiply/"steps of dot products to do in a pipeline", there's almost certainly a relevant interpretation as linear maps lurking. Outer products are one way to define a "simple" linear map. What SVD tells you is that every (finite dimensional) linear map is a sum of outer products; there are no other possibilities.

senderista 3y ago

This is great! I eventually figured out all these interpretations as well, but it took forever. In particular, the "sum of outer products" interpretation is crucial for understanding the SVD.