36 ms·

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3) will be symmetric (the Hessian). then just do your best to apply the single variable rules of differentiation (product rule, chain rule, etc.), and then mess around a bit with the result until all the dimensions match up and such that your result matches the appropriate case (1)-(3). For any more complicated functions you encounter such as the determinant, you can just look up its derivative when needed.

rademacher 6y ago
I suggest this reference [1] for anyone needing to look something up. [1] https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf
benrbray 6y ago
I think it conceptually helps a lot to distinguish between row and column vectors, even if they are coalesce in any production code you might write. (0a) An (nxm) matrix A represents a linear transformation f(x)=Ax from R^m -> R^n (0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax (3) The derivative of a function f : R^m -> R^n is a linear transformation with the same "type" R^m -> R^n as the original function. This means it is an (nxm) matrix. The other two rules can be derived from these rules: (1) The derivative of a function f : R^m -> R is a (1xm) matrix, aka a row vector. (2) The derivative of a function f : R^n -> R is an (nx1) matrix, aka a column vector. This perspective is helpful because, aided by the chain rule, we can easily derive all the other matrix algebra rules on the fly. Distinguishing row / column vectors helps catch conceptual type errors in places where it's inappropriate to add vectors and covectors together.
- omedusyo 6y ago
  In (2) I think you meant to write "f : R -> R^n"
  - benrbray 6y ago
    You're right, thanks!
- cambalache 6y ago
  > (0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax f'(x) is just A , not Ax, f'(x) != f(x)
  - benrbray 6y ago
    (I vouched for this comment since it exposes a common misconception people have about derivatives, and I'd rather reply than have it buried) It's unfortunate that the connection between matrices and linear transformations, and thereby between derivatives and linear transformations, is not adequately emphasized in the math courses that most people take. The derivative (or differential, or total derivative) of a function f : R^m -> R^n at a point x ∈ R^m is defined, formally, to be a linear transformation A : R^n -> R^m such that || f(x+h) - (f(x) + A(h)) || lim ------------------------ = 0 h->0 ||h|| Where ||.|| indicates a vector norm that measures the magnitude of its argument. Unpacking a bit, notice that > h is a vector in R^m > f(x+h) is the exact value of f at x+h > f(x)+A(h) is a linear approximation to f(x+h), using x as the base point and A as the linear approximator So this just a formal way of saying that the dervative A should be a good linear approximation to f. Every linear transformation has a corresponding matrix, and every matrix corresponds to a linear transformation (with respect to a choice of basis). So, when we say "the derivative of f at x is the matrix A", we really mean that "the derivative of f at x is the linear transformation represented by the matrix A". EDIT: In response to the parent poster, who has dug their heels in by posting a now-dead rant in reply to this post, I encourage you to refer to a copy of Rudin, "Principles of Mathematical Analysis", for a rigorous treatment of derivatives. See also: representation theory and group actions.
fn-mote 6y ago
I think that one of the reasons people look for a good exposition is so that they do not _have_ to muddle through. I know I look for a simple example or explanation that makes complicated cases clear. This 'explanation' might be correct, but I found the details that would help someone else follow it are missing. Especially someone with a computer science background who considers a m x 1 matrix different from a 1 x m matrix. I see a sister comment has explained this, but I want to point out that there are plenty of details... (like sister's statement 0b which looks to me more like an exponential function than a linear function) that's why one goes to a carefully written source. This paper is a carefully written explanation of the basics, ending with many applications of the chain rule. The "matrix product rule" is not mentioned, which is good in my opinion.
- yellowcake0 6y ago
  I admit it's a blue collar approach, but it's served me well when my goal is optimization or some such and I just need to get on with it.
PEJOE 6y ago
I struggling in DL when we started having to do tensor multiplications on the homework. Good with 2D matrices but the higher dimensional math gave me a lot of trouble when it should have been straight forward. Any tips there?
- hansvm 6y ago
  If you squint a bit tensor multiplications are still basically just matrix multiplications. At each coordinate in the output you'll place the sum of a bunch of element-wise products. Here are a few examples, some of which you've seen before and some of which you might not have: - Vector inner product (dot product) -- n-element vectors V and W are combined to create the single element (V_1 x W_1 + V_2 x W_2 + ...). Let's reframe this as a type of tensor contraction (multiplication). You have an (n,) tensor and another (n,) tensor (both "1D"), and you're getting rid of the first dimension in each of them, so the product is an (,) tensor ("0D"), and each element is computed by summing up n element-wise multiplications. - Vector outer product -- we have the same vectors V and W as before, but the result is a "2D" (n,n) matrix M where M_a,b = V_a x W_b. Reframing this as a type of tensor contraction, you aren't getting rid of, summing over, or contracting any dimensions, so the product includes every dimension from both inputs -- it's (n,n). Each element is computed by summing up 1 element-wise multiplication. - Matrix product -- Consider A an (i,j) matrix and B a (j,k) matrix. Standard matrix multiplication yields a result M (i,k). As a tensor multiplication, you're contracting (summing over) the "j" entries, so each output element is the sum of j element-wise multiplications. In particular, M_a,b = (A_a,1 x B_1,b + A_a,2 x B_2,b + ...). - Kronecker product (block matrix product) -- Consider the same matrices A and B as before. We aren't going to contract over any indices, so the result is a "4D" object M (i,j,j,k). To compute M_a,b,c,d you find all elements from A and B with those indices fixed, pair and multiply them, and sum them together. In particular, M_a,b,c,d = A_a,b x B_c,d. - Something higher dimensional -- Consider an input tensor A with shape (i,j,k,m,n) and another B with shape (m,n,w,y,z). As I'm sure you've noticed, even for objects as simple as vectors there are several choices of tensor multiplication (contraction) available. As with any other operation, the problem you're describing will govern which one you use (just like how you'd normally have a very clear reason for choosing an inner vs outer product in a vector problem). It's still interesting to consider what our options are though. The general rule is that sizes have to line up on both sides, so for this problem we could contract over {}, {m}, {n}, or {m,n}. The outputs for each of those products would have shapes, respectively, of (i,j,k,m,n,m,n,w,y,z) [everything], (i,j,k,n,n,w,y,z) [no m], (i,j,k,m,m,w,y,z) [no n], or (i,j,k,w,y,z) [no m or n]. I'm going to skip over the first three because they're enough like things you've seen before (kronecker product for the first, matrix product for the other two) that I don't think they're worth the comment space, but the last one is novel in some sense. Our methodology for multiplication is still straightforward though. Suppose the result is called M and we want to compute its value at some coordinate M_a,b,c,d,e,f. We'll pair up elements that look like A_a,b,c,?,? with elements that look like B_?,?,d,e,f (note that the inner, contracted coordinates need to match), multiply them together, and add them up. In particular, we get something like (A_a,b,c,1,1 x B_1,1,d,e,f + A_a,b,c,1,2 x B_1,2,d,e,f + A_a,b,c,2,1 x B_2,1,d,e,f + A_a,b,c,2,2 x B_2,2,d,e,f + ...). There are m x n element-wise products being added up to achieve _each_ coordinate of the output.

rademacher 6y ago

I suggest this reference [1] for anyone needing to look something up. [1] https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

benrbray 6y ago

I think it conceptually helps a lot to distinguish between row and column vectors, even if they are coalesce in any production code you might write. (0a) An (nxm) matrix A represents a linear transformation f(x)=Ax from R^m -> R^n (0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax (3) The derivative of a function f : R^m -> R^n is a linear transformation with the same "type" R^m -> R^n as the original function. This means it is an (nxm) matrix. The other two rules can be derived from these rules: (1) The derivative of a function f : R^m -> R is a (1xm) matrix, aka a row vector. (2) The derivative of a function f : R^n -> R is an (nx1) matrix, aka a column vector. This perspective is helpful because, aided by the chain rule, we can easily derive all the other matrix algebra rules on the fly. Distinguishing row / column vectors helps catch conceptual type errors in places where it's inappropriate to add vectors and covectors together.

omedusyo 6y ago
In (2) I think you meant to write "f : R -> R^n"
- benrbray 6y ago
  You're right, thanks!
cambalache 6y ago
> (0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax f'(x) is just A , not Ax, f'(x) != f(x)
- benrbray 6y ago
  (I vouched for this comment since it exposes a common misconception people have about derivatives, and I'd rather reply than have it buried) It's unfortunate that the connection between matrices and linear transformations, and thereby between derivatives and linear transformations, is not adequately emphasized in the math courses that most people take. The derivative (or differential, or total derivative) of a function f : R^m -> R^n at a point x ∈ R^m is defined, formally, to be a linear transformation A : R^n -> R^m such that || f(x+h) - (f(x) + A(h)) || lim ------------------------ = 0 h->0 ||h|| Where ||.|| indicates a vector norm that measures the magnitude of its argument. Unpacking a bit, notice that > h is a vector in R^m > f(x+h) is the exact value of f at x+h > f(x)+A(h) is a linear approximation to f(x+h), using x as the base point and A as the linear approximator So this just a formal way of saying that the dervative A should be a good linear approximation to f. Every linear transformation has a corresponding matrix, and every matrix corresponds to a linear transformation (with respect to a choice of basis). So, when we say "the derivative of f at x is the matrix A", we really mean that "the derivative of f at x is the linear transformation represented by the matrix A". EDIT: In response to the parent poster, who has dug their heels in by posting a now-dead rant in reply to this post, I encourage you to refer to a copy of Rudin, "Principles of Mathematical Analysis", for a rigorous treatment of derivatives. See also: representation theory and group actions.

fn-mote 6y ago

I think that one of the reasons people look for a good exposition is so that they do not _have_ to muddle through. I know I look for a simple example or explanation that makes complicated cases clear. This 'explanation' might be correct, but I found the details that would help someone else follow it are missing. Especially someone with a computer science background who considers a m x 1 matrix different from a 1 x m matrix. I see a sister comment has explained this, but I want to point out that there are plenty of details... (like sister's statement 0b which looks to me more like an exponential function than a linear function) that's why one goes to a carefully written source. This paper is a carefully written explanation of the basics, ending with many applications of the chain rule. The "matrix product rule" is not mentioned, which is good in my opinion.

yellowcake0 6y ago
I admit it's a blue collar approach, but it's served me well when my goal is optimization or some such and I just need to get on with it.

PEJOE 6y ago

I struggling in DL when we started having to do tensor multiplications on the homework. Good with 2D matrices but the higher dimensional math gave me a lot of trouble when it should have been straight forward. Any tips there?

hansvm 6y ago
If you squint a bit tensor multiplications are still basically just matrix multiplications. At each coordinate in the output you'll place the sum of a bunch of element-wise products. Here are a few examples, some of which you've seen before and some of which you might not have: - Vector inner product (dot product) -- n-element vectors V and W are combined to create the single element (V_1 x W_1 + V_2 x W_2 + ...). Let's reframe this as a type of tensor contraction (multiplication). You have an (n,) tensor and another (n,) tensor (both "1D"), and you're getting rid of the first dimension in each of them, so the product is an (,) tensor ("0D"), and each element is computed by summing up n element-wise multiplications. - Vector outer product -- we have the same vectors V and W as before, but the result is a "2D" (n,n) matrix M where M_a,b = V_a x W_b. Reframing this as a type of tensor contraction, you aren't getting rid of, summing over, or contracting any dimensions, so the product includes every dimension from both inputs -- it's (n,n). Each element is computed by summing up 1 element-wise multiplication. - Matrix product -- Consider A an (i,j) matrix and B a (j,k) matrix. Standard matrix multiplication yields a result M (i,k). As a tensor multiplication, you're contracting (summing over) the "j" entries, so each output element is the sum of j element-wise multiplications. In particular, M_a,b = (A_a,1 x B_1,b + A_a,2 x B_2,b + ...). - Kronecker product (block matrix product) -- Consider the same matrices A and B as before. We aren't going to contract over any indices, so the result is a "4D" object M (i,j,j,k). To compute M_a,b,c,d you find all elements from A and B with those indices fixed, pair and multiply them, and sum them together. In particular, M_a,b,c,d = A_a,b x B_c,d. - Something higher dimensional -- Consider an input tensor A with shape (i,j,k,m,n) and another B with shape (m,n,w,y,z). As I'm sure you've noticed, even for objects as simple as vectors there are several choices of tensor multiplication (contraction) available. As with any other operation, the problem you're describing will govern which one you use (just like how you'd normally have a very clear reason for choosing an inner vs outer product in a vector problem). It's still interesting to consider what our options are though. The general rule is that sizes have to line up on both sides, so for this problem we could contract over {}, {m}, {n}, or {m,n}. The outputs for each of those products would have shapes, respectively, of (i,j,k,m,n,m,n,w,y,z) [everything], (i,j,k,n,n,w,y,z) [no m], (i,j,k,m,m,w,y,z) [no n], or (i,j,k,w,y,z) [no m or n]. I'm going to skip over the first three because they're enough like things you've seen before (kronecker product for the first, matrix product for the other two) that I don't think they're worth the comment space, but the last one is novel in some sense. Our methodology for multiplication is still straightforward though. Suppose the result is called M and we want to compute its value at some coordinate M_a,b,c,d,e,f. We'll pair up elements that look like A_a,b,c,?,? with elements that look like B_?,?,d,e,f (note that the inner, contracted coordinates need to match), multiply them together, and add them up. In particular, we get something like (A_a,b,c,1,1 x B_1,1,d,e,f + A_a,b,c,1,2 x B_1,2,d,e,f + A_a,b,c,2,1 x B_2,1,d,e,f + A_a,b,c,2,2 x B_2,2,d,e,f + ...). There are m x n element-wise products being added up to achieve _each_ coordinate of the output.