5 ms·

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical op

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class was winding up, he joked and said "just kidding, don't do that... here's how we can do this with a Taylor expansion", and proceeded to derive the same identity in what felt like 30 seconds. Also, don't forget the Jacobian and gradient aren't the same thing!

edflsafoiewq 2y ago
Can you give an example?
- vismit2000 2y ago
  Check out this classic from 3b1b - How (and why) to raise e to the power of a matrix: https://youtu.be/O85OWBJ2ayo https://youtu.be/O85OWBJ2ayo
  - kgwgk 2y ago
    For those who prefer reading (I’ve not seen the video, but it seems related): https://sassafras13.github.io/MatrixExps/ https://sassafras13.github.io/MatrixExps/ “Thanks to a fabulous video by 3Blue1Brown [1], I am going to present some of the basic concepts behind matrix exponentials and why they are useful in robotics when we are writing down the kinematics and dynamics of a robot.”
  - esafak 2y ago
    They didn't show how to actually do it using matrix decomposition!
- godelski 2y ago
  https://math.stackexchange.com/questions/3680708/what-is-the-difference-between-the-jacobian-hessian-and-the-gradient https://carmencincotti.com/2022-08-15/the-jacobian-vs-the-hessian-vs-the-gradient/
- sfpotter 2y ago
  If you mean for how to use Taylor expansions and linear algebra, here's one I just made up. Let's say I want to differentiate tr(X^T X), tr is the trace, X is a matrix, and X^T is its transpose. Expand: tr((X + dX)^T (X + dX)) = tr(X^T X) + 2 tr(X^T dX) + tr(dX^T dX). Our knowledge of linear algebra tells us that tr is a linear map. Hence, dX -> 2 tr(X^T dX) is the linear mapping corresponding to the Jacobian of tr(X^T X). With a little more work we could figure out how to write it as a matrix.
C-x_C-f 2y ago
> Also, don't forget the Jacobian and gradient aren't the same thing! Every gradient is a Jacobian but not every Jacobian is a gradient. If you have a map f from R^n to R^m then the Jacobian at a point x is an m x n matrix which linearly approximates f at x. If m = 1 (namely if f is a scalar function) then the Jacobian is exactly the gradient. If you already know about gradients (e.g. from physics or ML) and can't quite wrap your head around the Jacobian, the following might help (it's how I first got to understand Jacobians better): 1. write your function f from R^n to R^m as m scalar functions f_1, ..., f_m, namely f(x) = (f_1(x), ..., f_m(x)) 2. take the gradient of f_i for each i 3. make an m x n matrix where the i-th row is the gradient of f_i The matrix you build in step 3 is precisely the Jacobian. This is obvious if you know the definition and it's not a mathematically remarkable fact but for me at least it was useful to demystify the whole thing.
- sfpotter 2y ago
  For m = 1, the gradient is a "vector" (a column vector). The Jacobian is a functional/a linear map (a row vector, dual to a column vector). They're transposes of one another. For m > 1, I would normally just define the Jacobian as a linear map in the usual way and define the gradient to be its transpose. Remember that these are all just definitions at the end of the day and a little bit arbitrary.
  - oddthink 2y ago
    I'd say a gradient is usually a covector / one-form. It's a map from vector directions to a scalar change. ie. df = f_x dx + f_y dy is what you can actually compute without a metric; it's in T*M, not TM. If you have a direction vector (e.g. 2 d/dx), you can get from there to a scalar.
    - sfpotter 2y ago
      I'm not a big Riemannian geometry buff, but I took a look at the definition in Do Carmo's book and it appears that "grad f" actually lies in TM, consistent with what I said above. Would love to learn more if I've got this mixed up. This would be nice, because it would generalize the "gradient" from vector calculus, which is clearly and unambiguously a vector.

edflsafoiewq 2y ago

Can you give an example?

vismit2000 2y ago
Check out this classic from 3b1b - How (and why) to raise e to the power of a matrix: https://youtu.be/O85OWBJ2ayo https://youtu.be/O85OWBJ2ayo
- kgwgk 2y ago
  For those who prefer reading (I’ve not seen the video, but it seems related): https://sassafras13.github.io/MatrixExps/ https://sassafras13.github.io/MatrixExps/ “Thanks to a fabulous video by 3Blue1Brown [1], I am going to present some of the basic concepts behind matrix exponentials and why they are useful in robotics when we are writing down the kinematics and dynamics of a robot.”
- esafak 2y ago
  They didn't show how to actually do it using matrix decomposition!
godelski 2y ago
https://math.stackexchange.com/questions/3680708/what-is-the-difference-between-the-jacobian-hessian-and-the-gradient https://carmencincotti.com/2022-08-15/the-jacobian-vs-the-hessian-vs-the-gradient/
sfpotter 2y ago
If you mean for how to use Taylor expansions and linear algebra, here's one I just made up. Let's say I want to differentiate tr(X^T X), tr is the trace, X is a matrix, and X^T is its transpose. Expand: tr((X + dX)^T (X + dX)) = tr(X^T X) + 2 tr(X^T dX) + tr(dX^T dX). Our knowledge of linear algebra tells us that tr is a linear map. Hence, dX -> 2 tr(X^T dX) is the linear mapping corresponding to the Jacobian of tr(X^T X). With a little more work we could figure out how to write it as a matrix.

C-x_C-f 2y ago

> Also, don't forget the Jacobian and gradient aren't the same thing! Every gradient is a Jacobian but not every Jacobian is a gradient. If you have a map f from R^n to R^m then the Jacobian at a point x is an m x n matrix which linearly approximates f at x. If m = 1 (namely if f is a scalar function) then the Jacobian is exactly the gradient. If you already know about gradients (e.g. from physics or ML) and can't quite wrap your head around the Jacobian, the following might help (it's how I first got to understand Jacobians better): 1. write your function f from R^n to R^m as m scalar functions f_1, ..., f_m, namely f(x) = (f_1(x), ..., f_m(x)) 2. take the gradient of f_i for each i 3. make an m x n matrix where the i-th row is the gradient of f_i The matrix you build in step 3 is precisely the Jacobian. This is obvious if you know the definition and it's not a mathematically remarkable fact but for me at least it was useful to demystify the whole thing.

sfpotter 2y ago
For m = 1, the gradient is a "vector" (a column vector). The Jacobian is a functional/a linear map (a row vector, dual to a column vector). They're transposes of one another. For m > 1, I would normally just define the Jacobian as a linear map in the usual way and define the gradient to be its transpose. Remember that these are all just definitions at the end of the day and a little bit arbitrary.
- oddthink 2y ago
  I'd say a gradient is usually a covector / one-form. It's a map from vector directions to a scalar change. ie. df = f_x dx + f_y dy is what you can actually compute without a metric; it's in T*M, not TM. If you have a direction vector (e.g. 2 d/dx), you can get from there to a scalar.
  - sfpotter 2y ago
    I'm not a big Riemannian geometry buff, but I took a look at the definition in Do Carmo's book and it appears that "grad f" actually lies in TM, consistent with what I said above. Would love to learn more if I've got this mixed up. This would be nice, because it would generalize the "gradient" from vector calculus, which is clearly and unambiguously a vector.