4 ms·

I never learned matrix calculus so I find this tool helpful for following technical papers that involve some matrix calculus

by j7ake 9y ago

jamessb 9y ago
You might also find the Matrix Cookbook [0] to be a useful reference. [0]: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf
gajomi 9y ago
I have done all kinds of work that required some kind of matrix calculus in one form or another. There are of course all kinds of references (sibling links to my favorite), but I have found that more often than not really the best way to get the results you want is just to calculate them yourself. The work involved is usually tedious but trivial. But working through it goes along way to help make some sense of the various identities that come out. In looking at an identity one may see a transpose here, an inner product there, but not be able to assign any importance or distinction to any particular term at first glance. Working through a few calculations help with this. EDIT: I suppose I should also say that I never "learned" matrix calculus either, in the sense that I internalized the various features unique to matrices under derivatives and integrals. The calculations I refer to above are crude, naive ones in the scalar notation under whatever coordinate system seems appropriate.
- jey 9y ago
  When I was starting out in machine learning, as a programmer with the most rudimentary calculus background, it was easy to derive algorithms that had terms like "gradient w.r.t X of log(det(inv(λI + A X A')))" which absolutely stumped me when trying to derive the gradient by hand by elementwise partials. However, thanks to Minka's notes and the Matrix Cookbook, I was able to eventually get a handle on easy techniques for these derivations! It's certainly no substitute for getting a handle on the theory first by studying a textbook, but these pattern-matching shorthands are important practical techniques.
  - platz 9y ago
    How did you start in machine learning. Did you come from a top 10 school?
    - jey 9y ago
      No, I failed out of university and am self-taught. I spent many years deliberately underemployed working on self-study and coding up projects at the limit of my abilities. I had decided to bet on investing in myself instead of just carrying out some boss’s wishes, which would be more optimized to benefit the company than my own growth and development. Here’s a great resource if you’re starting out today: http://datasciencemasters.org http://datasciencemasters.org
      - asddddd 9y ago
        Thanks for the link! That's more or less the road I'm starting down right now, albeit with likely a bumpier past (dropped out of HS in addition to college and spent a few years doing almost nothing productive.)
      - digitalshankar 9y ago
        Inspiring story!!
surajx 9y ago
The best resource I have come across is the Wikipedia page. It's comprehensive and concise. https://en.wikipedia.org/wiki/Matrix_calculus https://en.wikipedia.org/wiki/Matrix_calculus
- mindcrime 9y ago
  This kind of stuff just infuriates me. I can't see any good reason for having so many competing notational standards for the same thing: The notation used here is commonly used in statistics and engineering, while the tensor index notation is preferred in physics. Two competing notational conventions split the field of matrix calculus into two separate groups. The two groups can be distinguished by whether they write the derivative of a scalar with respect to a vector as a column vector or a row vector. Both of these conventions are possible even when the common assumption is made that vectors should be treated as column vectors when combined with matrices (rather than row vectors). A single convention can be somewhat standard throughout a single field that commonly uses matrix calculus (e.g. econometrics, statistics, estimation theory and machine learning). However, even within a given field different authors can be found using competing conventions. Authors of both groups often write as though their specific convention is standard. Seriously? So if I want to read a paper that uses Matrix Calculus, it's not enough to just understand Matrix Calculus in general.. no, first I have to decipher which of a legion of possible notations the author used, and then keep that state in mind when thinking about that paper in relation to another, which might use yet another notation. I understand that ultimately nobody is an position to mandate the adoption of a universal standard, but part of me wishes there were (this is, of course, not a problem that is limited to Matrix Calculus).
  - scentoni 9y ago
    Some other examples: sign conventions in thermodynamics conventions in Fourier transforms short billion vs long billion calorie vs Calorie English vs. metric units SI vs cgs metric esu vs Gaussian units in EM

I never learned matrix calculus so I find this tool helpful for following technical papers that involve some matrix calculus

by j7ake 9y ago

I never learned matrix calculus so I find this tool helpful for following technical papers that involve some matrix calculus

jamessb 9y ago

You might also find the Matrix Cookbook [0] to be a useful reference. [0]: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

gajomi 9y ago

I have done all kinds of work that required some kind of matrix calculus in one form or another. There are of course all kinds of references (sibling links to my favorite), but I have found that more often than not really the best way to get the results you want is just to calculate them yourself. The work involved is usually tedious but trivial. But working through it goes along way to help make some sense of the various identities that come out. In looking at an identity one may see a transpose here, an inner product there, but not be able to assign any importance or distinction to any particular term at first glance. Working through a few calculations help with this. EDIT: I suppose I should also say that I never "learned" matrix calculus either, in the sense that I internalized the various features unique to matrices under derivatives and integrals. The calculations I refer to above are crude, naive ones in the scalar notation under whatever coordinate system seems appropriate.

jey 9y ago
When I was starting out in machine learning, as a programmer with the most rudimentary calculus background, it was easy to derive algorithms that had terms like "gradient w.r.t X of log(det(inv(λI + A X A')))" which absolutely stumped me when trying to derive the gradient by hand by elementwise partials. However, thanks to Minka's notes and the Matrix Cookbook, I was able to eventually get a handle on easy techniques for these derivations! It's certainly no substitute for getting a handle on the theory first by studying a textbook, but these pattern-matching shorthands are important practical techniques.
- platz 9y ago
  How did you start in machine learning. Did you come from a top 10 school?
  - jey 9y ago
    No, I failed out of university and am self-taught. I spent many years deliberately underemployed working on self-study and coding up projects at the limit of my abilities. I had decided to bet on investing in myself instead of just carrying out some boss’s wishes, which would be more optimized to benefit the company than my own growth and development. Here’s a great resource if you’re starting out today: http://datasciencemasters.org http://datasciencemasters.org
    - asddddd 9y ago
      Thanks for the link! That's more or less the road I'm starting down right now, albeit with likely a bumpier past (dropped out of HS in addition to college and spent a few years doing almost nothing productive.)
    - digitalshankar 9y ago
      Inspiring story!!

surajx 9y ago

The best resource I have come across is the Wikipedia page. It's comprehensive and concise. https://en.wikipedia.org/wiki/Matrix_calculus https://en.wikipedia.org/wiki/Matrix_calculus

mindcrime 9y ago
This kind of stuff just infuriates me. I can't see any good reason for having so many competing notational standards for the same thing: The notation used here is commonly used in statistics and engineering, while the tensor index notation is preferred in physics. Two competing notational conventions split the field of matrix calculus into two separate groups. The two groups can be distinguished by whether they write the derivative of a scalar with respect to a vector as a column vector or a row vector. Both of these conventions are possible even when the common assumption is made that vectors should be treated as column vectors when combined with matrices (rather than row vectors). A single convention can be somewhat standard throughout a single field that commonly uses matrix calculus (e.g. econometrics, statistics, estimation theory and machine learning). However, even within a given field different authors can be found using competing conventions. Authors of both groups often write as though their specific convention is standard. Seriously? So if I want to read a paper that uses Matrix Calculus, it's not enough to just understand Matrix Calculus in general.. no, first I have to decipher which of a legion of possible notations the author used, and then keep that state in mind when thinking about that paper in relation to another, which might use yet another notation. I understand that ultimately nobody is an position to mandate the adoption of a universal standard, but part of me wishes there were (this is, of course, not a problem that is limited to Matrix Calculus).
- scentoni 9y ago
  Some other examples: sign conventions in thermodynamics conventions in Fourier transforms short billion vs long billion calorie vs Calorie English vs. metric units SI vs cgs metric esu vs Gaussian units in EM