5 ms·
Knowing matrix derivatives well is one of those skills that were essential in machine learning 10 years ago. Not so much anymore with the dominance of massive n
by eachro 3y ago
Knowing matrix derivatives well is one of those skills that were essential in machine learning 10 years ago. Not so much anymore with the dominance of massive neural networks.
- rdedev 3y agoI wouldn't completely discount it. Like I wanted to speed up a certain loss function by rewriting it in triton. But I have to manually code the backward pass function for it in triton. For that I need to know how to calculate the derivative. Then again I guess with pytorch 2.0 compile function it might not be necessary
- 6gvONxR4sf7o 3y agoIt depends what kind of work you do. I still find it to be really handy.
- p1esk 3y agoIt’s much more relevant now because of the massive neural networks - specifically the need to compress them. Most of the state of the art model compression methods (quantization or pruning) rely on second order gradients, so you need to know how to compute, invert, and approximate the Hessian matrix (a matrix of all second order partial derivatives of the loss function wrt model parameters).
- thomasahle 3y agoWhat quantization methods rely on second order gradients? I thought people just applied gradient decent, then round to whatever precision.
- p1esk 3y agoCurrent SOTA post training quantization method is GPTQ: https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- mcapodici 3y agoThis field is so interesting, sad I didn’t get into it sooner
- vinte 3y agoI guess it will be like assembly or low level programing languages like c for future ML engineers. Its good to know the basics but not always needed.
- deleted 3y ago[deleted]