3 ms·
It’s much more relevant now because of the massive neural networks - specifically the need to compress them. Most of the state of the art model compression meth
by p1esk 4y ago
It’s much more relevant now because of the massive neural networks - specifically the need to compress them. Most of the state of the art model compression methods (quantization or pruning) rely on second order gradients, so you need to know how to compute, invert, and approximate the Hessian matrix (a matrix of all second order partial derivatives of the loss function wrt model parameters).
- thomasahle 4y agoWhat quantization methods rely on second order gradients? I thought people just applied gradient decent, then round to whatever precision.
- p1esk 4y agoCurrent SOTA post training quantization method is GPTQ: https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- mcapodici 4y agoThis field is so interesting, sad I didn’t get into it sooner