25 ms·
Are eigenvalues or singular values used much in the popular recent stuff, like LLMs?
by bee_rider 1y ago
Are eigenvalues or singular values used much in the popular recent stuff, like LLMs?
- calebkaiser 1y agoLoRa uses singular value decomposition to get the low rank matrices. In different optimizers, you'll also see eigendecomposition or some approximation used (I think Shampoo does something like this, but it's been a while).