3 ms·
> to training LLMs LLMs are made from dense matrices, aren't they?
by littlestymaar 1y ago
> to training LLMs
LLMs are made from dense matrices, aren't they?
- WJW 1y agoNot always, or rather not exclusively. For example, some types of distillation benefit from sparse-ifying the dense-ish matrices the original was made of [1]. There's also a lot of benefit to be had from sparsity in finetuning [2]. LLMs were merely one of the examples though, don't focus too much on them. The point was that sparse matmul makes up the bulk of scientific computations and a huge amount of industrial computations too. It's probably second only to the FFT in importance, so it would be wild if radix sort managed to eclipse it somehow. [1] https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/#sparsity https://developer.nvidia.com/blog/mastering-llm-techniques-i... [2] https://arxiv.org/html/2405.15525v1 https://arxiv.org/html/2405.15525v1
- almostgotcaught 1y agoAlmost all performant kernels employ structured sparsity