5 ms·
This is broadly correct for currently favoured software, but in computer science optimization problems you can usually trade off compute for memory and vice ver
by fulafel 8mo ago
This is broadly correct for currently favoured software, but in computer science optimization problems you can usually trade off compute for memory and vice versa.
For example just now from the front page: https://news.ycombinator.com/item?id=47242637 https://news.ycombinator.com/item?id=47242637 "Speculative Speculative Decoding"
Or this: https://openreview.net/forum?id=960Ny6IjEr https://openreview.net/forum?id=960Ny6IjEr "Low-Rank Compression of Language Models Via Differentiable Rank Selection"
- saagarjha 8mo agoNone of these really change the fundamental shape of the problem.
- oofbey 8mo agoGood point on speculative decoding techniques. I'd forgotten about them, and they're good. Would love to see some of these get into llama.cpp and friends, but it does require somebody to come up with a distilled draft model. But low rank compression isn't trading off compute for memory - it's just compressing the model. And critically, that's lossy compression. That's primarily a trade-off of quality for speed/size, with a little bit of added compute. Same goals as quantization. If there was some compute-intensive lossless compression of parameters, lots of people would be happy. But those floating point values look a lot like gaussian noise, making them extremely difficult to compress.