3 ms·
How to evaluate performance of LLM inference frameworks
- brrrrrm 2y agoIf you're hitting a memory wall it means you're not scaling. This stuff really doesn't apply to scaled up inference but rather local small batch execution
- Havoc 2y ago>Aggressively pruning LLMs via quantization can significantly reduce their accuracy and you might be better off using a smaller model in the first place. Not sure that is correct. Quantization charts suggest its a fairly continous spectrum. i.e. an aggressive quant 13B ends up about same as a no quant 7B: https://www.researchgate.net/figure/Performance-degradation-of-quantized-models-Chart-available-at_fig1_377817624 https://www.researchgate.net/figure/Performance-degradation-...