2 ms·
And slow. They never tell you that quantization of many LLMs slows down your inference, sometimes by orders of magnitude.
by Der_Einzige 3y ago
And slow. They never tell you that quantization of many LLMs slows down your inference, sometimes by orders of magnitude.
- arugulum 3y agoIt depends on the quantization method, but yes some of the most commonly used ones are extremely slow.