2 ms·
There are a variety of inference engines that support this, regardless of whether or not there is native FP8 in Ampere - llama.cpp will do it quite happily. VLL
by cthalupa 4mo ago
There are a variety of inference engines that support this, regardless of whether or not there is native FP8 in Ampere - llama.cpp will do it quite happily. VLLM you can do W8A16 quant too.
There are a whole lot of ways to quantize models in general.
- binyu 4mo agoYeah, you'd need to use asymmetric quantization and other software techniques.