3 ms·
It depends but you cannot directly mix for example Ampere with Ada coz the lack of support for native FP8 in Ampere.
by binyu 5mo ago
It depends but you cannot directly mix for example Ampere with Ada coz the lack of support for native FP8 in Ampere.
- cthalupa 5mo agoThere are a variety of inference engines that support this, regardless of whether or not there is native FP8 in Ampere - llama.cpp will do it quite happily. VLLM you can do W8A16 quant too. There are a whole lot of ways to quantize models in general.
- binyu 5mo agoYeah, you'd need to use asymmetric quantization and other software techniques.