3 ms·
You can split tensors across an AMD GPU and Nvidia GPU - different architectures are not an issue. People run LLMs across some pretty crazy setups.
by cthalupa 4mo ago
You can split tensors across an AMD GPU and Nvidia GPU - different architectures are not an issue. People run LLMs across some pretty crazy setups.
- binyu 4mo agoIt depends but you cannot directly mix for example Ampere with Ada coz the lack of support for native FP8 in Ampere.
- cthalupa 4mo agoThere are a variety of inference engines that support this, regardless of whether or not there is native FP8 in Ampere - llama.cpp will do it quite happily. VLLM you can do W8A16 quant too. There are a whole lot of ways to quantize models in general.
- binyu 4mo agoYeah, you'd need to use asymmetric quantization and other software techniques.