4 ms·
Llama.cpp wasn't optimized for multi GPU setups. You need to split individual layers onto the GPU instead of simple offloading.
by imtringued 3y ago
Llama.cpp wasn't optimized for multi GPU setups. You need to split individual layers onto the GPU instead of simple offloading.