3 ms·
llama.cpp afaik does not run portion of the model on the CPU. --cpu-moe just offloads weights to RAM, but they are still loaded to GPU for compute.
by lostmsu 7mo ago
llama.cpp afaik does not run portion of the model on the CPU. --cpu-moe just offloads weights to RAM, but they are still loaded to GPU for compute.