LlamaCPP supports offloading some experts in a MoE model to CPU. The results are very good and even weaker GPUs can run larger models at reasonable speeds. n-c

LlamaCPP supports offloading some experts in a MoE model to CPU. The results are very good and even weaker GPUs can run larger models at reasonable speeds. n-cpu-moe in https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...