3 ms·
5090 has 32GB of RAM. Not sure if that’s enough to fit this model.
by p1esk 1y ago
5090 has 32GB of RAM. Not sure if that’s enough to fit this model.
- svnt 1y agoIt should fit enough of the layers to make it reasonably performant.
- IceWreck 1y agoLlamaCPP supports offloading some experts in a MoE model to CPU. The results are very good and even weaker GPUs can run larger models at reasonable speeds. n-cpu-moe in https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...