4 ms·
What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous tok
by ryao 2y ago
What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.
- hmottestad 2y agoThat’s probably the main performance benefit of using the GPU. If you’re changing the active expert for every single token then it wouldn’t be any faster than just running it on the CPU. Once you can reuse the active expert for two tokens you’re already going to be a lot faster than just the CPU. More GPUs let you keep more experts active at a time.
- hexaga 2y agoExpert distribution should be approximately random token-by-token, so not likely.
- superkuh 2y agoThat's not how llama.cpp works. It's a layer split. The GPUs handle a few layers and the CPU handles the rest. The GPU layers no matter how fast they complete still have to wait on the CPU layers.
- colorant 2y agoThe ipex-llm implementation extends llama.cpp and includes additonal CPU-GPU hybrid optimizations for sparse MoE