4 ms·
That's not how llama.cpp works. It's a layer split. The GPUs handle a few layers and the CPU handles the rest. The GPU layers no matter how fast they complete s
by superkuh 2y ago
That's not how llama.cpp works. It's a layer split. The GPUs handle a few layers and the CPU handles the rest. The GPU layers no matter how fast they complete still have to wait on the CPU layers.
- colorant 2y agoThe ipex-llm implementation extends llama.cpp and includes additonal CPU-GPU hybrid optimizations for sparse MoE