3 ms·
> Hybrid CPU/GPU Utilization: Seamlessly integrates memory/computation capabilities of CPU and GPU for a balanced workload and faster processing. Does this mea
by nextaccountic 3y ago
> Hybrid CPU/GPU Utilization: Seamlessly integrates memory/computation capabilities of CPU and GPU for a balanced workload and faster processing.
Does this means that it runs at same time at both CPU and GPU, being faster than a CPU-only or a GPU-only implementation on the same device?
edit: when running on integrated GPUs, can this benefit from the improved communication between CPU and GPU?
- rahimnathwani 3y agoGPU-only will be faster if you have enough VRAM. But if you want to run a model that requires more VRAM than you have, the current approach is to use llama.cpp and specify n_gpu_layers. That works, but is slower than GPU-only. OP claims to be 10x as fast as llama.cpp in the case when you can't fit the whole model in VRAM.