766 ms·
At this point, is gguf/llama.cpp a more performant solution for unbatched inference on CUDA devices, or is exllamav2+flashattention still reigning supreme?
by lxe 2y ago
At this point, is gguf/llama.cpp a more performant solution for unbatched inference on CUDA devices, or is exllamav2+flashattention still reigning supreme?
- GuuD 2y agoThe difference is negligible on 2x 4090. There are more important differences like 4 bit KV cache.