2 ms·
llama.cpp saw a 10% performance improvement from using CUDA graphs, so inference does benefit from it: https://developer.nvidia.com/blog/optimizing-llama-cpp-a
by ryao 2y ago
llama.cpp saw a 10% performance improvement from using CUDA graphs, so inference does benefit from it:
https://developer.nvidia.com/blog/optimizing-llama-cpp-ai-inference-with-cuda-graphs/ https://developer.nvidia.com/blog/optimizing-llama-cpp-ai-in...
- saagarjha 2y agoYeah I mean I am sure there are workloads that it helps but there are also a lot where you'd think it would help and the driver will actually just fail to cooperate.