The actual inference is not run in Python in PyTorch, and its usually not bottlenecked by it. The problem is CUDA, not Python. LLMs are uniquely suited to loc

The actual inference is not run in Python in PyTorch, and its usually not bottlenecked by it. The problem is CUDA, not Python. LLMs are uniquely suited to local inference in projects like GGML because they are so RAM bandwidth heavy (and hence relatively compute lite), and relatively simple. Your kernel doesn't need to be hyper optimized by 35 Nvidia engineers in 3 stacks before its fast enough to start saturating the memory bus generating tokens. And yet its still an issue... For instance, llama.cpp is having trouble getting prompt ingestion performance in a native implementation comparable cuBLAS, even though they theoretically have a performance advantage by using the quantization directly.