6 ms·
been running llama.cpp and vllm on same 4070, trying to batch more prompts for serving. llama.cpp was lagging bad once I hit batch 8 or so, even though GPU usag
by b0a04gl 1y ago
been running llama.cpp and vllm on same 4070, trying to batch more prompts for serving. llama.cpp was lagging bad once I hit batch 8 or so, even though GPU usage looked fine. vllm handled it way better.
later found vllm uses paged kv cache with layout that matches how the GPU wants to read fully coalesced without strided jumps. llama.cpp was using a flat layout that’s fine for single prompt but breaks L2 access patterns when batching.
reshaped kv tensors in llama.cpp to interleave ; made it [head, seq, dim] instead of [seq, head, dim], closer to how vllm feeds data into fused attention kernel. 2x speedup right there w.r.t same ops.
GPU was never the bottleneck. it was memory layout not aligning with SM’s expected access stride. vllm just defaults to layouts that make better use of shared memory and reduce global reads. that’s the real reason it scales better per batch.
this took its own time of say 2+days and had to dig under the nice looking GPU graphs to find real bottlenecks, it was widly trial and error tbf,
> anybody got idea on how to do this kinda experiment in hot reload mode without so much hassle??
- Der_Einzige 1y ago[flagged]
- mystified5016 1y ago[flagged]
- jcelerier 1y agodid you do a PR to integrate these changes back into llama.cpp ? 2x speedup would be absolutely wild
- zargon 1y agoAlmost nobody using llama.cpp does batch inference. I wouldn’t be surprised if the change is somewhat involved to integrate with all of llama.cpp’s other features. Combined with lack of interest and keeping up with code churn, that would probably make it difficult to get included, with the number of PRs the maintainers are flooded with.
- tough 1y agoif you open a PR, even if it doesnt get merged, anyone with the same issue can find it, and use your PR/branch/fix if it suits better their needs than master
- zargon 1y agoYeah good point. I have applied such PRs myself in the past. Eventually the code churn can sometimes make it too much of a pain to maintain them, but they’re useful for a while.
- buildxyz 1y agoAny speed up that is 2x is definitely worth fixing. Especially since someone has already figured out the issue and performance testing [1] shows that llamacpp* is lagging behind vLLM by 2x. This is a positive for all running LLMs locally using llamacpp. Even if llamacpp isnt used for batch inference now, this can allow those to finally run llamacpp for batching and on any hardware since vLLM supports only select hardware. Maybe finally we can stop all this gpu api software fragmentation and cuda moat as llamacpp benchmarks have shown Vulkan to be as or more performant than cuda or sycl. [1] https://miro.medium.com/v2/resize:fit:1400/format:webp/1*labdDV6bkGlbL5OGZXJkEA.png https://miro.medium.com/v2/resize:fit:1400/format:webp/1*lab...
- menaerus 1y agoSo, what exactly is batch inference workload and how would someone running inference on local setup benefit from it? Or how would I even benefit from it if I had a single machine hosting multiple users simultaneously? I believe batching is a concept only useful when during the training or fine tuning process.
- zargon 1y agoBatch inference is just running multiple inferences simultaneously. If you have simultaneous requests, you’ll get incredible performance gains, since a single inference doesn’t leverage any meaningful fraction of a GPU’s compute capability. For local hosting, a more likely scenario where you could use batching is if you had a lot of different data you wanted to process (lots of documents or whatever). You could batch them in sets of x and have it complete in 1/x the time. A less likely scenario is having enough users that you can make the first user wait a few seconds while you wait to see if a second user submits a request. If you do get a second request, then you can batch them and the second user will get their result back much faster than if they had had to wait for the first user’s request to complete first. Most people doing local hosting on consumer hardware won’t have the extra VRAM for the KV cache for multiple simultaneous inferences though.
- zozbot234 1y agoIt depends, if the optimization is too hardware-dependent it might hurt/regress performance on other platforms. One would have to find ways to generalize and auto-tune it based on known features of the local hardware architecture.
- amelius 1y agoYes, easiest is to separate it into a set of options. Then have a bunch of Json/yaml files, one for each hw configuration. From there, the community can fiddle with the settings and share new settings if new hardware is released.
- leeoniya 1y agotry https://github.com/ikawrakow/ik_llama.cpp https://github.com/ikawrakow/ik_llama.cpp
- tough 1y agodid you see yesterday nano-vllm [1] from a deepseek employee 1200LOC and faster than vanilla vllm? 1. https://github.com/GeeeekExplorer/nano-vllm https://github.com/GeeeekExplorer/nano-vllm
- chickenzzzzu 1y ago>GPU was never the botteneck >it was memory layout ah right so the GPU was the bottleneck then
- CardenB 1y agoNo because he was able to achieve the speedup without changing the GPU.
- chickenzzzzu 1y agoA more technically correct way to express this feeling is: "The computational power of the cores on the GPU was never the issue-- however the code that I wrote resulted in a memory bandwidth bottleneck that starved the GPU cores of data to work on, which is firmly within my responsibilities as a programmer -- to fully understand the bandwidth and latency characteristics of the device(s) i'm running on"
- saagarjha 1y agoI mean they didn't write the code
- chickenzzzzu 1y agoAnd that's the reason why they misspoke