7 ms·
Basic Facts about GPUs
- kittikitti 1y agoThis is a really good introduction and I appreciate it. When I was building my AI PC, the deep dive research into GPU's took a few days but this lays it out in front of me. It's especially great because it touches on high-value applications like generative artificial intelligence. A notable diagram from the page that I wasn't able to find represented well elsewhere was the memory hierarchy of the A100 GPU's. The diagrams were very helpful. Thank you for this!
- b0a04gl 1y agobeen running llama.cpp and vllm on same 4070, trying to batch more prompts for serving. llama.cpp was lagging bad once I hit batch 8 or so, even though GPU usage looked fine. vllm handled it way better. later found vllm uses paged kv cache with layout that matches how the GPU wants to read fully coalesced without strided jumps. llama.cpp was using a flat layout that’s fine for single prompt but breaks L2 access patterns when batching. reshaped kv tensors in llama.cpp to interleave ; made it [head, seq, dim] instead of [seq, head, dim], closer to how vllm feeds data into fused attention kernel. 2x speedup right there w.r.t same ops. GPU was never the bottleneck. it was memory layout not aligning with SM’s expected access stride. vllm just defaults to layouts that make better use of shared memory and reduce global reads. that’s the real reason it scales better per batch. this took its own time of say 2+days and had to dig under the nice looking GPU graphs to find real bottlenecks, it was widly trial and error tbf, > anybody got idea on how to do this kinda experiment in hot reload mode without so much hassle??
- Der_Einzige 1y ago[flagged]
- mystified5016 1y ago[flagged]
- jcelerier 1y agodid you do a PR to integrate these changes back into llama.cpp ? 2x speedup would be absolutely wild
- zargon 1y agoAlmost nobody using llama.cpp does batch inference. I wouldn’t be surprised if the change is somewhat involved to integrate with all of llama.cpp’s other features. Combined with lack of interest and keeping up with code churn, that would probably make it difficult to get included, with the number of PRs the maintainers are flooded with.
- tough 1y agoif you open a PR, even if it doesnt get merged, anyone with the same issue can find it, and use your PR/branch/fix if it suits better their needs than master
- zargon 1y agoYeah good point. I have applied such PRs myself in the past. Eventually the code churn can sometimes make it too much of a pain to maintain them, but they’re useful for a while.
- buildxyz 1y agoAny speed up that is 2x is definitely worth fixing. Especially since someone has already figured out the issue and performance testing [1] shows that llamacpp* is lagging behind vLLM by 2x. This is a positive for all running LLMs locally using llamacpp. Even if llamacpp isnt used for batch inference now, this can allow those to finally run llamacpp for batching and on any hardware since vLLM supports only select hardware. Maybe finally we can stop all this gpu api software fragmentation and cuda moat as llamacpp benchmarks have shown Vulkan to be as or more performant than cuda or sycl. [1] https://miro.medium.com/v2/resize:fit:1400/format:webp/1*labdDV6bkGlbL5OGZXJkEA.png https://miro.medium.com/v2/resize:fit:1400/format:webp/1*lab...
- 1y ago
- leeoniya 1y agotry https://github.com/ikawrakow/ik_llama.cpp https://github.com/ikawrakow/ik_llama.cpp
- tough 1y agodid you see yesterday nano-vllm [1] from a deepseek employee 1200LOC and faster than vanilla vllm? 1. https://github.com/GeeeekExplorer/nano-vllm https://github.com/GeeeekExplorer/nano-vllm
- chickenzzzzu 1y ago>GPU was never the botteneck >it was memory layout ah right so the GPU was the bottleneck then
- CardenB 1y agoNo because he was able to achieve the speedup without changing the GPU.
- chickenzzzzu 1y agoA more technically correct way to express this feeling is: "The computational power of the cores on the GPU was never the issue-- however the code that I wrote resulted in a memory bandwidth bottleneck that starved the GPU cores of data to work on, which is firmly within my responsibilities as a programmer -- to fully understand the bandwidth and latency characteristics of the device(s) i'm running on"
- saagarjha 1y agoI mean they didn't write the code
- chickenzzzzu 1y agoAnd that's the reason why they misspoke
- SoftTalker 1y agoContrasting colors. Use them!
- jasonjmcghee 1y agoIf the author stops by- the links and the comments in the code blocks were the ones that I had to use extra effort to read. It might be worth trying to increase the contrast a bit. The content is really great though!
- cubefox 1y agoThe website seems to use alpha transparency for text. A grave, contrast-reducing, sin.
- xeonmc 1y agoIt’s just liquid-glass text and you’ll get used to it soon enough.
- currency 1y agoThe author might be formatting for and editing in dark mode. I use edge://flags/#enable-force-dark and the links are readable.
- Yizahi 1y agofont-weight: 300; I'm 99% sure that author had designed this website on an Apple Mac with so called "font smoothing" enabled, which makes all regular fonts artificially "semi-bold". So to make a normal looking font, Mac designers use this thinner font weight and then Apple helpfully makes it kinda "normal". https://news.ycombinator.com/item?id=23553486 https://news.ycombinator.com/item?id=23553486
- neuroelectron 1y agoJfc
- elashri 1y agoGood article summarizing good chunk of information that people should have some idea about. I just want to comment that the title is a little bit misleading because this is talking about the very choices that NVIDIA follows in developing their GPU archs which is not what always what others do. For example, the arithmetic intensity break-even point (ridge-point) is very different once you leave the NVIDIA-land. If we take AMD Instinct MI300, it has up to 160 TFLOPS FP32 paired with ~6 TB/s of HBM3/3E bandwidth gives a ridge-point near 27 FLOPs/byte which is about double that of the A100’s 13 FLOPs/byte. The larger on-package HBM (128 – 256 GB) GPU memory also shifts the practical trade-offs between tiling depth and occupancy. Although this is very expensive and does not have CUDA (which can be good and bad at the same time).
- apitman 1y agoUnfortunately Nvidia GPUs are the only ones that matter until AMD starts taking their computer software seriously.
- fooblaster 1y agoThey are. It's just not at the consumer hardware level.
- have-a-break 1y agoYou could argue it's all the nice GPU debugging tools nVidia provides which makes GPU programming accessible. There are so many potential bottlenecks (normally just memory access patterns, but without tools to verify you have to design and run manual experiments).
- tucnak 1y agoThis misconception is repeated time and time again; software support of their datacenter-grade hardware is just as bad. I've had the displeasure of using MI50, MI100 (a lot), MI210 (very briefly.) All three are supposedly enterprise-grade computing hardware, and yet, it was a pathetic experience with a myriad of disconnected components which had to be patched, & married with a very specific kernel version to get ANY kind of LLM inference going. Now, the last of it I bothered with was 9 months ago; enough is enough.
- eapriv 1y agoSpoiler: it’s not about how GPUs work, it’s about how to use them for machine learning computations.
- oivey 1y agoIt’s a pretty standard run down of CUDA. Nothing to do with ML other than using relu in an example and mentioning torch.
- neuroelectron 1y agoASCII diagrams, really?
- LarsDu88 1y agoMaybe this should be titled "Basic Facts about Nvidia GPUs" as the WARP terminology is a feature of modern Nvidia GPUs. Again, I emphasize "modern" An NVIDIA GPU from circa 2003 is completely different and has baked in circuitry specific to the rendering pipelines used for videogames at that time. So most of this post is not quite general to all "GPUs" which a much broader category of devices that don't necessarily encompass the type of general purpose computation we use modern Nvidia GPUs for.
- bjornsing 1y agoSo how are we doing with whole program optimization on the compiler level? Feels kind of backwards that people are optimizing these LLM architectures, one at a time.
- geoffbp 1y ago“Arithmetic Intensity (AI)” Hmm
- naganotonicbuy 1y ago[dead]
- gdiamos 1y agoWow - the title is "basic facts" - but it should be "key insights" You wouldn't believe how many PhDs I've met who have no idea what a roofline is.
- Agentlien 1y agoI wasn't expecting the strong CUDA/ML focus. My own work is primarily in graphics and performance in video games; while this is all familiar and useful it feels like a very different view of the hardware than mine.
- saagarjha 1y ago> The “Peak Compute” roof of 19.5 TFLOPS is an ideal, achievable only with highly optimized instructions like Tensor Core matrix multiplications and high enough power limits. As mentioned below, 19.5 TFLOPS is the FP32 compute roofline, which doesn't support Tensor Cores. If you want to use those you need to use FP16 and you can get substantially improved performance.