7 ms·
The GPU is not always faster
- deleted 2y ago[deleted]
- tylermw 2y agoFor simple operations like the dot product (that also map extremely well to SIMD operations), yes, the CPU is often better, as there not much actual "computation" being done. More complex computations where the data does not need to transfer between the host and device amortize that transfer cost across multiple operations, and the balance can quickly weigh in favor of the GPU.
- deleted 2y ago[deleted]
- ramoz 2y agoSimpler research could've shown that there is a physical data transfer cost.
- hershey890 2y agoYeah classic use cases of GPUs like deep learning have you transfer the weights for the entire model to your GPU(s) at the of inference and after you that you only transfer your input over. The use case of transferring ALL data over every time is obviously misusing the GPU. If anyone’s ever tried running a model that’s too large for your GPU you will have experienced how slow this is when you have to pull in the model in parts for a single inference run.
- ssivark 2y agoA good mental model is to compare the number of floats being processed -vs- the number of primitive computations. Matrix multiplication has n^3 computation with n^2 data. Multiplication of large matrices is therefore special in the potential for "data re-use" (each float is used against all the columns or rows of the other matrix) -- so systems are designed to have a much higher flops throughput than memory bandwidth. A dot product is at the other extreme, where each float is used only once (loosely). Roofline plots [1] are framework to visualize system design from this perspective. [1] https://en.wikipedia.org/wiki/Roofline_model https://en.wikipedia.org/wiki/Roofline_model
- sigmoid10 2y agoThis is amplified even more by the fact that only the trivial implementation of matmul is O(n^3) whereas efficient ones (e.g BLAS) use things like the Strassen algorithm. You can also speed it up significantly by using cache-aware approaches when retrieving rows and columns. In practice there is a huge amount of theory behind this that is far beyond the average person's scope if they are not actual researchers.
- rnrn 2y agoIs there actually a BLAS implementation that uses strassen? I don’t think it’s accurate that only trivial implementations use the direct o(n^3) algorithm. AFAIK high performance BLAS implementations just use highly optimized versions of it.
- chessgecko 2y agoI remember reading that it’s too hard to get good memory bandwidth/l2 utilization in the fancy algorithms, you need to read contiguous blocks and be able to use them repeatedly. But I also haven’t looked at the gpu blas implementations directly.
- jcranmer 2y agoAIUI, Strassen gets used moderately commonly with non-floating-point datatypes, where numerical stability is less of a concern and multiplications are more useful to minimize than memory traffic. But from what I can tell, every floating-point BLAS library eschews Strassen, despite a steady trickle of papers saying "hey, there might be some small wins if we go to Strassen!"
- chillee 2y agoThe big issue with Strassen isn't performance - it's numerical stability.
- leecarraher 2y agoBLAS is just the library definition and not the implementation, so BLAS implementations could implement GEMM anyway they want. But in practice the triple loop method (n^3) is the most common, despite Strassen's and the more numerically stable, Winograd methods being well known and available for decades. But with most things involving real computing hardware, memory access patterns and locality tend to be more important for performance than operation counts
- SigmundA 2y agoWould be interesting to see what a unified memory setup can do like say an Apple M-series since this is the argument for unified memory, zero copy memory access between CPU and GPU.
- CowFreedom 2y agoEven the integrated Intel HD Graphics would be an interesting comparison.
- mirsadm 2y agoUnified memory is what makes using the GPU viable in my use case (mobile). The copy operation is almost always the slowest part. This is especially true for real time work.
- glitchc 2y agoIt's the relative difference of transfer overhead vs degree of compute. For one single operation, sure, the transfer overhead dominates. Add multiple compute steps (operations) however, and experiments will show that the GPU is faster as the transfer cost is fixed.
- moomin 2y agoEven if the GPU took literally no time at all to compute the results there would be workflows where doing it on the CPU was faster.
- Waterluvian 2y agoThe GPU is the gas station across town that’s five cents cheaper.
- adamc 2y agoGood analogy.
- _zoltan_ 2y agothis is simply not true. the post just uses outdated hardware, as pointed out above. a GH200 will run miles around any CPU.
- Dylan16807 2y agoIn this analogy, using better hardware affects the discount per gallon, but there are still situations where the closer gas station is the better choice.
- CowFreedom 2y agoThe gist of the post is that optimizations and interpretations thereof must always be made with respect to the underlying hardware.
- Legend2440 2y agoNo, it’s the bulk order from China that’s 100x cheaper but has a minimum order quantity of 100000 units and takes 6-8 weeks to get here.
- bob1029 2y agoL2 cache is 1000x closer than the PCIe bus. Pretty much anything that has to respond in ~realtime to outside events will run better on the CPU. You can use the GPU to visualize the state of the system with some small delay (e.g., video games), but it is not so great at modifying state in an efficient manner - especially when serialization of events & causality are important.
- dragontamer 2y agoThere is the compute vs communicate ratio. For problems like Matrix Multiplication, it costs N to communicate the problem but N^2 operations to calculate. For problems like dot product, it costs N to communicate but only N operations to calculate. Compute must be substantially larger than communication costs if you hope to see any benefits. Asymptotic differences obviously help, but linear too might help. You'd never transfer N data to perform a log(n) binary search for example. At that point communication dominates.
- AnotherGoodName 2y agoFor those skimming and to add to the above the article is using the gpu to work with system memory since that’s where they have the initial data and where they want the result in this case and comparing it to a cpu doing the same. The entire bottleneck is GPU to system memory. If you’re willing to work entirely with the gpu memory the gpu will of course be faster even in this scenario.
- saagarjha 2y agoAssuming your task is larger than kernel launch overhead, of course.
- HarHarVeryFunny 2y agoSure, but once you've justified moving data onto the GPU you don't want to incur the cost of moving the operation output back to the CPU unless you have to. So, for example, you might justify moving data to the GPU for a neural net convolution, but then also execute the following activation function (& subsequent operators) there because that's now where the data is.
- leeter 2y agoSo awhile back I was working on a chaotic renderer. This let me to a really weird set of situations: * If the GPU is a non-dedicated older style intel GPU, use CPU * If the GPU is a non-dedicated anything else do anything super parallel on the GPU but anything that can be BRRRRRT via CPU on the CPU because the memory is shared. * If the GPU is dedicated move everything to GPU memory and keep it there and only pull back small statistics if at all plausible.
- lmeyerov 2y agoComparing multicore wide AVX to CUDA is a bit of an unnecessary nuance for most folks. These make sense, but miss the forest from the trees: - Either way, you're writing 'cuda style' fine-grained data parallel code that looks and works very different from regular multithreaded code. You are now in a different software universe. - You now also have to think about throughput, latency hiding, etc. Nvidia has been commoditizing throughput-oriented hardware a lot better than others, and while AMD is catching up on some workloads, Nvidia is already advancing. This is where we think about bandwidth between network/disk=>compute unit. My best analogy here, when looking at things like GPU Direct Storage/Network, is CPU systems feel like a long twisty straw, while GPU paths are fat pipes. Big compute typically needs both compute + IO, and hardware specs tell you the bandwidth ceiling. To a large extent, ideas are cross-polinating -- CPUs looking more like GPUs, and GPUs getting the flexibility of CPUs -- but either way, you're in a different universe of how code & hardware works than 1990s & early 2000s intel.
- bee_rider 2y agoRealistically you should use Numpy or Cupy (or whatever the appropriate/fashionable library is) anyway, because tuning this stuff is a big pain. So, GPUs have the slight disadvantages that you have to think an about data movement and the drivers are a little less convenient to install, but it isn’t really a big deal.
- lmeyerov 2y agoAgreed! The bigger shift is switching to data parallel coding styles.
- PartiallyTyped 2y agoI am a big fan of jax for numerical computations these days.
- bee_rider 2y agoI’ve been seeing lots of posts about it lately. Haven’t had a chance to try it out, though.
- _ea1k 2y agoTBH, I'm finding that people underestimate the usefulness of CPU in both inference and fine tuning. PEFT with access to 64GB+ RAM and lots of cores can sometimes be cost effective.
- ramoz 2y agoI think engineers learn this quickly in high-scale/performance production environments. Even without hardware backgrounds. SLAs/costs create constraints you need to optimize against after promising the business line these magical models can enable that cool new feature for a million users. Traditional AI/ML models (including smaller transformers) can definitely be optimized for mass scale/performance on cpu-optimized infrastructure.
- rnrn 2y agohow can the multicore AVX implementation do a dot product (for arrays much larger than cache) at 340 GB/s on a system with RAM bandwidth < 50 GB/s
- alecco 2y agoI think the post is a bit disingenuous. But about bandwidth, matrix multiplications happen mostly in cache and that has a lot more bandwidth than RAM. Blocks of the matrix are loaded to cache (explicitly in CUDA) and used multiple times there. I'd exploit the better multi-level cache hierarchy in CPUs and make the code NUMA aware. But still I wouldn't bet against a recent GPU card.
- rnrn 2y ago> But about bandwidth, matrix multiplications happen mostly in cache and that has a lot more bandwidth than RAM. Blocks of the matrix are loaded to cache (explicitly in CUDA) and used multiple times there. The post is about dot product, not matrix multiply. Dot product has no data reuse
- rnrn 2y agoAnswer: it can’t. The author has updated the post with corrected AVX measurements, with the original ~340 GB/s revised down to 31.7 GB/s. (Thanks CowFreedom)
- alecco 2y ago> Each version is severely memory bandwidth bottlenecked, the CUDA version suffers the most with its practical 11.8 GB/s device-to-host bandwidth due to its PCI-Express 3.0 x16 interface. PCIe 3.0? What? https://cowfreedom.de/#appendix/computer_specs/ https://cowfreedom.de/#appendix/computer_specs/ > GeForce GTX 1050 Ti with Max-Q Design (PCIe 3.0 x16) (2016) > Intel Core i5-8300H (2020) This is a low-price 8 year old GPU and a 4 year old CPU. And he seems to be including loading the data to GPU. Newer cards have wide PCIe 5.0 or some faster interconnect, like Nvidia Grace-Hopper. Also he is comparing his own CUDA implementation. He should use one of the many available in CUBLAS/CUTLASS. Making a good CUDA GEMM is a very difficult art and very hardware specific
- jsheard 2y ago> Newer cards have wide PCIe 5.0 or some faster interconnect, like Nvidia Grace-Hopper. There aren't any (edit: consumer) GPUs with PCIe5 yet, though they probably aren't far off. Plenty already have PCIe4 though.
- alecco 2y agoConsumer cards are PCIe 4.0 x16. H100 PCIe version is PCIe 5.0 https://en.wikipedia.org/wiki/Hopper_(microarchitecture) https://en.wikipedia.org/wiki/Hopper_(microarchitecture). And it's been out 2 years already.
- throwway120385 2y agoSo everyone is supposed to do all of their testing on H100's?
- alecco 2y ago4090 (2022) PCIe 4.0 x16 is quite decent. The major limit is memory, not bandwidth. And 3090 (2020) is also PCIe 4.0 x16, and used cards are a bargain. You can hook them up with Nvlink. Nvidia is withholding new releases but the current hardware has more legs with new matrix implementations. Like FlashAttention doing some significant improvement every 6 months. Nvidia could make consumer chips with combined CPU-GPU. I guess they are too busy making money with the big cloud. Maybe somebody will pick up. Apple is already doing something like it even on laptops.
- hangonhn 2y agoQuestion from someone who doesn't know enough about GPUs: Recently a friend mentioned his workstation has 384 cores using 4 processors. This is starting to approach some of the core numbers of earlier GPUs. Is there a possibility that in the not too distant future that GPUs and CPUs will just converge? Or are the tasks done by GPUs too specialized?
- krapht 2y agoToo specialized. You can't use GPUs as general purpose computers. The basic unit of operation is the warp, which is 32 threads operating in lockstep (simplified). If you're not using all 32 threads, then you may as well not be using a GPU.
- immibis 2y agoYou'd also need extreme hyperthreading. A GPU can cycle between several warps in the same execution unit (barrel-processor-style), padding out the time per instruction to hide memory latency, while still getting the same throughput. That's counter to the fundamental design of CPUs.
- JonChesterfield 2y agoThey're really similar already. You can program a GPU much like you would a CPU (existence proof at https://news.ycombinator.com/item?id=42387267 https://news.ycombinator.com/item?id=42387267). There's a lot of obfuscation and hoarding of the ancient knowledge from the GPUs are special enthusiasts but the magic doesn't survive looking at the things. It's a masked vector isa. My dev GPU is a 6800XT. Cheapish gaming card from a little while ago, 16GB ram on the card. 72 "compute units" which are independent blocks of hardware containing memory ports, floating point unit, register file etc. Roughly "a core" from x64 world. Each of those can have up to 64 tasks ready to go, roughly a "hyperthread". It's 300W or so. There's some noise in the details, e.g. the size of the register file from the perspective of a hyperthread affects how many can be resident on the compute unit ready to run, the memory hierarchy has extra layers in it. The vector unit is 256byte wide as opposed to 64byte wide on x64. But if you wanted to run a web browser entirely on the GPU and were sufficiently bloody minded you'd get it done, with the CPU routing keyboard I/O to it and nothing else. If you want a process to sit on the GPU talking to the network and crunching numbers, don't need the x64 or arm host to do anything at all.
- deleted 2y ago[deleted]
- ltbarcly3 2y agoTaking a helicopter is not always faster than walking. Is this surprising or obvious?
- ryao 2y agoThe same thing applies to using a GPU to do inference with your weights in system memory. That is why nobody does that.
- shihab 2y agoAn otherwise valid point made using a terrible example.
- juunpp 2y agoTerrible post, really. Did they need 5 pages to say that a silly dot product micro-bench that is PCIE-bound loses to a CPU implementation? Why are they even comparing computation vs computation + memory transfer?
- refulgentis 2y agoBecause going to the GPU adds the overhead of memory and this is a simple way to demonstrate that. Did you know that already? Congrats, you're in the top 10% of software engineers
- juunpp 2y agoYou still don't need the unwieldy blog post to explain that. The SIMD section is unnecessary. Comparing the naive dot product on CPU vs GPU (+round-trip transfer) would have sufficed.
- kittikitti 2y agoBut how will people be impressed if you don't exhaust them with jargon?
- gdiamos 2y agoMemory locality depends on your perspective. The CPU would always be slower if the data originated in GPU memory.
- KeplerBoy 2y agoNot necessarily, i'm sure one could construct scenarios where offloading to the CPU makes sense. Think of highly branchy double precision stuff.
- ImHereToVote 2y agoThe GPU is never faster. It's parallel.
- fancyfredbot 2y agoThis article is a CPU benchmark and a PCI express bandwidth benchmark. It's masquerading as something else though.
- shaklee3 2y agoTo save some time for people, the title is absolutely incorrect. The GPU is significantly faster for this test, but the author is measuring PCIe bandwidth from a PCIe generation of 10 years ago. If instead they had used pcie5 the bandwidth would be quadrupled.