5 ms·
Great article! For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak perfor
by shihab 2y ago
Great article!
For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).
- zanussbaum 2y agothanks! and yes definitely not at CUDA levels :)
- weinzierl 2y ago75% can't be the best we can do. What would reach 100% or nearly 100%? Handcoded assembly?
- jsheard 2y agoWith GPUs it's not uncommon to run out of memory bandwidth before you max out the theoretical FLOPS. They may have a ton of bandwidth but it's never enough. That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.
- wbl 2y agoShouldn't the roofline inform capacity assessments?
- KeplerBoy 2y agoSure, but rooflines don't account for stuff like memory granularity. You not only have to do a lot of bytes per flop to achieve the necessary arithmetic intensity, you also have to access those bytes in a coalesced way. I.e., you want to access consecutive bytes, which are ideally already in registers.
- stephencanon 2y agoFor sufficiently large GEMM you should never run out of bandwidth before you max out FLOPS if your blocking is organized correctly, because the arithmetic scales like O(n^3) while the memory access scales like O(n^2).
- saagarjha 2y agoIn theory, yes. In practice you will probably be forced to tile your GEMM and incur the penalty of redundant memory accesses.
- stephencanon 2y agoSure, but still on each tile, you do O(k^3) compute with O(k^2) memory, and you generally arrange things so that at least one tile is in L1 and at least one other is in L2/LLC (using CPU idioms), so again, you have plenty of bandwidth (typical choices of k are in the ballpark of ~32, and a 32:1 compute to memory ratio is just fine on most hardware, especially if some of those accesses are coming from fast memory)
- saagarjha 2y agoI don't think so? It is too late for me to actually do the math on this but if you take the degenerate case where the tile size is literally 1 element then you will do as many loads as arithmetic operations. Thus I would consider any sort of fixed tiling (which you would be resigned to due to your caches being of limited size) requiring O(n^3) loads?
- adev_ 2y ago> That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process. It is not specific to the GPUs: this kind of optimizations are pretty common on CPU too where latency kills you and 200 cycles spent wasted on doing compute can actually be faster than a single cache miss trying to fetch data. This is pretty common for many SIMD algorithms actually. Memory is currently lagging behind compute on almost every type of modern hardware, and it will very likely become worst, not better.
- david-gpu 2y agoThe parameters of the matrix multiply, such as the size of the matrices, impose some limits to how close you can get to the peak theoretical performance in a particular GPU. Not all possible matrix multiplies are equally valuable to optimize a priori, so the hardware is designed to perform best on problems that are financially significant, such as modern LLMs. As for handcoded assembly, do you believe that it would be financially sound to hand code and maintain thousands of kernels that way, even if you believed that they would be faster?
- littlestymaar 2y ago> As for handcoded assembly, do you believe that it would be financially sound to hand code and maintain thousands of kernels that way, even if you believed that they would be faster? Why not? We do so for cryptographic primitives and video codecs. And why are you talking about “thousands of kernels”? AI programs only need a small amount of different kernels so it doesn't sound intractable.
- david-gpu 2y ago> AI programs only need a small amount of different kernels That is not the case. What appears like a simple matmul operation actually requires these libraries to select which specific kernel out of the many internally available to execute. If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.
- littlestymaar 2y agoIs that really different from AV codecs in terms of scale though?
- david-gpu 2y agoI am not at liberty to discuss more than that.
- 2y ago
- deleted 2y ago[deleted]
- Const-me 2y ago> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.
- ladyanita22 2y agoThat's exactly what I was wondering. That cannot be.
- stephencanon 2y agoProbably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.
- Const-me 2y agoOP implemented that stuff with WebGPU. In runtime, that weird language compiles into a compute shader which computes that stuff on GPU.
- saagarjha 2y agoI believe the point being made was that this could be done in the CPU faster than was achieved here.
- Const-me 2y agoYeah, but not on a single core. In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops. BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops FP32.
- saagarjha 2y ago
- brrrrrm 2y agohow are you running CUDA on the integrated Apple silicon GPU these days?
- KeplerBoy 2y agoYou are not.