5 ms·
> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.
by Const-me 2y ago
> you can reach ~75% of peak performance for same matrix config
Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.
- ladyanita22 2y agoThat's exactly what I was wondering. That cannot be.
- stephencanon 2y agoProbably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.
- Const-me 2y agoOP implemented that stuff with WebGPU. In runtime, that weird language compiles into a compute shader which computes that stuff on GPU.
- saagarjha 2y agoI believe the point being made was that this could be done in the CPU faster than was achieved here.
- Const-me 2y agoYeah, but not on a single core. In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops. BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops FP32.
- saagarjha 2y agoI am told the numbers above require the core to have a matrix multiply unit (such as SME)
- menaerus 2y ago> Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. Isn't it that zen4 doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units? Because of this, a single AVX-512 instruction will occupy both FMA units and therefore I think that the theoretical limit for a single zen4 core should be half of the 134 GFLOPS number?
- Const-me 2y agoOne FMA counts as two floating-point operations: one multiplication and one addition. According to uops.info, Zen 4 cores can do two 8-wide FMA instructions per cycle, or one 16-wide FMA per cycle. See VFMADD132PS (YMM, YMM, YMM) and VFMADD132PS (ZMM, ZMM, ZMM) respectively, the throughput column is labelled TP. That’s where 32 FLOP/cycle number comes from. > doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units That’s correct, AVX512 doesn’t deliver more FLOPs on that CPU. The throughput of 32-byte FMA and 64-byte FMA is the same, 32 FLOP/cycle for FP32 numbers.
- menaerus 2y ago> One FMA counts as two floating-point operations: one multiplication and one addition. Right. This is where the discrepancy comes from. I counted FMA as a single FLOP.
- Const-me 2y agoBTW, it’s the same for GPUs. In DXBC shader byte code, mad instruction does FMA. When reporting theoretical FLOPs, GPU vendors count that as 2 float operations. For example, I have GeForce 4070 Ti Super in my desktop. The chip has 8448 execution units; nVidia calls them CUDA cores but I don’t like the name, the correct number is 66 cores where each core can do 4 wavefronts of 32 threads each. Anyway, these EUs can do one FP32 FMA each cycle, and the boost clock frequency is 2.61 GHz. Multiplying these two numbers results in 22.04928E+12 cycles*EU/second, and nVidia reports 44E+12 FLOPs peak FP32 performance of the GPU.
- adrian_b 2y agoNope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if you have obtained 1 TFLOP/s when running the program on 1 core you will get much more when running it on multiple cores, because all cores of the same type will use the same shared accelerator.
- Const-me 2y agoOne interesting thing about these newfangled matrix/AI/ML accelerators that’s very rarely mentioned on the internets, they only deliver that many TFLOP because they operate in very low precision. nVidia tensor cores support int8, couple versions of FP16 (BF16 and the standard IEEE one) and FP19 which they call TensorFloat-32. I think Intel AMX only supports int8 and BF16. None of them supports FP32 let alone FP64 input numbers, which makes them completely useless for traditional GEMM stuff.
- wtallis 2y agohttps://github.com/corsix/amx https://github.com/corsix/amx indicates that Apple's AMX supports up to 64-bit FP, but I don't see any performance metrics. They also have the ANE, which is the low-precision ML-focused accelerator.
- Const-me 2y agoI wasn’t aware there’re two completely different things from different companies both called AMX. I assumed that AMX: https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions The Apple’s version is indeed interesting. I wonder why haven’t Apple exposed it to programmers, or implemented a BLAS library on top of that thing?
- 2y ago
- shihab 2y agoYes, that's why I was focusing on percentage of peak hardware performance, not actual flops.