5 ms·
> FFMA SASS interleaving > We observe a performance improvement in the CUTLASS FP8 kernel between NVCC 12.2 and 12.3. By comparing the compiled SASS, we discov
by Bimos 2y ago
> FFMA SASS interleaving
> We observe a performance improvement in the CUTLASS FP8 kernel between NVCC 12.2 and 12.3. By comparing the compiled SASS, we discover that one bit in a series of FADD instructions is flipped in an interleaving pattern. After referencing some open-source CUDA assembler implementations, we identified that this bit controls yield, which may enhance warp-level parallelism (just a guess, yielding the current warp and let other warps work).
> To leverage this, we develop a similar script to modify the FFMA instructions in the compiled binary. Besides simply modifying the yield bit, we also flip the reuse bit (registers cannot be reused if the warp is yielded). This adjustment improves performance (10%+ in some cases) for fine-grained scaling FP8 GEMMs by creating more opportunities to overlap MMA instructions with promotion FFMA instructions.
I would say it is really mind-blowing.
- ETH_start 2y ago[flagged]
- tough 2y agoI think he might mean hyperbolically figuratively so
- kneegerman 2y agoorthogonally
- Bimos 2y agoI edited it.
- dang 2y agoLiterally literally means not literally. I love it when words turn into their opposites!
- blackeyeblitzar 2y agoFrom what I read elsewhere, this is the type of typical performance optimization for matrix math you would see when performance is critical. It’s just not been applied yet to this specific problem by other AI players since it wasn’t a necessity for other companies. But eventually everyone would probably end up here regardless.
- Bimos 2y agoI think most AI players rely on high performance GEMM. But most people would be satisfied with cutlass or cublas, and the others implement gemm themselves, but not necessarily use undocumented features?
- creato 2y agoUsing undocumented features is not rare. People reverse engineered Apple's undocumented AMX instructions on their CPU, and I know people use undocumented/private extensions for several different kinds of GPUs.
- mitthrowaway2 2y agoHow many people does it take to implement this? A 10% gain in performance could pay for a lot of people's salaries when your company is spending hundreds of millions on GPU clusters.
- saagarjha 2y agoI mean it’s not a significant change so one? But that isn’t to say anyone could do it.
- rvz 2y agoJust a reminder, this is the third of many open source releases from DeepSeek that they are willing to release, and that release is a very trivial low bar for them to find optimizations when it is needed. I guess since the majority here are blown away by the very low-level code involved, it tells me that they're likely not ready to use it or have been stuck on very high level tools that abstract this away.
- randomNumber7 2y agoI tell you a secret. Most devs do something wrong when they start rolling out their own linear algebra library. Thats why people use LAPAC, BLAS, etc...
- shaklee3 2y agoscott grey figured out this exact thing and more back in 2015 for maxwell, and it's been written about many times since by other people.