4 ms·
TL;DR: make sure your matrix dimensions are divisible by 2 often.
by amelius 2y ago
TL;DR: make sure your matrix dimensions are divisible by 2 often.
- ykonstant 2y agoI have always done that instinctively, even when there is no formal demand by the system. Every time I have implemented matmuls, at any level, with any optimization requirement, partitioning into dyadic blocks had always sped things up. So I try to feed the system the nicest numbers I can muster!
- zusammen 2y ago[dead]
- carlmr 2y agoAnd if you can't make sure all of them are divisible by 2 often, at least pick the inner dimensions of your matmuls (if possible).
- chillee 2y agoWell, that'll help with a lot :) But dealing with wave quantization requires dimensions that aren't neceessarily a multiple of 2, and often are a multiple of the number of SMs on a GPU (i.e. 132 on an H100)