Most of the performance gains also came from smaller datatypes such as fp4. Going from bfloat16/fp16 to fp8 is easy. fp4 is challenging but possible. fp2 can't

Most of the performance gains also came from smaller datatypes such as fp4. Going from bfloat16/fp16 to fp8 is easy. fp4 is challenging but possible. fp2 can't exist independently, because it is just the sign bit plus a single bit. The next frontier would be training directly with block floating point, where you have a shared exponent plus the two remaining bits. It's getting tight. Maybe it is possible to have mini LoRA blocks where an n times n block is approximated by the outer product of two n sized vectors. For n = 4 the savings would be 50% less FLOPs and for n=8 the savings would be 75% less FLOPs.