The only reason you want to use bfloat16 is to train "compressed" models. Best inference performance tends to be at 4 bith quantization (and many people are try

The only reason you want to use bfloat16 is to train "compressed" models. Best inference performance tends to be at 4 bith quantization (and many people are trying to make it go down, even to 1 bit, and with sparsity, actually below that) So this just doesn't matter. Ironically, memory speed is what matters. And that's exactly the point where the M1 actually beats even the M3.