That may be true, but you don't need a fundamentally different number representation in order to throw away most of it. Also, the die size cost is quite small.

That may be true, but you don't need a fundamentally different number representation in order to throw away most of it. Also, the die size cost is quite small. The biggest saving you could make is probably foregoing exactly rounded results, and only stipulate that the result has to be within ±1 lsb of the true value. That would save multipliers from computing a bunch of bits that don't end up in the result anyway, except for the rare case where they decide the rounding. That would probably be a good trade-off for most AI chips. For general purpose CPUs I don't think it is worth the breakage.