3 ms·
Isn't this equivalent to changing the exponent bias? Like block floating point?
by deepnotderp 9y ago
Isn't this equivalent to changing the exponent bias? Like block floating point?
- simonbyrne 9y agoLooks similar, except they recenter as well.
- yorwba 9y agoIt's not just changing the exponent bias, but also adjusting the center of the representable range for gradients to be closer to the current value of the parameter (which is stored in full precision).