3 ms·
I'm not sure the lower and lower precision optimization is a good idea long term. It indicates that models are really sparse and that may be true right now but
by jmward01 2y ago
I'm not sure the lower and lower precision optimization is a good idea long term. It indicates that models are really sparse and that may be true right now but I think that is likely just because we have some bad ideas about how to train them and not because they really should be that sparse.
- rfoo 2y agoWell, let's enjoy free "sparsity" until it doesn't. Being able to train a really good model but in higher precision only is a research problem. Low precision training and inference is an engineering one. We've been doing this since CNN days (9 years ago if not more), and I believe we have a good few years left.
- sudosysgen 2y agoThe activation function throws away much of the dynamic range of floating point numbers, it's relatively clear that having a lot of range where the activation is already saturated is unlikely to be useful.
- jmward01 2y agoThat depends on the activation function. I personally think Layernorm is destroying density (and have some solid evidence for this) but it is in use because there is a lot of missing supporting structure to really help pump data into the weights so it helps so long as we are using simple linear combinations.
- sudosysgen 2y agoBasically every activation function throws away half of the dynamic range at every neuron, which across a deep network is a lot. You make a good point about LayerNorm, it's probably even worse.