3 ms·
The activation function throws away much of the dynamic range of floating point numbers, it's relatively clear that having a lot of range where the activation i
by sudosysgen 2y ago
The activation function throws away much of the dynamic range of floating point numbers, it's relatively clear that having a lot of range where the activation is already saturated is unlikely to be useful.
- jmward01 2y agoThat depends on the activation function. I personally think Layernorm is destroying density (and have some solid evidence for this) but it is in use because there is a lot of missing supporting structure to really help pump data into the weights so it helps so long as we are using simple linear combinations.
- sudosysgen 2y agoBasically every activation function throws away half of the dynamic range at every neuron, which across a deep network is a lot. You make a good point about LayerNorm, it's probably even worse.