3 ms·
That depends on the activation function. I personally think Layernorm is destroying density (and have some solid evidence for this) but it is in use because the
by jmward01 2y ago
That depends on the activation function. I personally think Layernorm is destroying density (and have some solid evidence for this) but it is in use because there is a lot of missing supporting structure to really help pump data into the weights so it helps so long as we are using simple linear combinations.
- sudosysgen 2y agoBasically every activation function throws away half of the dynamic range at every neuron, which across a deep network is a lot. You make a good point about LayerNorm, it's probably even worse.