Basically every activation function throws away half of the dynamic range at every neuron, which across a deep network is a lot. You make a good point about La

Basically every activation function throws away half of the dynamic range at every neuron, which across a deep network is a lot. You make a good point about LayerNorm, it's probably even worse.