3 ms·

I never liked the conventional normalization, this tanh looks like it should execute faster

• godelski 1y ago
  Depends on your context and goals. LayerNorm isn't going to bound you strictly into [-1,1] like this will. So that can have some advantages. A strict bounding can sometimes get you in trouble as it may not be as robust to novel inputs. For a basic example if you consider the "classic" normalization where you rescale your training data so that it is bound on [0,1] this does not mean that data from your test set will be in [0,1]. Does your model know how to generalize this? A scheme like this has the potential to land you in similar trouble. Think about the domain and range. If your training data is all in [-100, 100] then you might get a pretty wide tanh to accommodate that. But will the resultant filter be able to differentiate the value 100 from 1000? Probably not. The filter is going to optimize to the data it saw through training. Will there be some filter that has that capacity? Maybe. But also there are ways to process your data where this bound really wouldn't matter. We're getting into the weeds here but I'm just trying to illustrate why there are so many different normalization schemes. There's no one-size-fits all process and it is best to understand where certain methods have advantages and disadvantages (there's definitely advantages in that numbers closer to the origin have higher precision as the density of addressable values in fp{16,32,64} are not evenly distributed)