Cosine features tend to measure if two things point in the same direction. Convolutional features tend to measure how much of a template was mixed into a signal

Cosine features tend to measure if two things point in the same direction. Convolutional features tend to measure how much of a template was mixed into a signal. As such, one can expect cosine features to produce very similar results to a convolutional feature followed by a Sigmoid or Tanh. Remember what the first style transfer papers used for feature extraction? Conv+Tanh. After that, cosine features have been used with great success for generating the attention masks in transformer architectures. If I remember correctly, the original Transformer paper also suggested to normalize the features and to pre-scale the result with sqrt(1/N). So the new part in this formula is mostly the pow. The other parts (cosine distance & dividing by the norm) are already pretty much battle-tested in speech recognition. And the sign is just needed to restore what pow with an even exponent loses. In short, this suggests that applying a gamma-like transform to features increases contrast.