Sounds like the idea is that "outlier features" effectively give the model an inhibitory system for other features, and transformer models converge to consisten

Sounds like the idea is that "outlier features" effectively give the model an inhibitory system for other features, and transformer models converge to consistent outlier channel selection per layer at around the 6.7B param mark? It's interesting, biological neural networks are known to use inhibitory systems for signal gating and sharpening/sparsification, but it's typically handled by distinct neuron types that are strictly inhibitory. I wonder if one could boost smaller transformer performance with an analogous construction - add on a small additional module constructed for channel inhibition in the rest of the model.