I can see areas where being able to adjust weights to a wide range of sensitivities is important. Take the original transformer diagram for example. https://i0

I can see areas where being able to adjust weights to a wide range of sensitivities is important. Take the original transformer diagram for example. https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/06/Transformer.png https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/06... See the arrows coming in from the sides of the "Add & Norm" boxes. These come from bypassing the layer earlier. This gives back-propagation a easier path to adjust the weights earlier in the model. The results of the inputs are added, if the sensitivity were comparable to both inputs to the 'Add and Norm' a strong bypass signal would swamp the information from the path with more processing. The bypass allows for the easy early training to travel back through the bypass channels, but as the network develops more nuance, the longer path will provide additional accuracy, as this improves, the strength of the bypass signal can be reduced and while still remaining part of the overall signal it probably ends up way less significant than the longer path. A larger range of weights would allow this combination of shouting and whispers. note: not an expert, grain of salt etc.