It depends on the inputs. E.g., if two elements of the input are highly correlated, the network can assign an arbitrarily high positive weight to one element an

It depends on the inputs. E.g., if two elements of the input are highly correlated, the network can assign an arbitrarily high positive weight to one element and then correct this by assigning an arbitrarily high negative weight to the other, with the total contribution of the two elements being reasonable. However, if these input elements are not as tightly correlated in the test data, the results may be bad. L2 regularisation ensures that this does not happen. (In practice, the effect will be mitigated by the activation function, but weights do blow up in training, leading to "saturated" models, which exhibit weird behaviours.)