That's a basic part of regularization! Weights are "penalized" for being too big, and encouraged to be as small as possible. This results in most of the weights

That's a basic part of regularization! Weights are "penalized" for being too big, and encouraged to be as small as possible. This results in most of the weights being zero, which is the same as them not existing at all. There is some research on pruning networks after training, removing small weights and nodes that don't contribute much. This results in much smaller neural networks, which run better on smartphones or embedded devices. This isn't done often though because on GPUs it doesn't result in higher performance. Because of the way SIMD instructions work or something, there's no way to take advantage of sparser neural networks. A synapse of weight 0 still requires a multiplication by zero, and adding the zero to the sum.