Yeah, the original work this paper follows up (by the same group: https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635) received a lot of attention

Yeah, the original work this paper follows up (by the same group: https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635) received a lot of attention in 2018 when it was uploaded to Arxiv & spawned a lot of followup work. Even though it's been demonstrated that these lottery tickets - sparse trainable subnetworks exist and can approximate the complete nn arbitrarily well, their properties are still not really understood. What is understood is that these subnetworks depend heavily on the initialization of the network, but that training the entire network together is necessary for generalization. These findings generally advocate for two-stage pruning approaches as opposed to cts regularization/sparsification throughout training. The question is how best to find these lottery tickets, and encourage them from the get-go. A lot of this work is also related to training adversarially robust networks. A composition of ReLU layers corresponds to a piecewise linear function, where the # of 'pieces' is like exponential in the # of neurons. It's well known that standard training of networks results in a highly non-linear pwl that is easily fooled by adversarial examples. The robustness of a neural network against adversarial examples is typically characterized by its smoothness. One question is how to train the network or prune neurons to encourage smoothness & reduce the complexity (i.e. # of linear pieces) of the nn.