4 ms·

Tried it. It’s actually faster to prune a very oversized neural network (also stated in the paper)

• AstralStorm 7y ago
  I didn't mean making the ANN itself, that is well described. I meant to optimize pruning rather than pruning at random or using ancient "optimal" algorithms. (They do not optimize for retraining ability. The goal is robustness.) The pruning algorithm used in the study is rudimentary. Interesting bit of biology is that humans go through some sort of pruning phase during adolescence. It is not random. It is not one shot either and real neurons cannot be resurrected or reset either.
  • sdenton4 7y ago
    I haven't heard of particularly compelling advantages for one pruning algorithm over another... So at this point, it seems best to just keep it dead simple. Assuming the lottery ticket hypothesis, you're just looking for the strong sub network, which should stand out under a number of approaches. The vestigial neurons shouldn't be doing much...
    • AstralStorm 7y ago
      They had to train the various pruned nets 15+ times each to know the cutoff to be a success... Also what is a strong subnetwork? What we get to know it's that their pruning algorithm produces a network that is vulnerable to weight randomization but better performing with original weights. If there is a better pruning algorithm, it could tell us more about structure of such subnetworks.