4 ms·
This may be a silly question but - so rather than train a big network and hope a subnetwork wins the lottery - why not just train a smaller network with multipl
by DrScientist 2y ago
This may be a silly question but - so rather than train a big network and hope a subnetwork wins the lottery - why not just train a smaller network with multiple runs with different starting weights?
- MalphasWats 2y agoLargely for the same reason people don't "just win" the lottery by buying every possible ticket (any more).
- scarmig 2y agoThe larger network contains exponentially more subnetworks. 10x the size contains far more than 10x subnetworks (although it'd also take more than 10x as long to train).
- DrScientist 2y agoAh ok - so you are saying the explosion of subnetworks is higher than the explosion of training time - leading to a win. In this case a positive benefit of combinatorial complexity.