2 ms·
I think you misunderstand. Especially > this model contains a smaller model, that has similar performance to the trained large model, without training The poi
by bonoboTP 7y ago
I think you misunderstand. Especially
> this model contains a smaller model, that has similar performance to the trained large model, without training
The point is the opposite. There is a small net X within big net Y, such that training only X gives the same performance as training all of Y.
- nil-sec 7y agoWhat you are stating is the original Lottery Ticket Hypothesis. What they prove in this paper is the stronger version, empirically noticed here https://arxiv.org/abs/1905.01067 https://arxiv.org/abs/1905.01067 and referred to as "supermasks". To quote from the paper posted here: "within a sufficiently overparameterized neural network with random weights (e.g. at initialization), there exists a subnetwork that achieves competitive accuracy". Edit: See also https://arxiv.org/abs/1911.13299 https://arxiv.org/abs/1911.13299
- bonoboTP 7y agoSeems like a "Library of Babel" type of thing. I'd have to read the full paper for how they find the subnets, but their mere existence is not so surprising. There's a huge sea of possible subnetworks. Basically SGD is replaced by whatever procedure you use to traverse the space of parameter subsets. Definitely interesting direction.