4 ms·
I am surprised that the paper does not even cite the Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635, https://eng.u
by stared 5y ago
I am surprised that the paper does not even cite the Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635, https://eng.uber.com/deconstructing-lottery-tickets/ https://eng.uber.com/deconstructing-lottery-tickets/).
In the LTH paper (IMHO the most fundamental deep learning publication in the last few years), the number of tickets goes as layer_size^n_layers.
- renewiltord 5y agoConsidering the subject, it is at least somewhat amusing that you double posted this.
- gwern 5y agoI don't see how lottery tickets yield the isoperimetry result, even in a heuristic or handwavy sort of way. Yes, a larger network is more likely to have good-scoring subnetworks; sure. But that's all it says. What does that tell me about how efficiently I can construct an adversarial example? For that, I need something else, like, say a geometric argument about what sort of network will interpolate between high-dimensional datapoints with properties like "not changing much in response to small input changes"...
- paraph1n 5y agoIn the linked Uber paper, can someone explain the plots with the ellipses? It's a 2d plot of initial vs final weight, yes? But where does the ellipse shape come from and why?