4 ms·

As someone not familiar with AI - I'm wondering if this is really as simple and revolutionary as the article states? MIT is kind of known for highly optimistic

As someone not familiar with AI - I'm wondering if this is really as simple and revolutionary as the article states? MIT is kind of known for highly optimistic press releases that maybe oversell sometimes, which makes it hard to know what's actually a real breakthrough sometimes

itronitron 6y ago
There is a lot of pie-in-the-sky PR filler in the article, and it's very light on details. It likely is as simple as the article states, but probably not very revolutionary. At some point the learned model will cease to be useful.
orange3xchicken 6y ago
Yeah, the original work this paper follows up (by the same group: https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635) received a lot of attention in 2018 when it was uploaded to Arxiv & spawned a lot of followup work. Even though it's been demonstrated that these lottery tickets - sparse trainable subnetworks exist and can approximate the complete nn arbitrarily well, their properties are still not really understood. What is understood is that these subnetworks depend heavily on the initialization of the network, but that training the entire network together is necessary for generalization. These findings generally advocate for two-stage pruning approaches as opposed to cts regularization/sparsification throughout training. The question is how best to find these lottery tickets, and encourage them from the get-go. A lot of this work is also related to training adversarially robust networks. A composition of ReLU layers corresponds to a piecewise linear function, where the # of 'pieces' is like exponential in the # of neurons. It's well known that standard training of networks results in a highly non-linear pwl that is easily fooled by adversarial examples. The robustness of a neural network against adversarial examples is typically characterized by its smoothness. One question is how to train the network or prune neurons to encourage smoothness & reduce the complexity (i.e. # of linear pieces) of the nn.
deepnotderp 6y ago
It's like 99% hyperbole, the gain is like 20% versus fine tuning. An interesting idea, but incremental in terms of improvements.
janhenr 6y ago
It is, unfortunately, not revolutionary at all. The Lottery Ticket Hypothesis won a Best Paper Award because the idea is very intuitive and elegant. However, the compression rates (9x) of this follow-up work on ImageNet are far from impressive. Take for example this paper: https://arxiv.org/abs/1810.07378 https://arxiv.org/abs/1810.07378 that reaches 30x compression rate on ImageNet. (Disclaimer: I know the original author and co-published a paper with him - not this one however)

itronitron 6y ago

There is a lot of pie-in-the-sky PR filler in the article, and it's very light on details. It likely is as simple as the article states, but probably not very revolutionary. At some point the learned model will cease to be useful.

orange3xchicken 6y ago

Yeah, the original work this paper follows up (by the same group: https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635) received a lot of attention in 2018 when it was uploaded to Arxiv & spawned a lot of followup work. Even though it's been demonstrated that these lottery tickets - sparse trainable subnetworks exist and can approximate the complete nn arbitrarily well, their properties are still not really understood. What is understood is that these subnetworks depend heavily on the initialization of the network, but that training the entire network together is necessary for generalization. These findings generally advocate for two-stage pruning approaches as opposed to cts regularization/sparsification throughout training. The question is how best to find these lottery tickets, and encourage them from the get-go. A lot of this work is also related to training adversarially robust networks. A composition of ReLU layers corresponds to a piecewise linear function, where the # of 'pieces' is like exponential in the # of neurons. It's well known that standard training of networks results in a highly non-linear pwl that is easily fooled by adversarial examples. The robustness of a neural network against adversarial examples is typically characterized by its smoothness. One question is how to train the network or prune neurons to encourage smoothness & reduce the complexity (i.e. # of linear pieces) of the nn.

deepnotderp 6y ago

It's like 99% hyperbole, the gain is like 20% versus fine tuning. An interesting idea, but incremental in terms of improvements.

janhenr 6y ago

It is, unfortunately, not revolutionary at all. The Lottery Ticket Hypothesis won a Best Paper Award because the idea is very intuitive and elegant. However, the compression rates (9x) of this follow-up work on ImageNet are far from impressive. Take for example this paper: https://arxiv.org/abs/1810.07378 https://arxiv.org/abs/1810.07378 that reaches 30x compression rate on ImageNet. (Disclaimer: I know the original author and co-published a paper with him - not this one however)