6 ms·
Smarter Training of Neural Networks
- dplarson 7y agoSince the article didn't link to the paper: - "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" - https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635
- vincentmarle 7y agoThere's actually a link to the paper in the article's right sidebar: https://openreview.net/forum?id=rJl-b3RcF7 https://openreview.net/forum?id=rJl-b3RcF7 (the reviews here are also an interesting read)
- joshvm 7y agoThanks for that, nothing has changed in academic review! Reviewer 2, 3: 9/10 great. Reviewer 1: "The paper seems a bit preliminary and unfinished." Authors go on to win best paper with their submission.
- eli_gottlieb 7y agoMan, after getting beaten up in some recent reviews, I needed to see that.
- iovrthoughtthis 7y agonature nurture but for neural networks?
- bayesian_horse 7y agoI have thought for a while about "brain surgery" for deep neural nets, much like a permanent kind of dropout. The idea is that you black out a set of neurons/filters and then train for a short while to overcome the performance penalty. To find the "set" of blacked out cells you could use a genetic algorithm or something, gradually increasing the number of masks. The last step would be rearranging the network such that the not-blacked-out cells are contiguous, but form smaller layers. And I remember Hinton hinted at replacing multiple layers by one layer (or no layer), or big layers by smaller layers through retraining.
- unixhero 7y agoAnd then what would happen? What is the effect of this? Very deep an interesting thoughts.
- space_fountain 7y agoI think mostly faster training time. If you can work out connections that don't matter early in the process you could start ignoring them.
- bayesian_horse 7y agoMy idea was more about faster inference by reducing the required number of operations. It might not be worthwhile at all!
- _0ffh 7y agoWell, there are long standing network pruning techniques like Optimal Brain Damage (1990) [1] and Optimal Brain Surgeon [2]. Could you be more concrete about what you would be trying to do? [1] https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.7223 https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.... [2] https://papers.nips.cc/paper/647-second-order-derivatives-for-network-pruning-optimal-brain-surgeon.pdf https://papers.nips.cc/paper/647-second-order-derivatives-fo...
- bayesian_horse 7y agoI didn't invest any serious work in this idea, and didn't even research if somebody has come up with such things before.
- Chirono 7y agoGoogle recently created MorphNet which can take a large network and do the pruning stage automatically. https://ai.googleblog.com/2019/04/morphnet-towards-faster-and-smaller.html https://ai.googleblog.com/2019/04/morphnet-towards-faster-an...
- thesz 7y agoI have to add this: https://openai.com/blog/block-sparse-gpu-kernels/ https://openai.com/blog/block-sparse-gpu-kernels/ They train block-sparse neural networks, where sparseness learned during training.
- GistNoesis 7y agoThis is an interesting idea that may be combined with the article suggested idea. In the article, if I understood correctly, what they propose is train your network once, remove the x% of smaller absolute magnitude weights, retrain your network fixing those smaller magnitude weights to 0 starting from the same initial starting point. The idea behind is that your optimization process the first time is telling you that the solution is near a subspace where the weights are 0 but it can't really converge to it. So you project to this subspace by enforcing the weights to 0. Then you retrain again and the search will be easier because the space is smaller, but because you are starting from the same starting point you are kind of guaranteed that you will be able to reach the same optimum but projected. The problem of the sparsity in the article is that while some weights are 0, they are 0 through masking, therefore you are still doing the computations, and you don't really benefit from the sparsity. If you have enough 0, you can benefit from the sparsity by using some sparse representation, but those are typically an order of magnitude slower than the dense representation. Combining the idea of the article with the idea from OpenAI of block-sparse neural networks which reduce the operations done without suffering too much from the non-locality and indirection of a sparse representation. After training normally (provided you have enough memory) (eventually with a sparse-block regularization term to help induce block-sparsity) you may try to prune in such a way that the least significant sparse-blocks are pruned, therefore you may expect both the boost in speed, and the better accuracy and convergence properties. This is a kind of two phase search, first we look for a finer structure, then we restart to find the best weights for this finer structure.
- thesz 7y agoYou can use gather operations: https://stackoverflow.com/questions/22330322/dereference-pointers-in-xmm-register-gather https://stackoverflow.com/questions/22330322/dereference-poi... And they are not "order of magnitude slower": https://stackoverflow.com/questions/24756534/in-what-situation-would-the-avx2-gather-instructions-be-faster-than-individually https://stackoverflow.com/questions/24756534/in-what-situati... Unless you operate in binary, of course. Results from above suggest that you can leave 12.5% of your weights to be non-zero and get nice 4x speed up (1/8 operations done twice as slow). Accuracy start to drop at about 12.5% of weights remaining in both OP and OpenAI papers.
- vincentmarle 7y ago> Their key innovation was the idea that connections that were pruned after the network was trained might never have been necessary at all. To test this hypothesis, they tried training the exact same network again, but without the pruned connections. Importantly, they "reset" each connection to the weight it was assigned at the beginning of training. > “It was surprising to see that re-setting a well-performing network would often result in something better,” says Carbin. This, intuitively, makes sense to me. It seems that the pruned model has to waste less training cycles on inferior weights, and it can therefore spend more cycles on further optimizing the good weights.
- enriquto 7y agoI never understood the obsessive emphasis on the training of neural networks. Many people see neural networks as objects that you train. But this is just a small thing that you can do with them. Neural networks are, first and foremost, objects that compute. This computation can be tuned by setting some parameters, and one way to set these parameters is by training. But this is not the only way, and it is not necessarily the more interesting one.
- hour_glass 7y agoWhat does this mean? I can shape a network some other way?
- enriquto 7y agoYes, you can think very hard and set up the weights by hand, or by an automatic "compiler" that transforms your desired algorithm into a deep net.
- cshimmin 7y agoGood luck manually writing a program that achieves state of the art performance on any typical machine learning task... we'll see which "training" method takes longer.
- Ragib_Zaman 7y agoGood practitioners do both. Whatever knowledge about the weights you can predetermine, you set those as the initial values of your weights. Then you improve on that by training. It really depends on the problem and size of your network how many weights you'll have a good estimate for but yes, you should try to estimate good weights yourself first. This practice is well known but here's a concrete source from Andrej Karpathy: https://karpathy.github.io/2019/04/25/recipe/ https://karpathy.github.io/2019/04/25/recipe/ "init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias."
- plutonorm 7y agoWe should train another neural network to produce the initial values for the target network. Training data would be the initial values of the sub network along with its loss. Then we have the network generate initial values that are more likely to lead yo useful sub networks.
- intuitionist 7y agoThis is essentially the idea behind “meta-learning,” or “learning to learn,” with the slight difference that most of the meta-learning literature aims to initialize networks that can learn quickly (“few-shot learning”). It has pretty good theoretical grounding but in practice seems to be quite expensive.
- plutonorm 7y agoLol. I've seen idea after idea, dismissed and misunderstood on Internet forums, from hackernews to reddit.com/r/machine learning. Time after time I see ideas, my own and those of others, that were dismissed, spawning papers by researchers. Such a toxic atmosphere, I don't know why I bother posting. I am well aware of what meta learning is. If you think that because the idea above comes under the umbrella of metalearning it isn't an interesting idea, then I don't know what to say to you... and the fact that it's downvoted just goes to show the lack of creativity and intuition exhibited by your average hackernews member.
- intuitionist 7y agoI never implied that it wasn’t an interesting idea, and your weird defensiveness suggests to me that there might be a common thread to your getting repeatedly dismissed/downvoted other than misunderstood genius.
- andrewnc 7y agoThis is neat, but one important point is that networks NEED to be significantly over parameterized for SGD like methods to find the minima. I know that's slightly tangential. Another interesting point is that NNs can be thought about as preforming coordinate transformations on the data manifold, which means these sub nets are potentially approximations to those transformations, potentially up to some scaling factor. I'm excited to see where this goes
- sjg007 7y ago>networks NEED to be significantly over parameterized for SGD like methods to find the minima Why?
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- bigred100 7y agoI’ve only seen a small amount on this but the theoretical analysis I’ve seen used this as an assumption There may be some in here https://youtu.be/zZbHVaw_W9A https://youtu.be/zZbHVaw_W9A
- bitforger 7y agoRecent follow-up work from Uber at ICLR: https://eng.uber.com/deconstructing-lottery-tickets/ https://eng.uber.com/deconstructing-lottery-tickets/