4 ms·
Does this remind anyone else of the lottery ticket hypothesis deep learning paper that was published recently? [0] I think the gist was that neural networks ar
by jsharf 7y ago
Does this remind anyone else of the lottery ticket hypothesis deep learning paper that was published recently? [0]
I think the gist was that neural networks are usually much larger and more connected than necessary. The paper provided an algorithm to iteratively prune connections by seeing which links in the neural network had the lowest weights.
Here's a quote from the paper:
"The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the
original network after training for at most the same number of iterations."
[0] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635
- sbierwagen 7y agoOr EfficientNet, SOTA performance with a tenth of the compute: http://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html http://ai.googleblog.com/2019/05/efficientnet-improving-accu... You would hope the human brain would be pretty efficient, given the selection pressure but apparently not.
- jsharf 7y agoI mean, it's pretty incredible and coincidental that we evolved brains this sophisticated in the first place, out of a totally random process with selective pressure. After 13 billion years, I mean I think it's reasonable progress, but I wouldn't expect SOTA efficiency. Once you have intelligence, the evolutionary benefits are huge. There's all of a sudden not as much pressure to be efficient (well there was, for like 100,000 years, but then we invented fire and could cook, and that made getting energy/nutrition more efficient. FYI, at this point I'm just quoting from the book Sapiens)