13 ms·

Something about this reminds me of the 'Winning Ticket Hypothesis' in artifical neural networks: that some 'random' initializations prime a network far better f

by gojomo 5y ago

Something about this reminds me of the 'Winning Ticket Hypothesis' in artifical neural networks: that some 'random' initializations prime a network far better for later faster learning. From https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 - the abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy. Are these innate patterns of activity priming mammalian brains in a similar way?

teawrecks 5y ago
In the context of real life we draw a distinction between thus neonatal dreamlike state and post birth "reality". But in the context of a neural net, I don't think such a distinction is necessary. They both sound like they're training the network. Still interesting to consider the parallels though.
emtel 5y ago
It’s not just mammals. I’ve always been astounded that spiders know how to spin webs. Their mothers don’t teach them how!
- gota 5y ago
  Or this, which is mind-blowing: https://www.smithsonianmag.com/smart-news/dragonfly-undertakes-epic-multi-generational-migration-each-year-180971190/ https://www.smithsonianmag.com/smart-news/dragonfly-undertak... Insects that have large migration patterns - all of the individuals die before coming back to the initial spot; only their descendants continue the cycle. How?!
  - chromanoid 5y ago
    Maybe epigenetics? I mean blank slate theory is off the table anyway... In addition think about how bees can tell harvest locations to each other by dancing: https://en.wikipedia.org/wiki/Waggle_dance https://en.wikipedia.org/wiki/Waggle_dance This dance is probably also something those bees know before being born.
teruakohatu 5y ago
> Are these innate patterns of activity priming mammalian brains in a similar way? A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning? Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright. I have wondered about how true this is. Maybe our brains are just so much more powerful, or capable of much deeper understanding, that we require a much lower initial learning rate. The lottery ticket could explain why some children learn to walk at 6 months, while others are closer to 24 months, with no different in intelligence or motor skills in later life. Edit: As some people below have correctly pointed out, human babies could not walk within an hour for physiological reasons, but nor do they exhibit the basic motor skills, spatial reasoning or image processing required for walking. Many human babies even struggle to feed for the first 24-48 hours or longer. Meanwhile baboon babies can hold onto their mother from birth while the mother climbs trees.
- soheil 5y ago
  > A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning? Probably neither. How does the human baby "learn" to breath almost immediately after birth? I suspect the answer here is also the same.
- rowanG077 5y ago
  Interesting. Could this also partly explain child geniuses? In general they are worlds ahead early in life. But they aren't significantly better then the non child genius peers later in life.
- taneq 5y ago
  > Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints I strongly doubt that human babies are physiologically capable of walking before 6-9 months regardless of their level of brain development. The entire baby is born many months "early" compared with other mammals due to the limitations of the human pelvis.
gota 5y ago
I guess I can see some parallel the way you describe it, but it is not that tight of an analogy IMO. Correct me if I'm wrong but the Lottery Ticket Hypothesis mainly means that, for many of our tasks, we start with needlessly huge networks, and thus a good portion of the learning process is just the network learning to "cope with the noise" from the extraneous connections Hence why the lottery ticket network (being a smaller version of the same initial network) can achieve similar accuracy (but not 'better') in much shorter time - because you took away all the confounding stuff. That kind of 'counterfactual' doesn't really translate to a biological domain, right? Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right? Neither is the fetus brain compared to the baby brain (which is more to the point of the article). So the analogy breaks down there too I guess what the article describes is a bit like "pre-training" a full network with 'synthetic' experiences that quickly tune it for later. In that sense, I think its like these "dreams" are a 'distilled dataset' [1]. The question then is: how is this dataset being passed on from mother/father to child? [1] https://arxiv.org/abs/1811.10959 https://arxiv.org/abs/1811.10959 Abstract: > Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.
- UnFleshedOne 5y ago
  > Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right I think it is the other way around -- adult brain is a physically pruned version of baby brain. We start with lots of connections from everywhere to wherever and then lose a lot of them as we learn, while strengthening useful ones.
optimalsolver 5y ago
How hard would it be to find random networks that don't need to be trained at all?
- AnIdiotOnTheNet 5y ago
  Depends on the size of the network I would guess. A Boltzmann brain would apparently take about 10^10^50 years to form from quantum fluctuation in a vacuum. Our hypothetical neural network would be less complicated than that, assuming we're not aiming for general AI, so I think we can safely take that as an upper bound...
  - CaptainNegative 5y ago
    With a uniform initialization, presumably you'd never need much more than 2^(network bit complexity) number of samples to come across an ideally-initialized network with decent probability, so that's 2^10^24 for anything that fits in a yottabyte.
    - freemint 5y ago
      In the absence of dynamics and uniform random mutation, yes. But Both don't hold in the universe.
- belgian_guy 5y ago
  Surprisingly, it's quite doable in practice. See e.g. https://arxiv.org/abs/1911.13299 https://arxiv.org/abs/1911.13299.
voldacar 5y ago
"Neural networks" aren't very similar to brains, so I would doubt it
kbenson 5y ago
So, that sounds like a good reason to iterate over a bunch of random initializations and train them up to find those awesome resulting networks that are smaller but have the same accuracy (if I'm understanding that correctly and it's actually possible to isolate or take advantage of those subnetworks for a final smaller size). Or... has anyone tries training a network for determining the initialization state of another network? And now I'm wondering if those good starting initialization values only work well for a specific task or if they seem to span all or a subset of tasks, and maybe there's some inherent quality in how the values relate that we can tease out...
stared 5y ago
I think it is less relevant to Winning Ticket Hypothesis (in my opinion, the most important paper on deep learning theory in the last 5 years), but about architectures that work to some extend with random weights. See "Weight Agnostic Neural Networks" (https://weightagnostic.github.io/ https://weightagnostic.github.io/): > Inspired by precocial species in biology, we set out to search for neural net architectures that can already (sort of) perform various tasks even when they use random weight values.
dexen 5y ago
Reminds me the old anecdote, In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play." Minsky shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So the room will be empty." At that moment, Sussman was enlightened.

by gojomo 5y ago

teawrecks 5y ago

In the context of real life we draw a distinction between thus neonatal dreamlike state and post birth "reality". But in the context of a neural net, I don't think such a distinction is necessary. They both sound like they're training the network. Still interesting to consider the parallels though.

emtel 5y ago

It’s not just mammals. I’ve always been astounded that spiders know how to spin webs. Their mothers don’t teach them how!

gota 5y ago
Or this, which is mind-blowing: https://www.smithsonianmag.com/smart-news/dragonfly-undertakes-epic-multi-generational-migration-each-year-180971190/ https://www.smithsonianmag.com/smart-news/dragonfly-undertak... Insects that have large migration patterns - all of the individuals die before coming back to the initial spot; only their descendants continue the cycle. How?!
- chromanoid 5y ago
  Maybe epigenetics? I mean blank slate theory is off the table anyway... In addition think about how bees can tell harvest locations to each other by dancing: https://en.wikipedia.org/wiki/Waggle_dance https://en.wikipedia.org/wiki/Waggle_dance This dance is probably also something those bees know before being born.

teruakohatu 5y ago

> Are these innate patterns of activity priming mammalian brains in a similar way? A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning? Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright. I have wondered about how true this is. Maybe our brains are just so much more powerful, or capable of much deeper understanding, that we require a much lower initial learning rate. The lottery ticket could explain why some children learn to walk at 6 months, while others are closer to 24 months, with no different in intelligence or motor skills in later life. Edit: As some people below have correctly pointed out, human babies could not walk within an hour for physiological reasons, but nor do they exhibit the basic motor skills, spatial reasoning or image processing required for walking. Many human babies even struggle to feed for the first 24-48 hours or longer. Meanwhile baboon babies can hold onto their mother from birth while the mother climbs trees.

soheil 5y ago
> A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning? Probably neither. How does the human baby "learn" to breath almost immediately after birth? I suspect the answer here is also the same.
rowanG077 5y ago
Interesting. Could this also partly explain child geniuses? In general they are worlds ahead early in life. But they aren't significantly better then the non child genius peers later in life.
taneq 5y ago
> Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints I strongly doubt that human babies are physiologically capable of walking before 6-9 months regardless of their level of brain development. The entire baby is born many months "early" compared with other mammals due to the limitations of the human pelvis.

gota 5y ago

I guess I can see some parallel the way you describe it, but it is not that tight of an analogy IMO. Correct me if I'm wrong but the Lottery Ticket Hypothesis mainly means that, for many of our tasks, we start with needlessly huge networks, and thus a good portion of the learning process is just the network learning to "cope with the noise" from the extraneous connections Hence why the lottery ticket network (being a smaller version of the same initial network) can achieve similar accuracy (but not 'better') in much shorter time - because you took away all the confounding stuff. That kind of 'counterfactual' doesn't really translate to a biological domain, right? Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right? Neither is the fetus brain compared to the baby brain (which is more to the point of the article). So the analogy breaks down there too I guess what the article describes is a bit like "pre-training" a full network with 'synthetic' experiences that quickly tune it for later. In that sense, I think its like these "dreams" are a 'distilled dataset' [1]. The question then is: how is this dataset being passed on from mother/father to child? [1] https://arxiv.org/abs/1811.10959 https://arxiv.org/abs/1811.10959 Abstract: > Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.

UnFleshedOne 5y ago
> Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right I think it is the other way around -- adult brain is a physically pruned version of baby brain. We start with lots of connections from everywhere to wherever and then lose a lot of them as we learn, while strengthening useful ones.

optimalsolver 5y ago

How hard would it be to find random networks that don't need to be trained at all?

AnIdiotOnTheNet 5y ago
Depends on the size of the network I would guess. A Boltzmann brain would apparently take about 10^10^50 years to form from quantum fluctuation in a vacuum. Our hypothetical neural network would be less complicated than that, assuming we're not aiming for general AI, so I think we can safely take that as an upper bound...
- CaptainNegative 5y ago
  With a uniform initialization, presumably you'd never need much more than 2^(network bit complexity) number of samples to come across an ideally-initialized network with decent probability, so that's 2^10^24 for anything that fits in a yottabyte.
  - freemint 5y ago
    In the absence of dynamics and uniform random mutation, yes. But Both don't hold in the universe.
belgian_guy 5y ago
Surprisingly, it's quite doable in practice. See e.g. https://arxiv.org/abs/1911.13299 https://arxiv.org/abs/1911.13299.

voldacar 5y ago

"Neural networks" aren't very similar to brains, so I would doubt it

kbenson 5y ago

So, that sounds like a good reason to iterate over a bunch of random initializations and train them up to find those awesome resulting networks that are smaller but have the same accuracy (if I'm understanding that correctly and it's actually possible to isolate or take advantage of those subnetworks for a final smaller size). Or... has anyone tries training a network for determining the initialization state of another network? And now I'm wondering if those good starting initialization values only work well for a specific task or if they seem to span all or a subset of tasks, and maybe there's some inherent quality in how the values relate that we can tease out...

stared 5y ago

I think it is less relevant to Winning Ticket Hypothesis (in my opinion, the most important paper on deep learning theory in the last 5 years), but about architectures that work to some extend with random weights. See "Weight Agnostic Neural Networks" (https://weightagnostic.github.io/ https://weightagnostic.github.io/): > Inspired by precocial species in biology, we set out to search for neural net architectures that can already (sort of) perform various tasks even when they use random weight values.

dexen 5y ago

Reminds me the old anecdote, In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play." Minsky shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So the room will be empty." At that moment, Sussman was enlightened.