5 ms·

A single neural network with 8 bit weights and 1M parameters has 256^1000000 possibilities by that logic, but we can train it fine with simple evolutionary algo

A single neural network with 8 bit weights and 1M parameters has 256^1000000 possibilities by that logic, but we can train it fine with simple evolutionary algorithms like SPSA (Simultaneous perturbation stochastic approximation) E.g., https://openai.com/blog/evolution-strategies/ https://openai.com/blog/evolution-strategies/ One of the remarkable results is that convergence rate for each parameter is not strongly dependent on the number of parameters.

jcims 5y ago
Sure, but we can't really explain why that works either can we? However, I would suggest that there's a difference in evolution. With the neural networks we train, there is actually a highly engineered process required to create the state in which a network can be trained and inferences be driven through it. The necessary combination of hardware that is able to perform and persist operations on information and the algorithms required to do so in a way to yield this outcome is an extremely complex set of pre-conditions that we wouldn't expect to find in the computing equivalent of a primordial soup. With natural evolution, there is no obvious agency or intent behind it. Who is there to care whether or not life started on Earth, and/or who is driving the laws of nature such that the constructive, generative process of genetic evolution actually 'works' as well as it does? Seemingly nobody. Yet this process is able to create systems that operate on scales that we can only dream of. Look up YouTube videos on ATP Synthase for example. It's a nanomachine in every sense of the word. It uses the proton equivalent of a water wheel to spin a little machine that grabs a molecule of ADP, a molecule of inorganic phosphate, then literally snaps them together with mechanical leverage to make ATP. This little miracle machine that powers most of life on earth was built in literal and figurative darkness...it's so damn small light can't see it, and there was nobody there to appreciate its beauty until we came along billions of years later. Ultimately I'm not surprised natural evolution works, I'm surprised at its speed and efficacy.
- mtqwerty 5y ago
  This is an insightful comment but I think you’re missing one thing in your understanding of natural evolution. Evolution has a “reward function” and it’s survival of the fittest. As reproduction produces different variants of the same organism, some variations help the organism while others do not. Organisms with the helpful mutations will be more likely to pass those onto their offspring. Organisms with detrimental variations will be less likely to pass those variations to their kids. It’s not a precise process like gradient descent but when there are billions (trillions?) of organisms evoking simultaneously and independently, it makes more sense how the complexity of biology has come about.
- mrtnmcc 5y ago
  Actually the most remarkable thing you find is how _little_ structure you need to get evolution algorithms to work. I've thrown together random structures with feedback that are inherently nonlinear and chaotic, and evolutionary algorithms are able to quickly find parameters that stabilize them and optimize for the survival reward. The solutions are incredibly clever and in my case sometimes matched designed that were in PhD theses that took a human decades to design by hand. Similarly the details of the evolutionary 'training' also tend not to matter much, just about any algorithm that prefers better instances (by whatever metric) with a very slightly higher probability will converge after a reasonable number of generations. Exponential processes are always surprising. If you have a trait or parameter than confers only a 1% chance of helping survival, it will have an effect of (1.01)^100 =2.7x after only 100 generation. After 1000 generations the effect is 21,000x.

jcims 5y ago

Sure, but we can't really explain why that works either can we? However, I would suggest that there's a difference in evolution. With the neural networks we train, there is actually a highly engineered process required to create the state in which a network can be trained and inferences be driven through it. The necessary combination of hardware that is able to perform and persist operations on information and the algorithms required to do so in a way to yield this outcome is an extremely complex set of pre-conditions that we wouldn't expect to find in the computing equivalent of a primordial soup. With natural evolution, there is no obvious agency or intent behind it. Who is there to care whether or not life started on Earth, and/or who is driving the laws of nature such that the constructive, generative process of genetic evolution actually 'works' as well as it does? Seemingly nobody. Yet this process is able to create systems that operate on scales that we can only dream of. Look up YouTube videos on ATP Synthase for example. It's a nanomachine in every sense of the word. It uses the proton equivalent of a water wheel to spin a little machine that grabs a molecule of ADP, a molecule of inorganic phosphate, then literally snaps them together with mechanical leverage to make ATP. This little miracle machine that powers most of life on earth was built in literal and figurative darkness...it's so damn small light can't see it, and there was nobody there to appreciate its beauty until we came along billions of years later. Ultimately I'm not surprised natural evolution works, I'm surprised at its speed and efficacy.

mtqwerty 5y ago
This is an insightful comment but I think you’re missing one thing in your understanding of natural evolution. Evolution has a “reward function” and it’s survival of the fittest. As reproduction produces different variants of the same organism, some variations help the organism while others do not. Organisms with the helpful mutations will be more likely to pass those onto their offspring. Organisms with detrimental variations will be less likely to pass those variations to their kids. It’s not a precise process like gradient descent but when there are billions (trillions?) of organisms evoking simultaneously and independently, it makes more sense how the complexity of biology has come about.
mrtnmcc 5y ago
Actually the most remarkable thing you find is how _little_ structure you need to get evolution algorithms to work. I've thrown together random structures with feedback that are inherently nonlinear and chaotic, and evolutionary algorithms are able to quickly find parameters that stabilize them and optimize for the survival reward. The solutions are incredibly clever and in my case sometimes matched designed that were in PhD theses that took a human decades to design by hand. Similarly the details of the evolutionary 'training' also tend not to matter much, just about any algorithm that prefers better instances (by whatever metric) with a very slightly higher probability will converge after a reasonable number of generations. Exponential processes are always surprising. If you have a trait or parameter than confers only a 1% chance of helping survival, it will have an effect of (1.01)^100 =2.7x after only 100 generation. After 1000 generations the effect is 21,000x.