3 ms·

I'm a non-expert here but it seems intuitive to me that directly moving towards an improved model via linear regression is more efficient than randomly changing

by Aicy 3y ago

I'm a non-expert here but it seems intuitive to me that directly moving towards an improved model via linear regression is more efficient than randomly changing your model and then running a natural selection simulation to improve fitness.

jacquesm 3y ago
> directly moving towards an improved model via linear regression is more efficient than randomly changing your model and then running a natural selection simulation to improve fitness. That is objectively true, but don't underestimate how much of that process is simulated by the way we train our models. The natural selection bit never was natural to begin with (it's obviously artificial), and is the rough equivalent of the final step in training a model: verification on unseen data. If the model performs worse compared to a previous one then it is discarded! Evolutionary algorithms are somewhat interesting because they can come up with weird stuff that works anyway, that random element can result in entirely novel approaches (to the point that we have a hard time to understand what is going on) and that's something that I have not seen with neural nets. There are some interesting hybrids: https://www.sciencedirect.com/science/article/abs/pii/S095219760000021X https://www.sciencedirect.com/science/article/abs/pii/S09521...
dadoomer 3y ago
This a bit of a false dichotomy: genetic algorithms need not forego of directed or local search (for example, memetic and hybrid algorithms mix gradient search with populations). Also, you have to keep in mind, it is often very hard to frame problems in a way that makes linear regression or gradient descent practical.
MagicMoonlight 3y ago
But how do you know you’re in the right area? You could be linearly improving a really terrible solution, whereas random chance could take you somewhere completely new and better.
- mcv 3y ago
  I have no research to back this up, but I'd expect that the more unknown and unpredictable your problem and search space are, the bigger the benefit of an evolutionary approach. But it's never going to be efficient; it's inherently incredibly inefficient. It only really makes sense when no other method will work.
deterministic 3y ago
Also not an expert but I can imagine that models with a lot of binary and/or non-linear variables will work much better with genetic algorithms. In other words, genetic algorithms are strictly more powerful than linear regression.

by Aicy 3y ago

jacquesm 3y ago

> directly moving towards an improved model via linear regression is more efficient than randomly changing your model and then running a natural selection simulation to improve fitness. That is objectively true, but don't underestimate how much of that process is simulated by the way we train our models. The natural selection bit never was natural to begin with (it's obviously artificial), and is the rough equivalent of the final step in training a model: verification on unseen data. If the model performs worse compared to a previous one then it is discarded! Evolutionary algorithms are somewhat interesting because they can come up with weird stuff that works anyway, that random element can result in entirely novel approaches (to the point that we have a hard time to understand what is going on) and that's something that I have not seen with neural nets. There are some interesting hybrids: https://www.sciencedirect.com/science/article/abs/pii/S095219760000021X https://www.sciencedirect.com/science/article/abs/pii/S09521...

dadoomer 3y ago

This a bit of a false dichotomy: genetic algorithms need not forego of directed or local search (for example, memetic and hybrid algorithms mix gradient search with populations). Also, you have to keep in mind, it is often very hard to frame problems in a way that makes linear regression or gradient descent practical.

MagicMoonlight 3y ago

But how do you know you’re in the right area? You could be linearly improving a really terrible solution, whereas random chance could take you somewhere completely new and better.

mcv 3y ago
I have no research to back this up, but I'd expect that the more unknown and unpredictable your problem and search space are, the bigger the benefit of an evolutionary approach. But it's never going to be efficient; it's inherently incredibly inefficient. It only really makes sense when no other method will work.

deterministic 3y ago

Also not an expert but I can imagine that models with a lot of binary and/or non-linear variables will work much better with genetic algorithms. In other words, genetic algorithms are strictly more powerful than linear regression.