I think this is true, broadly. Biological evolution has a large advantage in that it introduces a large amount of exogenous variation into the system (it just r

I think this is true, broadly. Biological evolution has a large advantage in that it introduces a large amount of exogenous variation into the system (it just randomly changes DNA via a number of ways, and selects the "best" changes). This can be good as it allows you to do a lot more exploration than with a comparable gradient based approach, as the gradient based approach can't explore entirely new parameters; it can only look at parameters that have already been shown to be good. This is why, for instance, dead neurons are a problem with ReLUs. I don't think (but have not proof) that this would be a problem with evolution. I think that evolutionary approaches are a lot better at the exploration phase, but weak at exploitation; gradient methods tend to be the opposite. I think some sort of combination of the two, like Max Jaderberg's "Population based training of neural networks" [1], is the way to go. [1]: https://deepmind.com/blog/population-based-training-neural-networks/ https://deepmind.com/blog/population-based-training-neural-n...