Evolution strategies (essentially the same as genetic algorithms) have been successfully applied to GPU-training of neural networks and parallelise very well. L

Evolution strategies (essentially the same as genetic algorithms) have been successfully applied to GPU-training of neural networks and parallelise very well. Look at various papers on neuro-evolution (e.g. the ones from the now defunct Uber AI Labs). A simple, very inefficient implementation would be: do inference on n randomly initialised neural networks, calculate loss, select the m best performing and copy them to fill all n spots, add gaussian noise to each weight and bias of every individual, repeat. More efficient mutation and selection strategies exist, but the principle is similar and parallelisation on GPUs is trivial and actually easier than current approaches (atomic operations e.g. compare-and-swap can be used to avoid going back to the CPU for selection). I believe the real reason for neuro-evolution's unpopularity is that for most problems, gradient descent is just faster and more efficient. What would be interesting might be to combine both approaches, using evolution strategies on hyperparameter search, although I haven't read the literature on that front.