I think the premise (neural nets get stuck in local optima) is not trivial, and there has been a lot of research about non-convex optimisation showing that this

I think the premise (neural nets get stuck in local optima) is not trivial, and there has been a lot of research about non-convex optimisation showing that this is not much of an issue. I am not a researcher, but this answer [0] points to this research. I would also say that there are multiple ways to escape local optima (setting a larger learning rate, multiple random initialisations, ensembling). https://www.quora.com/How-come-neural-networks-dont-get-stuck-in-poor-local-optima-Why-are-there-so-many-high-quality-local-optima https://www.quora.com/How-come-neural-networks-dont-get-stuc...