Very early on, people experimented heavily with non-gradient based optimization (genetic algorithms and the like) to optimize neural networks, but the results w

Very early on, people experimented heavily with non-gradient based optimization (genetic algorithms and the like) to optimize neural networks, but the results were rather poor. For very large parameter spaces, gradient based optimization is really the only game in town. There's also lots of clever tricks (gradient clipping, momentum, adagrad, etc) that are involved in optimizing the convergence.