well, your neural network is just a very complicated - ideally continously differentiable - function. In principle you could just fit this function to data with

well, your neural network is just a very complicated - ideally continously differentiable - function. In principle you could just fit this function to data with nonlinear least squares (e.g. Levenberg-Marquardt), we use backpropagation and SGD, because it's faster. Concerning simulated annealing: have a look at the old Alphafold and check how they optimize their neural networks... So, could you explain again how the concepts and foundations of deep learning differs from plain old regression techniques?