If it was a smooth map, yes. But neural network error surfaces are messier. Computing the gradient on random mini-batches ends up helping "average things out".

If it was a smooth map, yes. But neural network error surfaces are messier. Computing the gradient on random mini-batches ends up helping "average things out". But adding more layers, different architectures, different regularization techniques, you get very different results than you would get by taking the simpler mini-batch only approach where my expectation would be more of an "averaging" effect along a smooth surface rather than the kind of learning we see from a neural network. It's remarkable because it seems you can get away with ignoring certain assumptions about the smoothness of the error surface and bootstrap predictive models from raw (read: cheaper to generate) data rather than features pre-designed to live in a nice space together. A lot of people didn't think you could do that, and the question of why you can remains unanswered.