There's a spectrum of generalization-memorization. The extreme case of memorization is that it would only correctly classify images with precisely identical pix

There's a spectrum of generalization-memorization. The extreme case of memorization is that it would only correctly classify images with precisely identical pixel values to the ones in the training set. When we say that people have "memorized" something, we are still far more general than that. We might key off a few words, say. The behavior of highly overparameterized networks is basically what you suggest: they will memorize if needed (which you can test by randomizing labels) but will usually generalize better when possible.