I agree with the philosophical conclusions here (e.g. I take their point on data augmentation to basically be that the name "data augmentation" is well chosen -

I agree with the philosophical conclusions here (e.g. I take their point on data augmentation to basically be that the name "data augmentation" is well chosen - it enlarges the restricted real world). I'm confused by the setup though. The training algorithm seeks to minimize the training loss on, in one case, the CIFAR-10 distribution D, and in the second case on D', their "ideal world" distribution. They find that these two tasks look very similar. I don't see what conclusions you can draw from this except that they did a good job training their generative model and D' is indeed very close to D, and so minimizers for the training loss on D' will look similar to those for D. (Close can mean something like the distributions D|_{label = c} and D|_{label=c'} are close in Earth-mover distance in some appropriate embedding space).