Paraphrasing slightly: > when you synthetically generate data ... you're not materially going to get > out something different than what you put in Well, yes

Paraphrasing slightly: > when you synthetically generate data ... you're not materially going to get > out something different than what you put in Well, yes and no. If I give you five points and tell you that they are actually samples from a normal distribution, you could easily generate lots of samples with the same mean and variance. If you were to then compute the mean and variance of this new and "improved" dataset, you would get results very nearly the same as the mean and variance of the original data. It would be fair to note at this point that the expanded dataset has no information that the original had. On the other hand, if you generate 10,000 synthetic points in the same way and compute the 99th percentile of these points, you will get a more interesting result than if you simply computed the 99th percentile of the original points. This makes it seem that the augmented dataset has better information in it than the original sample. In fact, the augmented dataset is only better because our estimation algorithm is deficient. A better algorithm would look at the mean and variance and find the 99th percentile of the corresponding normal distribution. This improved algorithm would get nearly the same result with the original or the augmented data. In this simple example, we can see and remedy the defect in our estimation algorithm, but with most machine learning methods it is much, much harder to remedy the algorithms' appetite for data. Augmentation is, however, a nice and simple alternative. A more complex example that isn't as trivial can be seen with the problem of finding a linear classifier for sets of points in the plane. If we don't have a lot of samples, we will often have the problem that there are an infinite number of solutions that will perfectly separate the sets in the training data. We would like a better answer, however, that is likely to work well on data we haven't seen yet. We have several choices of learning algorithms. One of the simplest is logistic regression, but that often doesn't converge well if the training set is small. We could add regularization and use fancier algorithms like ridge regression or LASSO or support vector methods to get a better result. OR We could just add samples that are "pretty near" each of our training examples. With a sufficiently sloppy definition of pretty near or with enough added samples, even the simplest algorithm will give us nice results. If the samples we add are normally distributed around the data, we get something like ridge regression. If the distance of the new samples from the originals are exponentially distributed, we get LASSO. If I were stuck on a desert island and had to implement regularization for linear classifiers, data augmentation is a pretty straightforward way to do it. For more complicated models or more complicated data, it rapidly becomes a really good approach.