4 ms·
But it is defined: In the real world, you have a finite dataset, and so have to use it over and over during training. You can shuffle the order, augment it with
by joe_the_user 6y ago
But it is defined: In the real world, you have a finite dataset, and so have to use it over and over during training. You can shuffle the order, augment it with transformations, divide it into batches, but ultimately you reuse the whole dataset (other than the subset withheld for evaluation) in each training epoch.
I have no idea how "you have a finite dataset" defines "real world generalization". This line of reasoning seems completely incoherent.
- webmaven 6y agoSorry, that got scrambled in an edit due to my conflation of "real world generalization" and "real world vs. ideal world training." The result is, indeed, incoherent. Anyway, while "real world vs. ideal world training" is defined, you are correct that generalization is not, at least here. But definitions do exist, and more or less conform to what I wrote. Withholding a subset of training data for validation is meant as a proxy for the entirely new samples that will (presumably) be encountered upon deploying the model, but if the training data is biased or otherwise not representative of the real world data, then it is probable that testing the withheld data won't reveal the problem. In other words, the model generalizes well within the parameters of the training set, but not to the real-world data that has a different distribution, because the training set wasn't representative. There is some interesting discussion of this within Gwern's investigation of the apocryphal 'tank story': https://www.gwern.net/Tanks https://www.gwern.net/Tanks