4 ms·
Is there a separate word from overfitting to refer to generalization performance in a way that explicitly includes distributional shift? Just curious as I neve
by breatheoften 9y ago
Is there a separate word from overfitting to refer to generalization performance in a way that explicitly includes distributional shift? Just curious as I never realized that overfitting was commonly constrained in the way you describe and would like to go back and relabel my knowledge to better distinguish between your definition of overfitting and a more general version.
- chillee 9y agoGeneralization performance is typically defined in roughly this manner. Assume your training data consists of a sample S of m elements from your full distribution D. Let L(S) be the average loss for your model (hypothesis is also a common term) on S, and L(D) be the average loss for your model on D. Now, we say that training generalizes if your model that minimizes L(S) also achieves similarly low L(D). In practice, we detect generalization by taking a separate sample (validation set) S_2 from D and using L(S_2) to approximate L(D). I'm not familiar with any literature of generalization to arbitrary distributional shift between training and test. If you allow arbitrary distributional shift (say we decide that all dogs are now called cats and all cats are now called dogs), it's pretty clear that you can get arbitrarily poor "generalization".
- breatheoften 9y agoOk — but in many cases there’s more that can be said about the error in the approximation to D of S and S_2 which would allow discussion of various forms of bounded or constrained distributional shift. I don’t think the only choices are “no distributional shift” and “arbitrary distributional shift” are they? What’s the vocabulary to refer to poor generalization performance to datasets with some form of constrained or procedurally introduced distributional shift?