5 ms·

Almost always the training and test set are taken from exactly the same Natural distribution, inclusive of any sampling errors, biases, censoring artifacts, dat

by mlthoughts2018 8y ago

Almost always the training and test set are taken from exactly the same Natural distribution, inclusive of any sampling errors, biases, censoring artifacts, data corruption or loss, etc. So while in some cases the distinction you point out does matter, it’s pretty rare, certainly rare enough that it doesn’t cause issues with generalization error analysis of the majority of experiments. For most domain applications like computer vision tasks, natural language tasks, etc., it would be pedantic and disingenuous to call test set generalization error a mere “estimate” of the real generalization error. Beyond that, in topics like PAC learning you can prove results about optimal algorithms, like SVMs as maximum margin classifiers, that can be proved over fully theoretical generalization error bounds. Overall, except for certain special case applications, I think the comments you’re indicating about generalization turn out not to matter much in practice.

YeGoblynQueenne 8y ago
Well, PAC learning assumes that your training and test datasets are drawn from the same distribution and offers no guarantees if that assumption cannot be made. Obviously, training and test data in cross-validation partitions have a common distribution- but that is the distribution associated to the sampling process, not that of whatever natural process (we assume) generates the data in the real world. So, my point is, once you move from the lab to the real world, any generalisation error (resp. accuracy) you calculated by cross-validation in the lab becomes a mere estimate- because you don't know the real-world distribution and you are not guaranteed anything by PAC learning, anymore. Look- your dataset is not the real world, your sampling method is not the natural process that generates the events you observe and the distribution of data in the real world is not the distribution of data in your data set. If it was- you wouldn't need machine learning, would you? You'd already have a model of the real world process you're interested in. I'm afraid the best you can ever hope to do with generalisation to real-world data is estimate it, and nothing more.
- mlthoughts2018 8y ago
  Several subfields of PAC learning also deal with quantifying error bounds under other assumptions about the KL divergence between your sampling distribution and the true distribution. Variational Bayesian methods also provide results on generalization in these cases. You seem to care a lot that these approaches rely on assumptions, but I find that silly. Further in _most_ specific applications of e.g. CNNs for computer vision tasks or RNNs for time series or sequential prediction tasks, you _do_ control the entire sampling distribution (e.g. user photos with exactly satisfied criteria), and you really can fully and unambiguously guarantee that the training and testing set come from the “true” distribution you care about. And in many other cases the degree to which that assumption fails is so minor that it makes no practical difference.
  - sgt101 8y ago
    Some domains are chaotic, some are very large, some are dynamic. In some domains errors of some types matter much more than other errors. Minor assumption failures can suddenly matter alot. The distinction between a lab result and a field trial has been somewhat lost in ml, there will be hell to pay in the long term...
    - srean 8y ago
      > Minor assumption failures can suddenly matter alot. Indeed. In fact this is nowadays painfully obvious in Fisher's maximum-likelihood based estimators -- they do not have nice 'continuity' properties that will give a graceful degradation. We figured out how to deal with this once the problem was recognized. However, we see the same problem raising its head in some of the deep neural net literature. Tiny variations often does not cause tiny distortions in the output -- thats cause for some legitimate concern.
  - YeGoblynQueenne 8y ago
    >> You seem to care a lot that these approaches rely on assumptions, but I find that silly. Thanks, I'll keep that in mind. >> Further in _most_ specific applications of e.g. CNNs for computer vision tasks or RNNs for time series or sequential prediction tasks, you _do_ control the entire sampling distribution (e.g. user photos with exactly satisfied criteria), and you really can fully and unambiguously guarantee that the training and testing set come from the “true” distribution you care about. One of my pet peeves is the Deep Gaydar Paper from Stanford (https://osf.io/zn79k/ https://osf.io/zn79k/). The authors trained a CNN to identify sexuality (gay or straight) from images of faces. The images were self-submitted to a US dating site and the users also stated their sexuality. The dataset included different numbers of images for each user, but of the users that had at least one image in the dataset (i.e. all users) 50% had stated their sexuality as "gay" and 50% as "straight"- this, for both men and women. The research concluded that it was possible to train an image classifier, a CNN, to identify sexuality from facial features. The authors justified this with a theory about the relation between the hormonal environment in the womb. I hold this up as a perfect example of why this assumption, that you can control the distribution in the real world because you can choose the attributes of a dataset that you wish to work with, is very risky. And I don't mean for the social implications of that reprehensible publication. The researchers chose a uniform distribution for their gay and straight subjects. The paper itself cites a 7% estimate of the chance that a person is "gay"- so what, exactly, did their classifier learn to represent? That "gay features" and "straight features" are equally distributed? In any case, "fully and unambiguously guarantee" is not something that happens outside science fiction. Not in machine learning, not in physics or medicine, not in any subject of knowledge. Guarantees always come with assumptions. If assumptions are violated, or if they were bad assumptions to begin with, guarantees fly out the window in haste.
- srean 8y ago
  You are technically somewhat correct (one of the better forms of correct) but it feels like criticisms made more for the sake of criticism. The assumption that train and test distributions are similar has served us phenomenally well. There are indeed cases where they are violated, I grapple with these violations everyday as a part of my job, but I would still claim that approximate equality of test and train covers a substantial volume of potential applications. When test and train do differ, there are formalisms to deal with that. You could look at transductive learning and transfer learning literature. Want to mention again that you are not wrong, but that for a large section of applications the distinction that you draw attention to does not matter. When it does matter there are theoretical and practical tools, but of course these are nowhere as mature as the conventional ones.
  - YeGoblynQueenne 8y ago
    >> The assumption that train and test distributions are similar has served us phenomenally well. To clarify, when I say "the (entire) dataset" I mean all the data that one has at the start of a research project. What I call the "training set" and the "test set" are partitions of that dataset, used to train and validate a model. The partitions might be chosen once, or the entire dataset may be split randomly in training and test partitions during cross-validation, etc. Now, what I say above is that the distribution of the dataset is known - you can count the instances in it, etc. What is not known is the distribution of data in the real world, where the dataset was sampled from. Most of the time, there is no way to tell whether a dataset is representative of the real world. Accordingly, there is no guarantee that an accurate model of the dataset is an accurate model of the real world. What's more, most of the time you can't measure the performance of a model in the real world- because you don't know the ground truth. Again, you can estimate- but you can't calculate. I think, by "test set" you may mean the real world. In that case, I don't disagree with what you say- assuming that a dataset is representative of the real world has "served us well". But, who are "we" and what is our purpose? If we are talking about machine learning research [1] then yes, that assumption has served us well: we can submit papers showing results on specific datasets and have them accepted, because everyone understands that this is the best anyone can do. And it would be tedious and pointless to accompany every result with a disclaimer about PAC learning assumptions- "this result is only meaningful if we accept that our dataset came from the same distribution as the real world" etc. However, my concern is that all this only serves to obscure the fact that we can't really tell how much real progress we are making, as a field. I am going to side with the Great Satan, Noam Chomsky, on this one- our progress until now can only be considered progress if we accept the definition of progress that we have chosen ourselves, that of beating benchmarks and solving datasets, both chosen completely arbitrarily and with no guarantee that they have anything to do with, again, the real world. Now we can have a big, post-modern fight about what is this "real world" I speak of, if you like :) ___________ [1] I'm on the second year of a machine learning PhD, though I should say it's symbolic, logic-based machine learning i.e. Inductive Logic Programming- which perhaps accounts for my preoccupation with subjects the mainstream of machine learning research today probably finds irrelevant.

YeGoblynQueenne 8y ago

Well, PAC learning assumes that your training and test datasets are drawn from the same distribution and offers no guarantees if that assumption cannot be made. Obviously, training and test data in cross-validation partitions have a common distribution- but that is the distribution associated to the sampling process, not that of whatever natural process (we assume) generates the data in the real world. So, my point is, once you move from the lab to the real world, any generalisation error (resp. accuracy) you calculated by cross-validation in the lab becomes a mere estimate- because you don't know the real-world distribution and you are not guaranteed anything by PAC learning, anymore. Look- your dataset is not the real world, your sampling method is not the natural process that generates the events you observe and the distribution of data in the real world is not the distribution of data in your data set. If it was- you wouldn't need machine learning, would you? You'd already have a model of the real world process you're interested in. I'm afraid the best you can ever hope to do with generalisation to real-world data is estimate it, and nothing more.

mlthoughts2018 8y ago
Several subfields of PAC learning also deal with quantifying error bounds under other assumptions about the KL divergence between your sampling distribution and the true distribution. Variational Bayesian methods also provide results on generalization in these cases. You seem to care a lot that these approaches rely on assumptions, but I find that silly. Further in _most_ specific applications of e.g. CNNs for computer vision tasks or RNNs for time series or sequential prediction tasks, you _do_ control the entire sampling distribution (e.g. user photos with exactly satisfied criteria), and you really can fully and unambiguously guarantee that the training and testing set come from the “true” distribution you care about. And in many other cases the degree to which that assumption fails is so minor that it makes no practical difference.
- sgt101 8y ago
  Some domains are chaotic, some are very large, some are dynamic. In some domains errors of some types matter much more than other errors. Minor assumption failures can suddenly matter alot. The distinction between a lab result and a field trial has been somewhat lost in ml, there will be hell to pay in the long term...
  - srean 8y ago
    > Minor assumption failures can suddenly matter alot. Indeed. In fact this is nowadays painfully obvious in Fisher's maximum-likelihood based estimators -- they do not have nice 'continuity' properties that will give a graceful degradation. We figured out how to deal with this once the problem was recognized. However, we see the same problem raising its head in some of the deep neural net literature. Tiny variations often does not cause tiny distortions in the output -- thats cause for some legitimate concern.
- YeGoblynQueenne 8y ago
  >> You seem to care a lot that these approaches rely on assumptions, but I find that silly. Thanks, I'll keep that in mind. >> Further in _most_ specific applications of e.g. CNNs for computer vision tasks or RNNs for time series or sequential prediction tasks, you _do_ control the entire sampling distribution (e.g. user photos with exactly satisfied criteria), and you really can fully and unambiguously guarantee that the training and testing set come from the “true” distribution you care about. One of my pet peeves is the Deep Gaydar Paper from Stanford (https://osf.io/zn79k/ https://osf.io/zn79k/). The authors trained a CNN to identify sexuality (gay or straight) from images of faces. The images were self-submitted to a US dating site and the users also stated their sexuality. The dataset included different numbers of images for each user, but of the users that had at least one image in the dataset (i.e. all users) 50% had stated their sexuality as "gay" and 50% as "straight"- this, for both men and women. The research concluded that it was possible to train an image classifier, a CNN, to identify sexuality from facial features. The authors justified this with a theory about the relation between the hormonal environment in the womb. I hold this up as a perfect example of why this assumption, that you can control the distribution in the real world because you can choose the attributes of a dataset that you wish to work with, is very risky. And I don't mean for the social implications of that reprehensible publication. The researchers chose a uniform distribution for their gay and straight subjects. The paper itself cites a 7% estimate of the chance that a person is "gay"- so what, exactly, did their classifier learn to represent? That "gay features" and "straight features" are equally distributed? In any case, "fully and unambiguously guarantee" is not something that happens outside science fiction. Not in machine learning, not in physics or medicine, not in any subject of knowledge. Guarantees always come with assumptions. If assumptions are violated, or if they were bad assumptions to begin with, guarantees fly out the window in haste.
srean 8y ago
You are technically somewhat correct (one of the better forms of correct) but it feels like criticisms made more for the sake of criticism. The assumption that train and test distributions are similar has served us phenomenally well. There are indeed cases where they are violated, I grapple with these violations everyday as a part of my job, but I would still claim that approximate equality of test and train covers a substantial volume of potential applications. When test and train do differ, there are formalisms to deal with that. You could look at transductive learning and transfer learning literature. Want to mention again that you are not wrong, but that for a large section of applications the distinction that you draw attention to does not matter. When it does matter there are theoretical and practical tools, but of course these are nowhere as mature as the conventional ones.
- YeGoblynQueenne 8y ago
  >> The assumption that train and test distributions are similar has served us phenomenally well. To clarify, when I say "the (entire) dataset" I mean all the data that one has at the start of a research project. What I call the "training set" and the "test set" are partitions of that dataset, used to train and validate a model. The partitions might be chosen once, or the entire dataset may be split randomly in training and test partitions during cross-validation, etc. Now, what I say above is that the distribution of the dataset is known - you can count the instances in it, etc. What is not known is the distribution of data in the real world, where the dataset was sampled from. Most of the time, there is no way to tell whether a dataset is representative of the real world. Accordingly, there is no guarantee that an accurate model of the dataset is an accurate model of the real world. What's more, most of the time you can't measure the performance of a model in the real world- because you don't know the ground truth. Again, you can estimate- but you can't calculate. I think, by "test set" you may mean the real world. In that case, I don't disagree with what you say- assuming that a dataset is representative of the real world has "served us well". But, who are "we" and what is our purpose? If we are talking about machine learning research [1] then yes, that assumption has served us well: we can submit papers showing results on specific datasets and have them accepted, because everyone understands that this is the best anyone can do. And it would be tedious and pointless to accompany every result with a disclaimer about PAC learning assumptions- "this result is only meaningful if we accept that our dataset came from the same distribution as the real world" etc. However, my concern is that all this only serves to obscure the fact that we can't really tell how much real progress we are making, as a field. I am going to side with the Great Satan, Noam Chomsky, on this one- our progress until now can only be considered progress if we accept the definition of progress that we have chosen ourselves, that of beating benchmarks and solving datasets, both chosen completely arbitrarily and with no guarantee that they have anything to do with, again, the real world. Now we can have a big, post-modern fight about what is this "real world" I speak of, if you like :) ___________ [1] I'm on the second year of a machine learning PhD, though I should say it's symbolic, logic-based machine learning i.e. Inductive Logic Programming- which perhaps accounts for my preoccupation with subjects the mainstream of machine learning research today probably finds irrelevant.