3 ms·

You are technically somewhat correct (one of the better forms of correct) but it feels like criticisms made more for the sake of criticism. The assumption that

• YeGoblynQueenne 8y ago
  >> The assumption that train and test distributions are similar has served us phenomenally well. To clarify, when I say "the (entire) dataset" I mean all the data that one has at the start of a research project. What I call the "training set" and the "test set" are partitions of that dataset, used to train and validate a model. The partitions might be chosen once, or the entire dataset may be split randomly in training and test partitions during cross-validation, etc. Now, what I say above is that the distribution of the dataset is known - you can count the instances in it, etc. What is not known is the distribution of data in the real world, where the dataset was sampled from. Most of the time, there is no way to tell whether a dataset is representative of the real world. Accordingly, there is no guarantee that an accurate model of the dataset is an accurate model of the real world. What's more, most of the time you can't measure the performance of a model in the real world- because you don't know the ground truth. Again, you can estimate- but you can't calculate. I think, by "test set" you may mean the real world. In that case, I don't disagree with what you say- assuming that a dataset is representative of the real world has "served us well". But, who are "we" and what is our purpose? If we are talking about machine learning research [1] then yes, that assumption has served us well: we can submit papers showing results on specific datasets and have them accepted, because everyone understands that this is the best anyone can do. And it would be tedious and pointless to accompany every result with a disclaimer about PAC learning assumptions- "this result is only meaningful if we accept that our dataset came from the same distribution as the real world" etc. However, my concern is that all this only serves to obscure the fact that we can't really tell how much real progress we are making, as a field. I am going to side with the Great Satan, Noam Chomsky, on this one- our progress until now can only be considered progress if we accept the definition of progress that we have chosen ourselves, that of beating benchmarks and solving datasets, both chosen completely arbitrarily and with no guarantee that they have anything to do with, again, the real world. Now we can have a big, post-modern fight about what is this "real world" I speak of, if you like :) ___________ [1] I'm on the second year of a machine learning PhD, though I should say it's symbolic, logic-based machine learning i.e. Inductive Logic Programming- which perhaps accounts for my preoccupation with subjects the mainstream of machine learning research today probably finds irrelevant.
  • joe_the_user 8y ago
    Thank you Goblyn Queenne for making what I was trying to say actually sound credible. I'd be tempted to even ask whether "probability distribution" could describe the real world. Certain no time-invariant probability distribution could describe this world.
  • srean 8y ago
    > And it would be tedious and pointless to accompany every result with a disclaimer about PAC learning assumptions- "this result is only meaningful if we accept that our dataset came from the same distribution as the real world" etc. That assumption is not necessary, as I mentioned and mlthoughts mentions above, we do have tools to deal with discrepancies between test and train distribution. BTW I want talking about research but meat and potatoes applications. I think it is intellectually lazy to raise questions that even the person asking the question and everyone else knows that there cannot be an answer, especially when we dont define things (example 'real world'). Its easy-peasy to ask for things that are epistemologically impossible. On the other hand working within the limitations of impossibility theorems, under assumptions that seems reasonable, to obtain theoretical results as well as practical algorithms to solve real problems -- now that is a valuable proposition. > [1] I'm on the second year of a machine learning PhD Goes without saying. Those who have been through that, can spot our old enlightened newbie self from a mile away :)
    • YeGoblynQueenne 8y ago
      >> I think it is intellectually lazy to raise questions that even the person asking the question and everyone else knows that there cannot be an answer, especially when we dont define things (example 'real world'). Its easy-peasy to ask for things that are epistemologically impossible. On the other hand working within the limitations of impossibility theorems, under assumptions that seems reasonable, to obtain theoretical results as well as practical algorithms to solve real problems -- now that is a valuable proposition. No, I believe that real progress is definitely possible, "epistemologically". That is how science advances- by solving one "impossible" problem after the other. Of course, that's not how the industry rolls. In industry, if you have something that makes money, you sell it, whether it works or not and you don't bother with silly stuff like theoretical justifications and whatnot. See Waymo's driverless cars (with drivers) and Tesla's Autopilot (that can't auto-pilot) etc. As to what is the real world- "everything that isn't in your data set" will do. My earlier comment was just me being gregarious. And this: >> Goes without saying. Those who have been through that, can spot our old enlightened newbie self from a mile away :) -is distasteful. I was having a conversation and you were having a pissing contest. My bad, I guess. No, I'm not a newbie, you are not me but more experienced and I'm probably older than you. Wrong assumptions. Again.