3 ms·

I’ve had the opposite experience. After finishing grad school and working in industry for a long time now, applying ML to computer vision, NLP, causal inference

by mlthoughts2018 8y ago

I’ve had the opposite experience. After finishing grad school and working in industry for a long time now, applying ML to computer vision, NLP, causal inference and quant finance for production systems, the vast, vast majority of problems are such that the training and test set are defined by very precise business operating constraints and it’s known with a high degree of certainty that future examples to which the model is required to generalize will also be from the same distribution reflected in model training, and any differences are so minor as to be practically ignorable all the time, and requiring really special evidence to show the super rare cases when it cannot be ignored. (Those cases also are most often just solved by collecting new/ more training & testing data and re-training / fine-tuning / calibrating the model to some slight shift in the distribution, and require no formal modeling techniques related to tricky generalization distribution differences). This has been true across commercial ML products or services at large tech companies, small startups, quant finance firms, defense research labs, and academic labs. Across projects ranging from face recognition, high-traffic search engine and related services like machine translation, trending models, search ranking, dimensionality reduction models in time series forecasting, reverse image search, audio similarity models, advertising models, and many more. It just turns out that in industry, the problems that have meaningful utility to the business are almost always defined by intrinsic constraints that nearly guarantee there won’t be a meaningful difference in the distribution of future prediction-time examples compared with training & testing distribution.

YeGoblynQueenne 8y ago
I find all this very hard to square with my understanding. I mean, what kind of "business operating constraints" stay the same over months or years? Or do you mean that, when constraints change, you just train a new model? I think basically what you're saying is "we choose the data we want to reprsent, so we know when we're representing it well". But that's just what I say above, about measuring progress by hitting the targets you choose, completely arbitrarily. It's as if I shot a bunch of arrows and then painted targets around them, then claimed I'm a great archer because I always hit bullseye. I don't think that's the result of some fundamental difference in understanding between academia and industry, either. Perhaps, in the industry, assuming that reality will comply with your assumptions helps achieve business aims, but that's not because the systems work. It's becaus they can be made to look like they do and there's no good way to show they don't, as there's no good way to show they do.
- mlthoughts2018 8y ago
  It sounds like you haven’t worked on many real world applied problems.
srean 8y ago
I would add that I have had my fair share of industry scale problems where labeled data is not drawn from the same distribution as where the system will be deployed. But as you pointed out earlier, there are theoretical and practical methods to deal with this. For someone who wants to get familiar with this territory, domain adaptation, transfer learning, transductive learning, semi-supervised learning, etc would be some of the keywords to seed an initial literature search with.

YeGoblynQueenne 8y ago

I find all this very hard to square with my understanding. I mean, what kind of "business operating constraints" stay the same over months or years? Or do you mean that, when constraints change, you just train a new model? I think basically what you're saying is "we choose the data we want to reprsent, so we know when we're representing it well". But that's just what I say above, about measuring progress by hitting the targets you choose, completely arbitrarily. It's as if I shot a bunch of arrows and then painted targets around them, then claimed I'm a great archer because I always hit bullseye. I don't think that's the result of some fundamental difference in understanding between academia and industry, either. Perhaps, in the industry, assuming that reality will comply with your assumptions helps achieve business aims, but that's not because the systems work. It's becaus they can be made to look like they do and there's no good way to show they don't, as there's no good way to show they do.

mlthoughts2018 8y ago
It sounds like you haven’t worked on many real world applied problems.

srean 8y ago

I would add that I have had my fair share of industry scale problems where labeled data is not drawn from the same distribution as where the system will be deployed. But as you pointed out earlier, there are theoretical and practical methods to deal with this. For someone who wants to get familiar with this territory, domain adaptation, transfer learning, transductive learning, semi-supervised learning, etc would be some of the keywords to seed an initial literature search with.