4 ms·

> I have done enough research on production ML now to know that it pays to simply overfit to the most recent data and constantly retrain. > It puzzles me when

by sfifs 4y ago

> I have done enough research on production ML now to know that it pays to simply overfit to the most recent data and constantly retrain. > It puzzles me when people say that small companies can’t retrain every day because they don’t have FAANG-style budgets. It costs a few dollars, at best, to retrain many xgboost or scikit-learn models. if re-training every-day actually gives you a significant business material benefit (excepting cases where you specifically want to significantly over-weight recent data - phone keyboard autocomplete predictions as an example), you likely don't have a model that is actually picking signal from noise or generalizing on the data. This is memorization, not learning. This is essentially how you get expensive ML disasters.

jwmoz 4y ago
Sounds like max overfit.
auxym 4y ago
To be fair, that quote does say "it pays to overfit", which is the inverse of "picking signal from noise". I'm not convinced it actually pays, though.
InefficientRed 4y ago
The back half of the essay was unrecognizable for me, but that's probably because I'm working in different contexts and on different product types. It would be helpful to know what systems the author has worked on and why these findings applied to that context. That way, if I ever find myself in one of those contexts, I can leverage the author's experience. Generalizing individual experiences to general prognostications sells well as a genre, especially in software. This writing pattern was epitomized by OOP thought leaders in the 90s/00s, but was also present in the process evangelists who followed and the more iconoclastic folks like Dijkstra who preceded. However, actually following such general advice without taking into account the author's context can easily end in tears. Ideally, these sorts of pieces should be written up as case studies rather than as general prognostications. With the context that a case study provides, these pieces can teach us a lot about how to engineer systems. Without that context, the reader is left to blindly follow advice that may not be relevant to their context. Or, best case, reverse engineer the context.
- fleetwoodsnack 4y ago
  It’s a blog, my guy.
  - InefficientRed 4y ago
    It's a comment, my guy.
vandreas2 4y ago
What is the difference between memorization and learning? Could you please elaborate on this? It always seemed to me that a lot of learning is in fact memorization otherwise you wouldn't need a large dataset of cars photos from every angle (or some angles so that ML can work out in the in-between poses, no amount of 'learning' of photos from the front can work out what a car looks like from the side) to be able to recognise them. Also in what context would you get expensive ML disasters? If you keep retraining on cars as new car models come out then you get 100% recognition memorization notwithstanding, which in the end is what you would want.
- atty 4y ago
  Learning in this sense means it’s able to extrapolate to unseen data (I.e., it’s learned correlations that are inherent to the problem, not just inherent in the training data). Memorizing, on the other hand, implies that it will do very well on any examples in the domain of the training data, and will break (sometimes pretty hilariously) on unseen data. The fact that refitting once a day improves real world performance actually makes me think that the problem/data they work on is highly non-stationary, not that the model is memorizing. If it was purely memorizing then the model would perform poorly on all non-training data, and would not work for even one day. To your point about large datasets - the large datasets are what allow learning to take place. With the most common forms of models we have now, they will memorize when they only have a few examples, and only “learn” when the training data is large enough. There is work to improve learning from a handful of examples, but in many of these cases they require a model that was already trained on the domain in question, and then are specialized to a specific use-case.
  - vandreas2 4y ago
    OK I think I got what you're saying, is that memorizing means that it will perform only on data it has seen in training that is exactly identical to the current one, and it will fail on even slight variations. My question then becomes what's the boundary of that extrapolation in order for it to be considered learning vs memorization? For example to come back to my earlier example, let's say a model is trained on photos of cars taken from only some angles, then it should be able to extrapolate to the intermediate ones without having seen the exact same photo. A large dataset would ensure that it has enough angles to be able to do that right? But there is no amount of photos that will make it identify a car from an underneath photo if it hasn't already seen one from underneath I wouldn't think. So this is the limit of 'learning' as opposed to 'memorization'?
Der_Einzige 4y ago
No, this is because "online/incremental" learning doesn't work due to a wide variety of reasons, most notably the phenomenon of "Catestrophic Forgetting". Ideally, we would simply never stop training.

by sfifs 4y ago

jwmoz 4y ago

Sounds like max overfit.

auxym 4y ago

To be fair, that quote does say "it pays to overfit", which is the inverse of "picking signal from noise". I'm not convinced it actually pays, though.

InefficientRed 4y ago

The back half of the essay was unrecognizable for me, but that's probably because I'm working in different contexts and on different product types. It would be helpful to know what systems the author has worked on and why these findings applied to that context. That way, if I ever find myself in one of those contexts, I can leverage the author's experience. Generalizing individual experiences to general prognostications sells well as a genre, especially in software. This writing pattern was epitomized by OOP thought leaders in the 90s/00s, but was also present in the process evangelists who followed and the more iconoclastic folks like Dijkstra who preceded. However, actually following such general advice without taking into account the author's context can easily end in tears. Ideally, these sorts of pieces should be written up as case studies rather than as general prognostications. With the context that a case study provides, these pieces can teach us a lot about how to engineer systems. Without that context, the reader is left to blindly follow advice that may not be relevant to their context. Or, best case, reverse engineer the context.

fleetwoodsnack 4y ago
It’s a blog, my guy.
- InefficientRed 4y ago
  It's a comment, my guy.

vandreas2 4y ago

What is the difference between memorization and learning? Could you please elaborate on this? It always seemed to me that a lot of learning is in fact memorization otherwise you wouldn't need a large dataset of cars photos from every angle (or some angles so that ML can work out in the in-between poses, no amount of 'learning' of photos from the front can work out what a car looks like from the side) to be able to recognise them. Also in what context would you get expensive ML disasters? If you keep retraining on cars as new car models come out then you get 100% recognition memorization notwithstanding, which in the end is what you would want.

atty 4y ago
Learning in this sense means it’s able to extrapolate to unseen data (I.e., it’s learned correlations that are inherent to the problem, not just inherent in the training data). Memorizing, on the other hand, implies that it will do very well on any examples in the domain of the training data, and will break (sometimes pretty hilariously) on unseen data. The fact that refitting once a day improves real world performance actually makes me think that the problem/data they work on is highly non-stationary, not that the model is memorizing. If it was purely memorizing then the model would perform poorly on all non-training data, and would not work for even one day. To your point about large datasets - the large datasets are what allow learning to take place. With the most common forms of models we have now, they will memorize when they only have a few examples, and only “learn” when the training data is large enough. There is work to improve learning from a handful of examples, but in many of these cases they require a model that was already trained on the domain in question, and then are specialized to a specific use-case.
- vandreas2 4y ago
  OK I think I got what you're saying, is that memorizing means that it will perform only on data it has seen in training that is exactly identical to the current one, and it will fail on even slight variations. My question then becomes what's the boundary of that extrapolation in order for it to be considered learning vs memorization? For example to come back to my earlier example, let's say a model is trained on photos of cars taken from only some angles, then it should be able to extrapolate to the intermediate ones without having seen the exact same photo. A large dataset would ensure that it has enough angles to be able to do that right? But there is no amount of photos that will make it identify a car from an underneath photo if it hasn't already seen one from underneath I wouldn't think. So this is the limit of 'learning' as opposed to 'memorization'?

Der_Einzige 4y ago

No, this is because "online/incremental" learning doesn't work due to a wide variety of reasons, most notably the phenomenon of "Catestrophic Forgetting". Ideally, we would simply never stop training.