3 ms·
>> A model is now running in production but not producing the same results (or at the same level of accuracy) as what was demonstrated during experimentation… n
by YeGoblynQueenne 8y ago
>> A model is now running in production but not producing the same results (or at
the same level of accuracy) as what was demonstrated during experimentation…
no one knows why
The simplest explanation is that the training sample (meaning the entire
dataset; not just the training partition in cross-validation) was not drawn
from the same distribution as the distribution of the data that is being
processed in production.
Machine learning is guaranteed some performance bounds under PAC learning,
assuming that the training dataset and unseen data, on which the training
model will be used, come from the same distribution. Absent this,
performance cannot be predicted. You might as well classify stuff by throwing
a bunch of dice.
Unfortunately, this assumption, that we're representing the real-world
distribution in our training dataset, cannot be justified as long as we don't
know the ground truth in the real world. Which is most of the time.
Essentially, there's no way to know for sure that a model that has performed
very well in experiments will continue to do so once it's deployed in
production.
- Scene_Cast2 8y agoI agree with the intuition, but not sure that I agree with the takeaways (in particular the lack of knowledge of the ground truth). I mean it is a problem, but there are bigger, more practical mechanisms at work tool. In terms of sample distribution - let's say that you have an online model serving traffic (and the outcomes of that traffic are logged as your training & holdout data). Then, when you train a wildly different model, it may start picking other things to serve that the original model never served - and therefore, there was no training data. This is a pretty fundamental and hard problem. Second, funny thing, but sometimes you can't use the same metric for training as you do for evaluating your model online. I don't want to (can't) get into details, but it's also a pretty fundamental and hard problem. Last, the production traffic always shifts. Using historical data, given how much current models are "reactive" and "compression-like", as opposed to true generalization - they perform worse given fresh traffic that changed slightly from e.g. 5 days ago. If your training data is "day -10 to day -3", your holdout is "day -2 to day 0" - models will likely perform worse on holdout than pure overfit theories would have you assume (mind you, still plenty enough to have a ton of value), but when you launch it on day 1, it will perform worse still - as day 1 is different from days -10 to day -3. I haven't done the analysis, but I'd assume that for non-historical models where you don't need to structure your holdout data to be from the "future" - that they'd perform better when you first launch them online.
- YeGoblynQueenne 8y ago>> In terms of sample distribution - let's say that you have an online model serving traffic (and the outcomes of that traffic are logged as your training & holdout data). Then, when you train a wildly different model, it may start picking other things to serve that the original model never served - and therefore, there was no training data. This is a pretty fundamental and hard problem. I think we're talking about the same thing here, that the real world can be very different than your training sample. Sorry, I think I contacted the jargon flu this week :)