4 ms·

> Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to

> Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to any of the typical confusion matrix outcomes you see in the model validation. Once the model makes predictions, they need to be surfaced to a client/user. For example, I once worked on ML pipelines for car companies that predicted which parts of vehicles would fail & when. Every day, we generated probability of part failure for each part & vehicle. First we decided to give the car companies the probability of failure for each part, but they didn't know how to act on that (like does 60% engine failure in the next month mean I should get the engine replaced or not?). At the end of the day, what was evaluated was: how much $ the car company saved in performing a car part recall, which was a function of the binary decision whether a client replaced the car part or not. So this is what I mean by choosing a threshold before deployment (i.e., tell the car company all the cars & parts that should be fixed, > 50% or some threshold failure) or leave the thresholding to the client (i.e., give them the probabilities and see what they do). > I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes Yeah, a lot of ads, content recommendation, personalization, fraud detection, etc tasks have frequent retraining. It feels uncomfortable for sure. When I was an MLE I would do all sorts of backtesting, but it felt kind of like voodoo magic because even if I validated a change to the model on old data, who is to say the change will positively impact future predictions (arguably all that matters to the business)?

spywaregorilla 4y ago
> who is to say the change will positively impact future predictions (arguably all that matters to the business)? My point is that you are the person who can do this. Not up front, but you have the ability to look at your past prediction of future outcomes and see if what you're doing is actually working or not. You already had the predictions. You now know the actual outcomes. How does the performance metrics of your model's predictions from back then on your validation set compare to the performance of your models when they were used for real? How would the prior day's model do. Or the prior year's?

spywaregorilla 4y ago

> who is to say the change will positively impact future predictions (arguably all that matters to the business)? My point is that you are the person who can do this. Not up front, but you have the ability to look at your past prediction of future outcomes and see if what you're doing is actually working or not. You already had the predictions. You now know the actual outcomes. How does the performance metrics of your model's predictions from back then on your validation set compare to the performance of your models when they were used for real? How would the prior day's model do. Or the prior year's?