3 ms·

Original post author here. Thanks for taking the time to write this---it made me realize that I should have discussed the nature of tasks I was working with in

Original post author here. Thanks for taking the time to write this---it made me realize that I should have discussed the nature of tasks I was working with in my post, especially because different tasks/datasets have different behaviors under the same model choices. The overfitting phenomenon I've seen at the companies I work/have worked with are all doing some kind of ranking. None use auc to evaluate---in fact, I don't see many production models evaluated with auc because usually some threshold is fixed before deployment, or the client has their own threshold they will act on. Re: hparamaters varying the metric---a point on the ROC can change a lot while the auc stays relatively the same, especially in class imbalance settings. Re data validation---at large companies with 10s-100s of models with hundreds of thousands of features, often data gets corrupted. Alarm bells do go off but there is alarm fatigue. Furthermore, not all Task MLEs are Platform MLEs and vice versa...a lot of people have to build tools to monitor ML pipelines they don't own. Anyways I really appreciate your comment & it made me realize how ML people have vastly different experiences. I will clarify assumptions on the task & evaluation next time I write :)

spywaregorilla 4y ago
Hey, cool to get a response. I'm never in the space of creating tens, let alone hundreds of different models. I'm curious what this means > usually some threshold is fixed before deployment, or the client has their own threshold they will act on. Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to any of the typical confusion matrix outcomes you see in the model validation. I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes
- shreya_shankar 4y ago
  > Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to any of the typical confusion matrix outcomes you see in the model validation. Once the model makes predictions, they need to be surfaced to a client/user. For example, I once worked on ML pipelines for car companies that predicted which parts of vehicles would fail & when. Every day, we generated probability of part failure for each part & vehicle. First we decided to give the car companies the probability of failure for each part, but they didn't know how to act on that (like does 60% engine failure in the next month mean I should get the engine replaced or not?). At the end of the day, what was evaluated was: how much $ the car company saved in performing a car part recall, which was a function of the binary decision whether a client replaced the car part or not. So this is what I mean by choosing a threshold before deployment (i.e., tell the car company all the cars & parts that should be fixed, > 50% or some threshold failure) or leave the thresholding to the client (i.e., give them the probabilities and see what they do). > I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes Yeah, a lot of ads, content recommendation, personalization, fraud detection, etc tasks have frequent retraining. It feels uncomfortable for sure. When I was an MLE I would do all sorts of backtesting, but it felt kind of like voodoo magic because even if I validated a change to the model on old data, who is to say the change will positively impact future predictions (arguably all that matters to the business)?
  - spywaregorilla 4y ago
    > who is to say the change will positively impact future predictions (arguably all that matters to the business)? My point is that you are the person who can do this. Not up front, but you have the ability to look at your past prediction of future outcomes and see if what you're doing is actually working or not. You already had the predictions. You now know the actual outcomes. How does the performance metrics of your model's predictions from back then on your validation set compare to the performance of your models when they were used for real? How would the prior day's model do. Or the prior year's?

spywaregorilla 4y ago

Hey, cool to get a response. I'm never in the space of creating tens, let alone hundreds of different models. I'm curious what this means > usually some threshold is fixed before deployment, or the client has their own threshold they will act on. Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to any of the typical confusion matrix outcomes you see in the model validation. I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes

shreya_shankar 4y ago
> Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to any of the typical confusion matrix outcomes you see in the model validation. Once the model makes predictions, they need to be surfaced to a client/user. For example, I once worked on ML pipelines for car companies that predicted which parts of vehicles would fail & when. Every day, we generated probability of part failure for each part & vehicle. First we decided to give the car companies the probability of failure for each part, but they didn't know how to act on that (like does 60% engine failure in the next month mean I should get the engine replaced or not?). At the end of the day, what was evaluated was: how much $ the car company saved in performing a car part recall, which was a function of the binary decision whether a client replaced the car part or not. So this is what I mean by choosing a threshold before deployment (i.e., tell the car company all the cars & parts that should be fixed, > 50% or some threshold failure) or leave the thresholding to the client (i.e., give them the probabilities and see what they do). > I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes Yeah, a lot of ads, content recommendation, personalization, fraud detection, etc tasks have frequent retraining. It feels uncomfortable for sure. When I was an MLE I would do all sorts of backtesting, but it felt kind of like voodoo magic because even if I validated a change to the model on old data, who is to say the change will positively impact future predictions (arguably all that matters to the business)?
- spywaregorilla 4y ago
  > who is to say the change will positively impact future predictions (arguably all that matters to the business)? My point is that you are the person who can do this. Not up front, but you have the ability to look at your past prediction of future outcomes and see if what you're doing is actually working or not. You already had the predictions. You now know the actual outcomes. How does the performance metrics of your model's predictions from back then on your validation set compare to the performance of your models when they were used for real? How would the prior day's model do. Or the prior year's?