3 ms·

Data drift is not a good indicator of model performance degradation

• santiviquez 3y ago
  The world is a dynamic mess. So, it is natural for things to change. And, don't get me wrong, data drift methods are good tools for detecting those changes. But in the context of ML monitoring, data drift methods, which are often presented as the go-to solution for detecting performance degradation in ML models are not the right tool. In my findings, data drift doesn't always imply a decline in the model's performance. Some reasons might be: - The drifted feature may have low importance for the model's predictions, meaning that changes in that specific feature have minimal impact on overall performance. - The model might be able to correctly extrapolate or generalize from the available data, even in the presence of drifting features. - Even if multiple features exhibit drift, the relationship between these features and other relevant features may remain unchanged, resulting in stable model performance. So, because of these limitations, drift methods often generate lots of false alarms. This makes them somewhat noisy solutions for ML monitoring. I'm interested in reading your opinions about it and knowing if any of you have experienced something similar.