Fitting a binary classifier on failure doesn't seem quite right to me. There are some quite natural questions you can't answer unless you can predict some stat

Fitting a binary classifier on failure doesn't seem quite right to me. There are some quite natural questions you can't answer unless you can predict some stats on when failure will occur, and if you have any drift in the dataset, you might just find features correlated with age (and hence having more time to fail) that don't inherently increase risk. I would recommend: (a) using survival analysis https://lifelines.readthedocs.io/en/latest/ https://lifelines.readthedocs.io/en/latest/ https://xgboost.readthedocs.io/en/latest/tutorials/aft_survival_analysis.html https://xgboost.readthedocs.io/en/latest/tutorials/aft_survi... or (b) flattening the data to be one observation per time window, and predicting failure over specific time windows. This is a good talk also. https://dataorigami.net/blogs/napkin-folding/18308867-video-presentation-on-lifelines-survival-analysis-in-python-sept-23-2014 https://dataorigami.net/blogs/napkin-folding/18308867-video-... Although I'm not sure about this case, in general the approach in the article (tree-ensemble predicting binary classier) works well for many business problems!