This is disingenuous. A prediction is a prediction. There is a world of difference between 51% certain and 99% certain. People excited that their data science m

This is disingenuous. A prediction is a prediction. There is a world of difference between 51% certain and 99% certain. People excited that their data science models get good predictions on their test set without considering, say, the RoC curve are setting themselves up for failure. If a model says 75% chance of X rather than Y, that doesn’t mean you should expect X. Depending on context, you might be better off flagging it as unclear or even safer to conclude a possible Y. Testing calibration of many models across many different trials as a test of data scientist competence is a little strange, but if you want to answer the question of how valuable the predicted likelihood’s are, it’s a decent measure.