Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do

Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction? If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.