2 ms·

That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given predi

That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction. I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year. I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government. For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply. For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?

yorwba 3y ago
Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction? If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.
harperlee 3y ago
But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.