2 ms·
If you really don't know the loss function and are not willing to guess at it, classification is hopeless. If you estimate there's a 70% chance that instance A
by mikebenfield 9y ago
If you really don't know the loss function and are not willing to guess at it, classification is hopeless. If you estimate there's a 70% chance that instance A belongs to category X, there is literally no way to decide whether to classify instance A as X or not.
Anyway, the point of Brier score is that it evaluates your probability estimation (without the loss function), so this is no objection.
- pps43 9y agoBut in this example the banks are not really doing the classification. They're trying to figure whether it's more profitable to approve or to decline a credit application. That decision depends not only on probability of default (that risk score predicts), but also on other factors such as APR and type of product. There's another model with some P&L assumptions for that, and turtles all the way down. Besides, banks typically adjust their credit policy a lot more often than credit bureaus update their scorecards, hence scorecard developers cannot really rely on the fast changing loss function.