4 ms·

I came to the conclusion that area under ROC is pretty much garbage. What you really want is two separate steps: estimate the probability that instance A belong

I came to the conclusion that area under ROC is pretty much garbage. What you really want is two separate steps: estimate the probability that instance A belongs in classification X, and then a decision step where you decide how to classify A based on a loss function (this varies depending on how harmful a false positive is). Area under ROC forces you to conflate these two. Why not evaluate the probability estimation directly, using Brier score [1] or something similar? [1] https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score

pps43 9y ago
Unfortunately you do not usually know the loss function when developing a model. A typical example would be credit bureau developing something like FICO score for the banks to use. Banks might know the loss function, but the credit bureaus don't. Hence the need to use a metric like KS or Gini coefficient.
- mikebenfield 9y ago
  If you really don't know the loss function and are not willing to guess at it, classification is hopeless. If you estimate there's a 70% chance that instance A belongs to category X, there is literally no way to decide whether to classify instance A as X or not. Anyway, the point of Brier score is that it evaluates your probability estimation (without the loss function), so this is no objection.
  - pps43 9y ago
    But in this example the banks are not really doing the classification. They're trying to figure whether it's more profitable to approve or to decline a credit application. That decision depends not only on probability of default (that risk score predicts), but also on other factors such as APR and type of product. There's another model with some P&L assumptions for that, and turtles all the way down. Besides, banks typically adjust their credit policy a lot more often than credit bureaus update their scorecards, hence scorecard developers cannot really rely on the fast changing loss function.
rm999 9y ago
>I came to the conclusion that area under ROC is pretty much garbage. I cannot disagree more. IMO AUC is a rare great metric: it's principled, useful, universally applicable (e.g. invariant to class imbalances), and easy to explain adequately to non-statistician ("probability of choosing a positive sample over a negative one"). >What you really want is two separate steps: estimate the probability that instance A belongs in classification X, and then a decision step where you decide how to classify A based on a loss function (this varies depending on how harmful a false positive is). Yes, but those are two inherently separate steps and should be measured separately. AUC is a metric of the first step (the "model"). The second step is a business decision and will often be made separately from the modeling process, by different people, with a different cadence, and with a different goal in mind. For example if I am designing a model to find a disease, I just want to make the best prediction I can, which is cleanly measured by AUC. Then, when it comes to actual diagnosis, someone else will choose cutoffs based on various factors like false positive costs (treatment cost, human toll), false negative cost (disease damage, death toll), supply of treatment, etc. I can picture scenarios where the same model is used for decades, but the cutoffs change seasonally.
- mikebenfield 9y ago
  > invariant to class imbalances I think this is a red herring that comes from not thinking probabilistically. If the distribution of your training data does not resemble the distribution of your real-world data (or cannot be made to resemble it), you're just guessing anyway. If it does resemble the real distribution, then you want those class imbalances. In fact, I tend to think that the fact that ROC is "invariant to class imbalances" is a significant downside: it means in some sense your score is just as sensitive to things that rarely happen as it is to things that happen all the time. > "probability of choosing a positive sample over a negative one" I find the practical implications of this pretty opaque, and it's never been clear to me whether this is measuring anything I actually care about. As far as I know there aren't theoretical guarantees that a better AUC score means anything real. I haven't thought deeply about it, but I am reasonably sure I could find some simple examples illustrating how to "cheat" AUC by getting a higher score with predictions that are worse in any practical sense. I still like the Brier score: just give me a number indicating how well my estimated probability predictions do on a test/validation set. There are even theoretical guarantees about it, because it's a proper score function.

pps43 9y ago

Unfortunately you do not usually know the loss function when developing a model. A typical example would be credit bureau developing something like FICO score for the banks to use. Banks might know the loss function, but the credit bureaus don't. Hence the need to use a metric like KS or Gini coefficient.

mikebenfield 9y ago
If you really don't know the loss function and are not willing to guess at it, classification is hopeless. If you estimate there's a 70% chance that instance A belongs to category X, there is literally no way to decide whether to classify instance A as X or not. Anyway, the point of Brier score is that it evaluates your probability estimation (without the loss function), so this is no objection.
- pps43 9y ago
  But in this example the banks are not really doing the classification. They're trying to figure whether it's more profitable to approve or to decline a credit application. That decision depends not only on probability of default (that risk score predicts), but also on other factors such as APR and type of product. There's another model with some P&L assumptions for that, and turtles all the way down. Besides, banks typically adjust their credit policy a lot more often than credit bureaus update their scorecards, hence scorecard developers cannot really rely on the fast changing loss function.

rm999 9y ago

>I came to the conclusion that area under ROC is pretty much garbage. I cannot disagree more. IMO AUC is a rare great metric: it's principled, useful, universally applicable (e.g. invariant to class imbalances), and easy to explain adequately to non-statistician ("probability of choosing a positive sample over a negative one"). >What you really want is two separate steps: estimate the probability that instance A belongs in classification X, and then a decision step where you decide how to classify A based on a loss function (this varies depending on how harmful a false positive is). Yes, but those are two inherently separate steps and should be measured separately. AUC is a metric of the first step (the "model"). The second step is a business decision and will often be made separately from the modeling process, by different people, with a different cadence, and with a different goal in mind. For example if I am designing a model to find a disease, I just want to make the best prediction I can, which is cleanly measured by AUC. Then, when it comes to actual diagnosis, someone else will choose cutoffs based on various factors like false positive costs (treatment cost, human toll), false negative cost (disease damage, death toll), supply of treatment, etc. I can picture scenarios where the same model is used for decades, but the cutoffs change seasonally.

mikebenfield 9y ago
> invariant to class imbalances I think this is a red herring that comes from not thinking probabilistically. If the distribution of your training data does not resemble the distribution of your real-world data (or cannot be made to resemble it), you're just guessing anyway. If it does resemble the real distribution, then you want those class imbalances. In fact, I tend to think that the fact that ROC is "invariant to class imbalances" is a significant downside: it means in some sense your score is just as sensitive to things that rarely happen as it is to things that happen all the time. > "probability of choosing a positive sample over a negative one" I find the practical implications of this pretty opaque, and it's never been clear to me whether this is measuring anything I actually care about. As far as I know there aren't theoretical guarantees that a better AUC score means anything real. I haven't thought deeply about it, but I am reasonably sure I could find some simple examples illustrating how to "cheat" AUC by getting a higher score with predictions that are worse in any practical sense. I still like the Brier score: just give me a number indicating how well my estimated probability predictions do on a test/validation set. There are even theoretical guarantees about it, because it's a proper score function.