3 ms·

There are countries where more than 90% of court cases result in "guilty" verdict. My AI judge that always says "guilty" has very high accuracy in these jurisdi

by Grue3 10y ago

There are countries where more than 90% of court cases result in "guilty" verdict. My AI judge that always says "guilty" has very high accuracy in these jurisdictions.

libeclipse 10y ago
Therein lies the problem of the accuracy of predictions. If there's three AI judges, one gives a court case a 35% chance of being guilty, the other a 40%, and the last a 45%. The case was a guilty. Were the predictions correct?
- kgwgk 10y ago
  Measuring the accuracy of a single prediction is difficult, but there are reasonable ways to measure the accuracy of a set of predictions. For example, https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score
binalpatel 10y ago
This is the imbalanced class problem that comes up often in the real world. You can't optimize for overall accuracy, otherwise you'd have classifiers that always just predict guilty (or not sick, or not fradulent, and so on) because they'd achieve 95%+ accuracy.
- hammock 10y ago
  So what is the metric to optimize?
  - binalpatel 10y ago
    I use AUC often, and always look at the confusion matrix. Metrics based off of it (recall, precision) are both useful to use as well. In the end the probability cutoffs you choose are more often than not based on business context and costs associated with various actions. For example - if we're trying to predict whether a customer is going to leave or not, I'd choose a different threshold based on whether we're mass e-mailing people, or if we're having customer service reps call people, since the underlying costs for each action are so different. In the first case it's fine to cast a wide net, and e-mail lots of people who we're less certain about, in the second we'd optimize to target people we're most sure are going to leave, since it costs so much more to reach out to them.
  - jpfed 10y ago
    Maybe mutual information?
  - joeyo 10y ago
    In my opinion d-prime: https://en.wikipedia.org/wiki/Sensitivity_index https://en.wikipedia.org/wiki/Sensitivity_index
  - matk 10y ago
    AUC is robust to class imbalance. It can be understood as follows: given an instance of the positive class and an instance of the negative class, it measures the probability of your learned model being able to distinguish between the two.

by Grue3 10y ago

There are countries where more than 90% of court cases result in "guilty" verdict. My AI judge that always says "guilty" has very high accuracy in these jurisdictions.

libeclipse 10y ago

Therein lies the problem of the accuracy of predictions. If there's three AI judges, one gives a court case a 35% chance of being guilty, the other a 40%, and the last a 45%. The case was a guilty. Were the predictions correct?

kgwgk 10y ago
Measuring the accuracy of a single prediction is difficult, but there are reasonable ways to measure the accuracy of a set of predictions. For example, https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score

binalpatel 10y ago

This is the imbalanced class problem that comes up often in the real world. You can't optimize for overall accuracy, otherwise you'd have classifiers that always just predict guilty (or not sick, or not fradulent, and so on) because they'd achieve 95%+ accuracy.

hammock 10y ago
So what is the metric to optimize?
- binalpatel 10y ago
  I use AUC often, and always look at the confusion matrix. Metrics based off of it (recall, precision) are both useful to use as well. In the end the probability cutoffs you choose are more often than not based on business context and costs associated with various actions. For example - if we're trying to predict whether a customer is going to leave or not, I'd choose a different threshold based on whether we're mass e-mailing people, or if we're having customer service reps call people, since the underlying costs for each action are so different. In the first case it's fine to cast a wide net, and e-mail lots of people who we're less certain about, in the second we'd optimize to target people we're most sure are going to leave, since it costs so much more to reach out to them.
- jpfed 10y ago
  Maybe mutual information?
- joeyo 10y ago
  In my opinion d-prime: https://en.wikipedia.org/wiki/Sensitivity_index https://en.wikipedia.org/wiki/Sensitivity_index
- matk 10y ago
  AUC is robust to class imbalance. It can be understood as follows: given an instance of the positive class and an instance of the negative class, it measures the probability of your learned model being able to distinguish between the two.