6 ms·

If the net outputs probability of 0.9999 but lower CI is 0.4, then you don't have to trust it. With linear models you can estimate the CI from the data and mean

by lqdc13 10y ago

If the net outputs probability of 0.9999 but lower CI is 0.4, then you don't have to trust it. With linear models you can estimate the CI from the data and mean squared error of the classifier.

tgflynn 10y ago
It sounds like you want to stack another probability on top of a probability. The net gives you an estimate of P(C|x) and then you want a probabilistic measure of how accurate an estimate that is. This starts to get a little difficult for me to think about. It seems like you might be able to estimate it by running the net on a test set (disjoint of course from the training set) and applying standard statistical methods to the results, just like you could for any experiment. EDIT: Maybe it's more complicated than that though because the experiment is itself estimating a probability, not just the value of some random variable. Maybe we need a real statistics expert to look into this.
- hikarudo 10y ago
  The net output is overconfident, but this can be (partially) corrected with probability calibration. It's also used for SVM, which is usually underconfident.
  - lqdc13 10y ago
    SVM doesn't give CI either. In fact, it doesn't even give probability estimates per sample unless using Platt's technique whose probabilities don't always agree with the classifier's outputs. What I mean is a confidence interval on probabilities. That is, if you did the training on 100 different training sets that follow the same distribution as your training set, they would have a range of outputs for a particular sample. The 95th highest and 5th lowest would have specific values. These values can be estimated and are the bounds of the CI. For example, for logistic regression, you can estimate standard error per coefficient like so: https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_sample_properties https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_... That would give you a vector of standard errors the size of your variable(feature) set. Then you can use that to estimate overall range of CI per sample. This would be useful in the example provided in OP's link. For example, let's say that the classifier says there is 86% probability that the license plate is RBX735. Can we really trust this 86% probability or is the model primarily relying on some information that is available in only 2 samples out of 1000 that might have been outliers?

by lqdc13 10y ago

If the net outputs probability of 0.9999 but lower CI is 0.4, then you don't have to trust it. With linear models you can estimate the CI from the data and mean squared error of the classifier.

tgflynn 10y ago

It sounds like you want to stack another probability on top of a probability. The net gives you an estimate of P(C|x) and then you want a probabilistic measure of how accurate an estimate that is. This starts to get a little difficult for me to think about. It seems like you might be able to estimate it by running the net on a test set (disjoint of course from the training set) and applying standard statistical methods to the results, just like you could for any experiment. EDIT: Maybe it's more complicated than that though because the experiment is itself estimating a probability, not just the value of some random variable. Maybe we need a real statistics expert to look into this.

hikarudo 10y ago
The net output is overconfident, but this can be (partially) corrected with probability calibration. It's also used for SVM, which is usually underconfident.
- lqdc13 10y ago
  SVM doesn't give CI either. In fact, it doesn't even give probability estimates per sample unless using Platt's technique whose probabilities don't always agree with the classifier's outputs. What I mean is a confidence interval on probabilities. That is, if you did the training on 100 different training sets that follow the same distribution as your training set, they would have a range of outputs for a particular sample. The 95th highest and 5th lowest would have specific values. These values can be estimated and are the bounds of the CI. For example, for logistic regression, you can estimate standard error per coefficient like so: https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_sample_properties https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_... That would give you a vector of standard errors the size of your variable(feature) set. Then you can use that to estimate overall range of CI per sample. This would be useful in the example provided in OP's link. For example, let's say that the classifier says there is 86% probability that the license plate is RBX735. Can we really trust this 86% probability or is the model primarily relying on some information that is available in only 2 samples out of 1000 that might have been outliers?