3 ms·
The net output is overconfident, but this can be (partially) corrected with probability calibration. It's also used for SVM, which is usually underconfident.
by hikarudo 10y ago
The net output is overconfident, but this can be (partially) corrected with probability calibration. It's also used for SVM, which is usually underconfident.
- lqdc13 10y agoSVM doesn't give CI either. In fact, it doesn't even give probability estimates per sample unless using Platt's technique whose probabilities don't always agree with the classifier's outputs. What I mean is a confidence interval on probabilities. That is, if you did the training on 100 different training sets that follow the same distribution as your training set, they would have a range of outputs for a particular sample. The 95th highest and 5th lowest would have specific values. These values can be estimated and are the bounds of the CI. For example, for logistic regression, you can estimate standard error per coefficient like so: https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_sample_properties https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_... That would give you a vector of standard errors the size of your variable(feature) set. Then you can use that to estimate overall range of CI per sample. This would be useful in the example provided in OP's link. For example, let's say that the classifier says there is 86% probability that the license plate is RBX735. Can we really trust this 86% probability or is the model primarily relying on some information that is available in only 2 samples out of 1000 that might have been outliers?