4 ms·

Uh, the lead of the article claims it achieves "97% accuracy on MNIST". If you are removing 12% of the MNIST images at your neural network's discretion, you can

Uh, the lead of the article claims it achieves "97% accuracy on MNIST". If you are removing 12% of the MNIST images at your neural network's discretion, you can't claim the 96% accuracy on the remaining set as your overall accuracy! You're essentially self-selecting for test data the classifier is confident in, so of course that accuracy will be high. If this was allowed then I'd make an ImageNet classifier which just discards all images except for one I'm really good at classifying, then claim I have 100% accuracy on ImageNet.

paraschopra 8y ago
My main point is not about aggregate accuracy, it's about comparing confidence across classes of datasets. Edit: the entire point of bayesian approach is that you can make decisions on a loss function where you can make a tradeoff on the (business) cost of making a wrong decision and (business) regret of not making a decision. For real world applications, accuracy is a useless metric. What matters is the net (business) benefit or loss.
- chrisfosterelli 8y ago
  Your article bolds the following in the introductory paragraph: > My final classifier had accuracy of ~97% on MNIST and it refused to classify white noise and the majority of unrelated (non-MNIST) images. I'm saying that you are making a misleading claim when you say this. It is categorically untrue that you achieve 97% on MNIST with this approach. Accuracy on a standardized test set makes no sense when you select a subset from the test set for maximal accuracy. I'm not saying that your approach doesn't have value or make sense in some business cases, because I think the overall idea is interesting but these statements are not correct.
  - pacala 8y ago
    Correct. That's why people use https://en.wikipedia.org/wiki/F1_score https://en.wikipedia.org/wiki/F1_score instead of just accuracy.
dmurray 8y ago
Yes, this is buried deeply in the article, but the classifier rejected 12% of the handwritten digits - and did not even have good accuracy when forced to guess on those 12%.

paraschopra 8y ago

My main point is not about aggregate accuracy, it's about comparing confidence across classes of datasets. Edit: the entire point of bayesian approach is that you can make decisions on a loss function where you can make a tradeoff on the (business) cost of making a wrong decision and (business) regret of not making a decision. For real world applications, accuracy is a useless metric. What matters is the net (business) benefit or loss.

chrisfosterelli 8y ago
Your article bolds the following in the introductory paragraph: > My final classifier had accuracy of ~97% on MNIST and it refused to classify white noise and the majority of unrelated (non-MNIST) images. I'm saying that you are making a misleading claim when you say this. It is categorically untrue that you achieve 97% on MNIST with this approach. Accuracy on a standardized test set makes no sense when you select a subset from the test set for maximal accuracy. I'm not saying that your approach doesn't have value or make sense in some business cases, because I think the overall idea is interesting but these statements are not correct.
- pacala 8y ago
  Correct. That's why people use https://en.wikipedia.org/wiki/F1_score https://en.wikipedia.org/wiki/F1_score instead of just accuracy.

dmurray 8y ago

Yes, this is buried deeply in the article, but the classifier rejected 12% of the handwritten digits - and did not even have good accuracy when forced to guess on those 12%.