3 ms·
I thought you were going adding a class called I don't know.
by sharemywin 8y ago
I thought you were going adding a class called I don't know.
- paraschopra 8y agoI tried doing that, but I didn't find any pydontknow module on pip. Your comment reminded me of this xkcd comic https://xkcd.com/353/ https://xkcd.com/353/
- svantana 8y agoThe nice thing about using likelihood as the training criterion, you don't even need an extra category, you can just combine the outputs with basic decision theory, like so: thres = wrong_cost / (wrong_cost+right_gain) response = (max(out_probs) > thres) ? argmax(out_probs) : "I don't know" While the bayesian methods can theoretically perform better for out-of-set inputs, it feels like an oversight to not include this simple method as a baseline. My gut feeling is that this would perform equally well, if not better.
- paraschopra 8y agoThis works if you get nicely balanced probabilities out of neural network. Softmax which is popular way to transform network outputs to probabilities pushes probabilities either to zero or one. There may be other ways to transform outputs to probabilities where your approach could work but I'm not aware of them. Also to reply to OP's comments, to have a class like 'I don't know' you need to have a dataset of things that don't look like your training dataset. And you'll always miss examples of things that don't belong to your input class. Bayesian solves it in a nice way by not requiring this infinite not-input training data.