3 ms·
I think that post is quite misleading. 90 and 100 are extreme values representing extreme confidence. Your model could just as easily learn [0.09, 0.10, 0.05]
by jeremysalwen 8y ago
I think that post is quite misleading. 90 and 100 are extreme values representing extreme confidence. Your model could just as easily learn [0.09, 0.10, 0.05] on the final layer, and the probabilities would then be [0.337, 0.340, 0.323]. And look, now softmax has the opposite problem! The final activation layers differed by a factor of 2, but softmax output almost identical probabilities!
Except this should not be surprising, because softmax is not scale invariant. Values on the unit scale is what you would normally expect, so if you feed it values 100x larger, it's going to be way more confident. If you feed it values 100x smaller, it's going to be way less confident.
- paraschopra 8y agoThis is interesting. In theory, what you are suggesting can be true but in practice you do think the last layer can output 0.1, 0.09, etc especially when last layer is typically not pushed through sigmoid.
- rjtavares 8y agoIsn't that mostly true with highly curated datasets with clearly defined categories?