4 ms·

Last I checked that's called a softmax function. Most neural models have this built in.

opportune 4y ago
Been a while since I did DL, but I thought softmax was used in training so that “pick the most likely among X” would converge while training, not because it actually computed the probability distribution among the output classes in isolation. When running inference the softmax function yields the class the model thinks is most likely, it’s not a truly valuable measure of how likely the answer is correct, right? I could be wrong, but let’s take an example like a model trained to take pictures and return either “dog, cat, bird”. If I put in a picture of a fish, softmax will give me a probability distribution across dog, cat, and bird and find eg bird is the most likely - but that does not mean the model is telling me it is confident the image is a bird, just that it’s more confident that it’s a bird vs a dog or cat. I think for DL you’d need to design the model so that “unknown” is an acceptable output and has a lower (but non-zero) error/loss compared to an incorrect proper classification. Maybe people already do that or something similar
- KRAKRISMOTT 4y ago
  I am curious how that could be implemented. Given a cat/dog classifier, theoretically assuming everything is iid, an "unknown" would be equally far from either category in latent space. So in some ways, you can gate the softmax output if it's too close to 50-50. But this is more of a hack then a real solution to representing the idea of "unknown".
  - opportune 4y ago
    By introducing the concept of “unknown” you are also introducing to the model the concept of a false positive and (if there are inputs either in training or in applying the model that are truly unknown) false negative alongside “wrong”. I think you’d want to make this assessment before applying softmax as with softmax you are only getting the relative probabilities of various classifications, when “unknown” should be done based on absolute probability - I don’t care if Bird is 100x more likely than Dog or Cat if it’s still only 1% chance. I am not a DL genius so if I had to do this, I would probably just make “unknown” a true output and introduce some extra data / augmented data to make it a true input. I am guessing to do this the right way you would need a way to incentivize some convergence towards unknown, maybe involving some loss function looking at the false positive and false negative rate across a batch.
  - nothing0001 4y ago
    Just make a loss function that gives +1 for correct answer, -1 for incorrect answers and 0 for unknown. Since this idea took me like 10 seconds, I suppose something like this must/should have been used before. Unrelated, what happen when we train a model using a discontinuous function? Could the trained model be used to detect some pattern in the data?, for example if the input vector to such a model is a direct sum of two independent variables could such model be used to detect that the problem can be decomposed in two independent problems. Sorry of being off-topic and thinking aloud. Edited: The following result is from (1): In summary, while softmax classifier probabilities are not directly useful as confidence estimates, simple statistics derived from softmax distributions provide a surprisingly effective way to determine whether an example is misclassified or from a different distribution from the training data, as demonstrated by our experimental results (1) https://arxiv.org/abs/1610.02136 https://arxiv.org/abs/1610.02136

Been a while since I did DL, but I thought softmax was used in training so that “pick the most likely among X” would converge while training, not because it actually computed the probability distribution among the output classes in isolation. When running inference the softmax function yields the class the model thinks is most likely, it’s not a truly valuable measure of how likely the answer is correct, right? I could be wrong, but let’s take an example like a model trained to take pictures and return either “dog, cat, bird”. If I put in a picture of a fish, softmax will give me a probability distribution across dog, cat, and bird and find eg bird is the most likely - but that does not mean the model is telling me it is confident the image is a bird, just that it’s more confident that it’s a bird vs a dog or cat. I think for DL you’d need to design the model so that “unknown” is an acceptable output and has a lower (but non-zero) error/loss compared to an incorrect proper classification. Maybe people already do that or something similar

KRAKRISMOTT 4y ago
I am curious how that could be implemented. Given a cat/dog classifier, theoretically assuming everything is iid, an "unknown" would be equally far from either category in latent space. So in some ways, you can gate the softmax output if it's too close to 50-50. But this is more of a hack then a real solution to representing the idea of "unknown".
- opportune 4y ago
  By introducing the concept of “unknown” you are also introducing to the model the concept of a false positive and (if there are inputs either in training or in applying the model that are truly unknown) false negative alongside “wrong”. I think you’d want to make this assessment before applying softmax as with softmax you are only getting the relative probabilities of various classifications, when “unknown” should be done based on absolute probability - I don’t care if Bird is 100x more likely than Dog or Cat if it’s still only 1% chance. I am not a DL genius so if I had to do this, I would probably just make “unknown” a true output and introduce some extra data / augmented data to make it a true input. I am guessing to do this the right way you would need a way to incentivize some convergence towards unknown, maybe involving some loss function looking at the false positive and false negative rate across a batch.
- nothing0001 4y ago
  Just make a loss function that gives +1 for correct answer, -1 for incorrect answers and 0 for unknown. Since this idea took me like 10 seconds, I suppose something like this must/should have been used before. Unrelated, what happen when we train a model using a discontinuous function? Could the trained model be used to detect some pattern in the data?, for example if the input vector to such a model is a direct sum of two independent variables could such model be used to detect that the problem can be decomposed in two independent problems. Sorry of being off-topic and thinking aloud. Edited: The following result is from (1): In summary, while softmax classifier probabilities are not directly useful as confidence estimates, simple statistics derived from softmax distributions provide a surprisingly effective way to determine whether an example is misclassified or from a different distribution from the training data, as demonstrated by our experimental results (1) https://arxiv.org/abs/1610.02136 https://arxiv.org/abs/1610.02136