In those cases, people tend to use something like a categorical cross-entropy loss where you assign a continuous likelihood score to each discrete possibility,

In those cases, people tend to use something like a categorical cross-entropy loss where you assign a continuous likelihood score to each discrete possibility, thereby making things differentiable again. https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits https://www.tensorflow.org/api_docs/python/tf/nn/sparse_soft...