The concept of calibration - actually being right at the rate that matches your estimated percent confidence - is important for machine learning. It's important

The concept of calibration - actually being right at the rate that matches your estimated percent confidence - is important for machine learning. It's important to know how well you can trust the prediction of a ML model (ironically to that you can do something like send low confidence predictions for manual review, which apparently might not be that calibrated either)... Deep learning models in particular are not good at accurately estimating confidence, so there is a whole body of work on getting better confidence estimates, e.g. [0]. It's nice to see that people make the same kind of mistakes. [0]Guo et al., https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1706.04599