You do that with human annotators. “Annotator agreement” is a measure of confidence in the correctness of labels. And you should always keep an eye out for how

You do that with human annotators. “Annotator agreement” is a measure of confidence in the correctness of labels. And you should always keep an eye out for how these are handled, when reading papers that present a dataset. Saying we should start doing model agreement is a really good idea imho.