Yes, it contradicts itself if you call it multiple times. So I get to measure ensemble agreement as a proxy for correctness. But what do I do with the examples

Yes, it contradicts itself if you call it multiple times. So I get to measure ensemble agreement as a proxy for correctness. But what do I do with the examples where the ensemble is not confident? They are too many to label by hand, and a pity to throw out - they could be the most interesting training examples.