This could be handled by treating the annotations for toxicity/accuracy as distributions rather than scalar values (which is probably how we'd get them in the f

This could be handled by treating the annotations for toxicity/accuracy as distributions rather than scalar values (which is probably how we'd get them in the first place, since we'd have redundant evaluators for token sequences) and training the AI to predict both the mean and variance of the distribution. Areas where the AI had limited knowledge and things that people disagree on might have the same mean, but they'd get high variance scores, which would help people differentiate.