They're comparing two probabilities within the same distribution. KL divergence is for comparing two probability distributions. The difference between the prob

They're comparing two probabilities within the same distribution. KL divergence is for comparing two probability distributions. The difference between the probability of the top choice and the probability of the 2nd-top choice is their ad-hoc attempt at capturing the confidence of the top answer. It's maybe not the most principled approach, but KL divergence would not be appropriate here. You could argue that maybe the entropy of the distribution would be interesting (lower entropy indicating higher model confidence). But entropy is "global" and takes into account the distribution over all tokens, when really all we care about is the distribution among the top tokens. So you could do something like find an "elbow" in the token probability distribution and look at the entropy just among those top tokens (maybe top 20 tokens, or top 80% of probability with normalization for # of tokens involved), but then you're back in the world of ad-hoc measurements. Unless you were talking about the KL divergence between the model output distribution and the uniform distribution, but that's very closely related to entropy anyway.