RLHF (and DPO) are used and aren't doing token level scoring. The article is talking about inference. Most models people are actually using have gone through R

RLHF (and DPO) are used and aren't doing token level scoring. The article is talking about inference. Most models people are actually using have gone through RLHF or DPO. So the uncertainty at inference includes all dimensions of uncertainty. A token choice can effectively be a branch from a conceptual perspective.