11 ms·

They're estimating a probability distribution over the next token, from which a sample is taken. Close enough.

• danielmarkbruce 29d ago
  It's not an estimation of something. It's a policy.
  • gwerbin 29d ago
    Sure, you're right. But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
    • danielmarkbruce 28d ago
      The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.
      • gwerbin 27d ago
        So? tokens are emitted one at a time according to the output distribution & sampler algorithm, and the next token distribution is a function of the preceding token sequence only. The process by which the output distribution is shaped doesn't change the core mental model, and doesn't reduce its value. It's a prediction in the jargonic sense that an inference about future values of a time series is broadly called "prediction", and it's relevant for reasoning about LLMs because they are fundamentally limited to converting tokens sequences into next-token predictive distributions, and that bears on how they can do what they do and what their limitations are. Nothing about the training process changes that.