3 ms·

The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out els

by garrinm 29d ago

The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means. What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict. So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning. And I think effectively the same distinction applies here.

Planktonne 29d ago
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing. There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
- danielmarkbruce 29d ago
  It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move. So, it's not a next move predictor. It's a game result predictor.
  - YeGoblynQueenne 28d ago
    Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
    - danielmarkbruce 28d ago
      Brush up :) The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.
      - YeGoblynQueenne 28d ago
        Alright, I'll have to check up on that. Thanks for being nice about it.
        YeGoblynQueenne 28d ago
        Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return. To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode). So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function. Sutton & Barto 2ed is free here btw: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf https://web.stanford.edu/class/psych209/Readings/SuttonBarto... The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.
        danielmarkbruce 28d ago

by garrinm 29d ago

Planktonne 29d ago

I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing. There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.

danielmarkbruce 29d ago
It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move. So, it's not a next move predictor. It's a game result predictor.
- YeGoblynQueenne 28d ago
  Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
  - danielmarkbruce 28d ago
    Brush up :) The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.
    - YeGoblynQueenne 28d ago
      Alright, I'll have to check up on that. Thanks for being nice about it.
      - YeGoblynQueenne 28d ago
        Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return. To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode). So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function. Sutton & Barto 2ed is free here btw: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf https://web.stanford.edu/class/psych209/Readings/SuttonBarto... The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.
        danielmarkbruce 28d ago