3 ms·

I don't think your intuition about computer chess is going to help you here with transformer architecture. Usually in transformer models[1], for each attention

I don't think your intuition about computer chess is going to help you here with transformer architecture. Usually in transformer models[1], for each attention head there are 3 vectors of weights, known as q(uery), k(ey) and v(alue). I was assuming that the q in q* applied to the q vector, so q-learning is training this vector. In a transformer model you don't have an objective function for any sort of state evaluation so it can't be the Q you're thinking of. If they've done something funky with the q vector that could indeed be a breakthrough since a lot of people feel that we are sort of running out of juice with scaling transformers as-is and need a new architecture to really have a step change in capability. That's pure speculation on my part though. [1] Here's Vaswani et al, the paper that first set out the transformer architecture https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762

Buttons840 3y ago
Good point. I'm thinking of the Q from reinforcement learning, but as you say, this Q is probably different.
a_wild_dandan 3y ago
Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall. The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.] Serendipitously, Karpathy broadly discussed these two issues yesterday! Toward the lecture's end: https://www.youtube.com/watch?v=zjkBMFhNj_g https://www.youtube.com/watch?v=zjkBMFhNj_g
- visarga 3y ago
  How do you apply Q-learning to LLMs? They would need to estimate the action-value of each action in a given state. How can you pick the best action when you only have a scoring function that would rate how good each action is, but the action space is infinite. You can't pick the best action. It is probably something else, like Decision Transformer.
  - silveraxe93 3y ago
    That got solved using Deep-Q learning. I think David Silver did loads of work on it? Basically, instead of computing every state value like in normal Q learning. You use a Neural Network to estimate the best value.
    - visarga 3y ago
      If I remember correctly DQLearning is using discrete action space, limited number of possible actions which makes it possible to simply take the argmax_i(qvalue(state, action_i))
shwaj 3y ago
They’re not assuming that the Q refers to the Transformer’s “query”, they’re talking about the reward function in reinforcement learning, using chess as an example.

Buttons840 3y ago

Good point. I'm thinking of the Q from reinforcement learning, but as you say, this Q is probably different.

a_wild_dandan 3y ago

Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall. The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.] Serendipitously, Karpathy broadly discussed these two issues yesterday! Toward the lecture's end: https://www.youtube.com/watch?v=zjkBMFhNj_g https://www.youtube.com/watch?v=zjkBMFhNj_g

visarga 3y ago
How do you apply Q-learning to LLMs? They would need to estimate the action-value of each action in a given state. How can you pick the best action when you only have a scoring function that would rate how good each action is, but the action space is infinite. You can't pick the best action. It is probably something else, like Decision Transformer.
- silveraxe93 3y ago
  That got solved using Deep-Q learning. I think David Silver did loads of work on it? Basically, instead of computing every state value like in normal Q learning. You use a Neural Network to estimate the best value.
  - visarga 3y ago
    If I remember correctly DQLearning is using discrete action space, limited number of possible actions which makes it possible to simply take the argmax_i(qvalue(state, action_i))

shwaj 3y ago

They’re not assuming that the Q refers to the Transformer’s “query”, they’re talking about the reward function in reinforcement learning, using chess as an example.