6 ms·
Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall. The industry ha
by a_wild_dandan 3y ago
Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall.
The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.]
Serendipitously, Karpathy broadly discussed these two issues yesterday! Toward the lecture's end: https://www.youtube.com/watch?v=zjkBMFhNj_g https://www.youtube.com/watch?v=zjkBMFhNj_g
- visarga 3y agoHow do you apply Q-learning to LLMs? They would need to estimate the action-value of each action in a given state. How can you pick the best action when you only have a scoring function that would rate how good each action is, but the action space is infinite. You can't pick the best action. It is probably something else, like Decision Transformer.
- silveraxe93 3y agoThat got solved using Deep-Q learning. I think David Silver did loads of work on it? Basically, instead of computing every state value like in normal Q learning. You use a Neural Network to estimate the best value.
- visarga 3y agoIf I remember correctly DQLearning is using discrete action space, limited number of possible actions which makes it possible to simply take the argmax_i(qvalue(state, action_i))