6 ms·
Could it be named in reference to something like the A* search algorithm? What if Q stands for Query in transformer attention? How would that type of search tra
by ilaksh 3y ago
Could it be named in reference to something like the A* search algorithm? What if Q stands for Query in transformer attention? How would that type of search translate to transformers?
They said that it's able to solve simple math problems. If it's related to A* then maybe it's trying to find a path to something. The answer to a word problem?
- bitshiftfaced 3y agoMy mind went to Q learning.
- ilaksh 3y agoOk so maybe nothing to do with A*, but actually a way for GPT-powered models or agents to learn through automated reinforcement learning. Or something. I wonder if DeepMind is working on something similar also. If your hunch is right, this could lead to the type of self-improvement that scares people.
- Davidzheng 3y agoCould easily be both
- flibble 3y agoMine to Q of Star Trek.
- Izkata 3y agoBut were you thinking of Q, Q, or Q?
- deleted 3y ago[deleted]
- eli_gottlieb 3y agoMy mind went to some kind of Q-learning combined with something like a Monte Carlo Tree Search with some kind of A*-style heuristic to effectively combine Q-learning and with short-horizon planning.
- theGnuMe 3y agoThis was alpha-go and alpha-zero right?
- a-dub 3y agolikewise. i can already imagine a* being useful for efficiently solving basic algebra and proofs. it could form the basis of a generalized planning engine and that planning engine could potentially be dangerous given the inherent competitive reasoning behind any minmax style approach.
- vanshg 3y agoIn reinforcement learning, Q* represents the optimal action-value function
- andy_xor_andrew 3y agowhich makes sense. you can pretty easily imagine the problem of "selecting the next token" as a tree of states, with actions transitioning from one to another, just like a game. And you already have naive scores for each of the states (the logits for the tokens). It's not hard to imagine applying well-known tree searching strategies, like monte-carlo tree search, minimax, etc. Or, in the case of Q*, maybe creating another (smaller) action/value model that guides the progress of the LLM.
- sudosysgen 3y agoAbsolutely, maximizing conditional probabilities is easily modeled as a Markov decision process, which is why you can use RL to train Transformers so well (hence RLHF, I've also been experimenting with RL based training for Transformers for other applications - it's promising!). Using a transformer as a model for RL to try to choose tokens to maximize overall likelihood given immediate conditional likelihood estimation is something that I imagine many people experimented with, but I can see it being tricky enough for OpenAI to be the only ones to pull it off.
- kelseyfrog 3y ago*(spelled STAR) can also refer to Semantic-parsing Transformer and ASP Reasoner, which while a bit of a stretch, is a concept that shows up in literature.
- bhy 3y agoMy bet is a general search algorithm similar AlphaGo that uses LLM as world model, and heuristic search to find the right path to goal. There's already evidence in academics that this can significantly boost model performance [1]. [1] https://llm-mcts.github.io/ https://llm-mcts.github.io/