6 ms·
Q* Hypothesis: Enhancing Reasoning, Rewards, and Synthetic Data
- romesc 3y agoSure A* is awesome, but taking the "star" and immediately attributing it to A* is probably a bridge too far. Q* or any X* for that matter is extremely common for referring to the optimal function under certain assumptions. (usually cost / reward structure).
- tunesmith 3y agoYeah I just saw the video from that researcher (later an OpenAI researcher?) that talked about it back in 2016... not that I understood much, but it definitely seemed that Q* was a generalization of the Q algorithm described on the previous slide. The optimum something across all somethings.
- maaaaattttt 3y agoIf you have the possibility I would be quite interested in a link to the video or alternatively the name of the researcher you mention.
- andrew3726 3y agoIt's Noam Brown, he worked at Meta AI before on Cicero and No-hands Poker before that.
- resource0x 3y agoLeCun: Please ignore the deluge of complete nonsense about Q*. https://twitter.com/ylecun/status/1728126868342145481 https://twitter.com/ylecun/status/1728126868342145481
- zaptrem 3y agoAs someone with a borderline acceptable understanding of RL this is the most accurate take so far.
- Zolde 3y agoIt will be nice to see the breakthroughs resulting from what people _believed_ Q* to have been.
- erikaww 3y agocertainly more things to throw at the wall! Excited to see the "accidental" progress
- bschne 3y agoI love this take. Reminds me of how the Mechanical Turk apparently indirectly inspired someone to build a weaving machine b/c "how hard could it be if machines can play chess" -- https://x.com/gordonbrander/status/1385245747071787008?s=20 https://x.com/gordonbrander/status/1385245747071787008?s=20
- jiggawatts 3y agoThis is one of my favourite "errors" of human thinking: mistaking something false for reality, and then making it real based on the confidence gained from that initial mistake. An example I heard was that one of the programmers working on the original Unreal engine saw a demo of John Carmack's constructive solid geometry (CSG) editor. He incorrectly surmised that this was a real-time editor, so he hurriedly made one for the Unreal game engine to "keep up" with Quake. In reality, the Quake editor wasn't nearly as responsive as he assumed, and in fact he had to significantly advance the state of the art to "keep up"!
- vsnf 3y agoI recall that this is the story behind overlapping windows on the old Macintosh computers, although a concrete source seems difficult to find[0] [0] https://news.ycombinator.com/item?id=2998463 https://news.ycombinator.com/item?id=2998463
- spicyusername 3y agoI have trouble believing this isn't just a sneaky marketing campaign.
- dmix 3y agoNothing OpenAI has released product-wise (ChatGPT, Dall-E) has required 'marketing'. The value speaks for itself. People raving about it on twitter, telling their friends/coworkers, and journos documenting their explorations is more than enough. If this was an extremely competitive market that'd be more plausible. But they enjoy some pretty serious dominance and are struggling to handle the growth they already have with GPT. If Q* is real, you likely wouldn't need to hype up something that has the potential to solve math / logic problems without having seen the problem/solution before hand. Something that novel would be hugely valuable and generate demand naturally.
- djvdq 3y agoOf course they are doing PR stunts to kepp media talking about them. Remember Altman saying that they shouldn't release GPT-2 because of it being too dangerous? It's the same thing with this Q* thing.
- FeepingCreature 3y agoBecause it could be used to generate spam, yes, and he was right about that. And to set a precedent that models should be released cautiously, and he was right about that too, and it is to our detriment that we don't take that more seriously.
- mdekkers 3y ago> it is to our detriment that we don't take that more seriously. Why?
- FeepingCreature 3y agoBecause when a network turns out to be dangerous in some unexpected way, you cannot exactly unrelease it.
- ben_w 3y agoI definitely need to blog more. A* search with a neural network as the heuristic function seemed like a good idea to investigate… a month or two ago, and I never got around to it.
- haltist 3y agoI have an idea for a great AI project and it's about finding the first logical inconsistency in an argument about a formal system like an LLM. I think if OpenAI can deliver that then I will believe they have achieved AGI. I am a techno-optimist and I believe this is possible and all I need is a lot of money. I think $80B would be more than sufficient. I will be awaiting a reply from other techno-optimists like Marc Andreesen and those who are techno-optimist adjacent like millionaires and billionaires that read HN comments.
- varelse 3y ago[dead]
- kelseyfrog 3y agoA* is a red-herring based on availability bias. Q* is already a thing and it's the Bellman equation describing the optimal action-value function.
- bertil 3y agoAre you saying that the Bellman equations already use the notation Q*, or are you saying that those equations (I’m not as familiar as I should be, sorry) are the obvious connection between the incoherent ramblings from Reuters? Because having similar acronyms or notations used for multiple contexts that end up collapsing with cross-pollination of ideas is far too frequent these days. I once made a dictionary of terms used in A/B testing / Feature Flags / DevOps / Statistics / Econometrics, and most keywords had multiple, incompatible acceptions depending on the exact context, all somewhat relevant to A/B testing. Every reader came out of it so defeated, like language itself was broken…
- ElectricalUnion 3y agoCan you link this dictionary here or is it proprietary?
- bertil 3y agoIt was proprietary when I made it, but no one seemed to care–there wasn't anything that wasn't painfully public in there. I'll probably edit and include it in a class I'm planning about teaching people about A/B testing.
- tnecniv 3y agoQ* is an incredibly common notation for the above version of the Bellman equation. I think it’s stupid to call an algorithm Q* for the same reason it is to read too much into this: it’s an incredibly nondescript name.
- kelseyfrog 3y agoI'm saying that everyone already uses that notation including OpenAI[1]. 1. https://spinningup.openai.com/en/latest/algorithms/ddpg.html https://spinningup.openai.com/en/latest/algorithms/ddpg.html
- adamnemecek 3y agoBoth RL and A* are both approaches to dynamic programming, this would not be surprising.
- jbrisson 3y agoImho, in order to reach AGI you have to get out of the LLM space. It has to be something else. Something close to biological plausability.
- bob1029 3y agoI think big parts of the answer include time domain, multi-agent and iterative concepts. Language is about communication of information between parties. One instance of an LLM doing one-shot inference is not leveraging much of this. Only first-order semantics can really be explored. There is a limit to what can be communicated in a context of any size if you only get one shot at it. Change over time is a critical part of our reality. Imagine if your agent could determine that it has been thinking about something for too long and adapt strategy automatically. Increase to higher param model, adapt the context, etc. Perhaps we aren't seeking total AGI/ASI either (aka inventing new physics). From a business standpoint, it seems like we mostly have what we need now. The next ~3 months are going to be a hurricane in our shop.
- hackinthebochs 3y agoLLMs as we currently understand them won't reach AGI. But AGI will very likely have an LLM as a component. What is language but a way to represent arbitrary structure? Of course that's relevant to AGI.
- valine 3y agoCovering an airplane in feathers isn't going to make it fly faster. Biological plausibility is a red haring imho.
- foooorsyth 3y agoThe training space is more important. I don’t think a general intelligence will spawn from text corpuses. A person only able to consume text to learn would be considered severely disabled. A significant part of intelligence comes from existence in meatspace and the ability to manipulate and observe that meatspace. A two year old learns much faster with much less data than any LLM.
- varelse 3y ago[dead]
- janalsncm 3y agoIs it possible they were referring to this research they published in May? https://openai.com/research/improving-mathematical-reasoning-with-process-supervision https://openai.com/research/improving-mathematical-reasoning...
- fizx 3y agoThe most likely hypothesis I've seen for Q*: https://twitter.com/alexgraveley/status/1727777592088867059 https://twitter.com/alexgraveley/status/1727777592088867059
- urbandw311er 3y agoSee also https://news.ycombinator.com/item?id=38407741 https://news.ycombinator.com/item?id=38407741
- bijinpanicker 3y ago[dead]