3 ms·

Does anyone have any references to published works discussing the idea he’s pushing here that all the top labs are trying to replace next token prediction with

Does anyone have any references to published works discussing the idea he’s pushing here that all the top labs are trying to replace next token prediction with planning? (Note: I’m not interested in using a token-prediction LLM to solve planning problems; I’m interested in work that tries to replace token sampling with action sampling by formulating the text completion problem as a planning domain.) Or is that not what he’s talking about…?

loveparade 3y ago
If you do search for Reinforcement Learning based decoding in language models (or neural machine translation) you'll find hundreds of papers. It's a super common research topic that has been around forever, way before LLMs were a "thing". Just follow the reference from there. Nothing new here, really. But doing it on the scale of something like GPT-4 is only something OAI can do, and I don't doubt that there's some breakthrough there. But IMO it's likely that the breakthrough isn't algorithmic, but a result of scale, like most of the other OAI breakthroughs.
- __loam 3y ago
  > But doing it on the scale of something like GPT-4 is only something OAI can do It's shocking to me that people think Google and Meta don't have the scale to operate at this level.
  - loveparade 3y ago
    They have the scale in terms of hardware, but they likely don't have the infrastructure to experiment efficiently on this scale on LLMs specifically. OpenAI has been building LLM-specific infrastructure for years. Why do you think Google Bard sucks so much? It's not because there's some special magic secret that only OAI knows. It's because you need a huge amount of purpose-built infrastructure and trial-and-error iteration from experiments each of which take months. You can't easily catch up to that in a few months. Even if you throw 10x more staff at it, you are not going to make progress faster. The only way you could catch up faster is by having orders of magnitude more hardware. But due the global shortage on hardware right now, nobody has that, not even Google.
    - sangnoir 3y ago
      > They have the scale in terms of hardware, but they likely don't have the infrastructure to experiment efficiently on this scale on LLMs specifically This reads like wishful thinking, in the absence of a real moat. Do you think the hyperscalers who are buiding data centers as quickly as they can get permits do not have the infra scale to match OpenAI? Who do you suppose has been providing ever-increasing revenue to Nvidia these past few years? They didn't start doing ML experimentation when ChatGPT was announced. I agree that they aren't specifically giving LLMs all of their attention - their infrastructure still has to support existing, profitable products that bring in real revenue. I disagree that they can't experiment to the same scale.
    - __loam 3y ago
      > But due the global shortage on hardware right now, nobody has that, not even Google. Google has nearly unlimited resources to pursue this shit.

loveparade 3y ago

If you do search for Reinforcement Learning based decoding in language models (or neural machine translation) you'll find hundreds of papers. It's a super common research topic that has been around forever, way before LLMs were a "thing". Just follow the reference from there. Nothing new here, really. But doing it on the scale of something like GPT-4 is only something OAI can do, and I don't doubt that there's some breakthrough there. But IMO it's likely that the breakthrough isn't algorithmic, but a result of scale, like most of the other OAI breakthroughs.

__loam 3y ago
> But doing it on the scale of something like GPT-4 is only something OAI can do It's shocking to me that people think Google and Meta don't have the scale to operate at this level.
- loveparade 3y ago
  They have the scale in terms of hardware, but they likely don't have the infrastructure to experiment efficiently on this scale on LLMs specifically. OpenAI has been building LLM-specific infrastructure for years. Why do you think Google Bard sucks so much? It's not because there's some special magic secret that only OAI knows. It's because you need a huge amount of purpose-built infrastructure and trial-and-error iteration from experiments each of which take months. You can't easily catch up to that in a few months. Even if you throw 10x more staff at it, you are not going to make progress faster. The only way you could catch up faster is by having orders of magnitude more hardware. But due the global shortage on hardware right now, nobody has that, not even Google.
  - sangnoir 3y ago
    > They have the scale in terms of hardware, but they likely don't have the infrastructure to experiment efficiently on this scale on LLMs specifically This reads like wishful thinking, in the absence of a real moat. Do you think the hyperscalers who are buiding data centers as quickly as they can get permits do not have the infra scale to match OpenAI? Who do you suppose has been providing ever-increasing revenue to Nvidia these past few years? They didn't start doing ML experimentation when ChatGPT was announced. I agree that they aren't specifically giving LLMs all of their attention - their infrastructure still has to support existing, profitable products that bring in real revenue. I disagree that they can't experiment to the same scale.
  - __loam 3y ago
    > But due the global shortage on hardware right now, nobody has that, not even Google. Google has nearly unlimited resources to pursue this shit.