3 ms·

AlphaGo works because it's grounded in the rules of Go. The rules are unambiguous and programmed into the system manually. There are no such unambiguous rules

AlphaGo works because it's grounded in the rules of Go. The rules are unambiguous and programmed into the system manually. There are no such unambiguous rules to ground language models. Iterative "improvement" could easily degenerate into nonsense without grounding in the real world. That's why some people think that true AGI will need to be grounded by the laws of physics, via experience interacting with the real physical world using robot bodies.

deleted 3y ago
[deleted]
CGamesPlay 3y ago
Any output of the system that interacts with humans can be used for this, it doesn't necessarily need to be robots and physical manipulation. For example, it could be "code a product" where the models all get the same spec and compete to provide the best implementation, or "create the funniest meme" where the models are judged by each other and then grounded in the real-world rules of reddit upvotes.
- modeless 3y ago
  Any grounding that comes from humans is scarce. For training we really want automatic grounding from rules that can be applied trillions of times per second rather than labor intensive human grounding at much lower rates. Of course, grounding from robots is also scarce unless you build a whole lot of robots. Ultimately we need systems that generalize rules from a small amount of grounding data. I guess that describes world models, which large language models currently lack (explicitly, at least).
  - skybrian 3y ago
    Grounding from programming languages (or similar sandboxes) wouldn't be scarce and feedback would be faster. It only goes so far, but considering how far LLM's got already, it seems promising.
contravariant 3y ago
Language is not necessarily best modelled by predicting the next token in sentence, just like Go is not best modelled by predicting the next move. It makes perfect sense that predicting the next token can be improved by going back every so often to reevaluate if the sequence of tokens makes any sense as a whole. If we're conjecturing anyway I reckon the next major step won't come from changes that merely improve training or predicting, but one that fundamentally makes the model capable of learning. Removing the distinction between 'conversing' and 'learning'. This is similar to what you call 'interacting', but I reckon that mere interaction won't be enough if the distinction between letting the model predict and training the model doesn't disappear.
gary_0 3y ago
The rules of Go are not exactly "programmed in" to AlphaGo; they are only provided so the model knows what the valid moves and win conditions are as it self-trains. The latest versions of the model can be quickly adapted to play other games[0] and MuZero can "master games without knowing their rules"[1]. Also, some of DeepMind's earliest work was a deep reinforcement learning model that could play seven different Atari games with no adjustments needed to switch games[2]. [0] https://en.wikipedia.org/wiki/AlphaGo_Zero https://en.wikipedia.org/wiki/AlphaGo_Zero [1] https://en.wikipedia.org/wiki/MuZero https://en.wikipedia.org/wiki/MuZero [2] https://arxiv.org/abs/1312.5602 https://arxiv.org/abs/1312.5602
- modeless 3y ago
  The rules of Go very much are programmed in to AlphaGo. The neural nets are generic, sure. But the game rules programmed manually into the training system are vital to the system working. Even with MuZero: while it uses a world model which is learned, ultimately during training there is still grounding in the manually programmed rules of whatever game it's playing. When it trains on Atari, the Atari system (manually programmed, obviously) provides the grounding rules. What I am saying is there is no analogous source of manually programmed rules to ground language models during training.
bitshiftfaced 3y ago
Seems like now that we have multimodal LLMs and a virtually infinite supply of digitized video, we could get it to prompt itself about what might happen in the video and then tie rewards to what actually happens later in the video.
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

CGamesPlay 3y ago

Any output of the system that interacts with humans can be used for this, it doesn't necessarily need to be robots and physical manipulation. For example, it could be "code a product" where the models all get the same spec and compete to provide the best implementation, or "create the funniest meme" where the models are judged by each other and then grounded in the real-world rules of reddit upvotes.

modeless 3y ago
Any grounding that comes from humans is scarce. For training we really want automatic grounding from rules that can be applied trillions of times per second rather than labor intensive human grounding at much lower rates. Of course, grounding from robots is also scarce unless you build a whole lot of robots. Ultimately we need systems that generalize rules from a small amount of grounding data. I guess that describes world models, which large language models currently lack (explicitly, at least).
- skybrian 3y ago
  Grounding from programming languages (or similar sandboxes) wouldn't be scarce and feedback would be faster. It only goes so far, but considering how far LLM's got already, it seems promising.

contravariant 3y ago

Language is not necessarily best modelled by predicting the next token in sentence, just like Go is not best modelled by predicting the next move. It makes perfect sense that predicting the next token can be improved by going back every so often to reevaluate if the sequence of tokens makes any sense as a whole. If we're conjecturing anyway I reckon the next major step won't come from changes that merely improve training or predicting, but one that fundamentally makes the model capable of learning. Removing the distinction between 'conversing' and 'learning'. This is similar to what you call 'interacting', but I reckon that mere interaction won't be enough if the distinction between letting the model predict and training the model doesn't disappear.

gary_0 3y ago

The rules of Go are not exactly "programmed in" to AlphaGo; they are only provided so the model knows what the valid moves and win conditions are as it self-trains. The latest versions of the model can be quickly adapted to play other games[0] and MuZero can "master games without knowing their rules"[1]. Also, some of DeepMind's earliest work was a deep reinforcement learning model that could play seven different Atari games with no adjustments needed to switch games[2]. [0] https://en.wikipedia.org/wiki/AlphaGo_Zero https://en.wikipedia.org/wiki/AlphaGo_Zero [1] https://en.wikipedia.org/wiki/MuZero https://en.wikipedia.org/wiki/MuZero [2] https://arxiv.org/abs/1312.5602 https://arxiv.org/abs/1312.5602

modeless 3y ago
The rules of Go very much are programmed in to AlphaGo. The neural nets are generic, sure. But the game rules programmed manually into the training system are vital to the system working. Even with MuZero: while it uses a world model which is learned, ultimately during training there is still grounding in the manually programmed rules of whatever game it's playing. When it trains on Atari, the Atari system (manually programmed, obviously) provides the grounding rules. What I am saying is there is no analogous source of manually programmed rules to ground language models during training.

bitshiftfaced 3y ago

Seems like now that we have multimodal LLMs and a virtually infinite supply of digitized video, we could get it to prompt itself about what might happen in the video and then tie rewards to what actually happens later in the video.

deleted 3y ago

[deleted]