3 ms·
>That said, GPT has no model of the world. I don't think this is necessarily true. Here is an example where researchers trained a transformer to generate legal
by sebzim4500 4y ago
>That said, GPT has no model of the world.
I don't think this is necessarily true. Here is an example where researchers trained a transformer to generate legal sequences of moves in the board game Othello. Then they demonstrated that the internal state of the model did, in fact, have a representation of the board.
https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2210.13382
- gowld 4y agoThat's a GPT and it's specific for one dataset of one game. How would someone extend that to all games and all other fields of human endeavor?
- sebzim4500 4y agoI'm not sure, the reason you could prove for Othello that the 'world model' exists is that the state is so simple there is really only one reasonable way to represent it with a vector (one component for each square). Even for something like chess there is a huge amount of choice for how to represent the board, yet alone trying represent the state of the actual world.