4 ms·

By the same reasoning if you train a neural net to output next action from the output of the autoencoder then the whole system also has a "world model", but if

By the same reasoning if you train a neural net to output next action from the output of the autoencoder then the whole system also has a "world model", but if you accept that definition of "world model" then it is extremely weak and not the intelligence-like capability that is being implied. And as I said in my original comment they are probably not even able to extract the board state very well, otherwise they would depict some kind of direct representation of the state, not all of the other figures of board move causality etc. Note also that the board state is not directly encoded in the neural network: they train another neural network to find weights to approximate the board state if given the internal weights of the Othello network. It's a bit of fishing for the answer you want.

IanCal 2y ago
> And as I said in my original comment they are probably not even able to extract the board state very well, They do measure and report on this, both in summary in the blog post and in more detail in the paper. > otherwise they would depict some kind of direct representation of the state If you can perfectly accurately extract the state the result would be pretty boring to show right? It'd just be a picture of a board state and next to it the same board state with "these are the same". > Note also that the board state is not directly encoded in the neural network: they train another neural network to find weights to approximate the board state if given the internal weights of the Othello network. If you can extract them, they are encoded in the activations. That's pretty much by definition surely. > It's a bit of fishing for the answer you want. How so? Given a sequence of moves, they can accurately identify which state most of the positions of the board are in just by looking at the network. In order for that to work, the network must be turning a sequence of moves into some representation of a current board state. Assume for the moment they can accurately identify them do you agree with that conclusion?
- foobarqux 2y ago
  > They do measure and report on this, both in summary in the blog post and in more detail in the paper. I didn't see this in the blog post, where is it? Presumably they omitted it from the blog post because the results are bad as I describe below, which is precisely why I cited it as a red flag. > If you can perfectly accurately extract the state the result would be pretty boring to show right? It'd just be a picture of a board state and next to it the same board state with "these are the same". But they don't! They have nearly 10% tile level error on the human-data-trained model. That's nearly 100% board error. It's difficult to understand how bad this is but if you visualize it somehow (for example sampling random boards) it becomes obvious that it is really bad. On average about 6 or 7 tiles are going to be wrong. With nearly 100% probability you get an incorrect board. > If you can extract them, they are encoded in the activations. That's pretty much by definition surely. No, that's silly. For example you can cycle through every algorithm/transformation imaginable until you hit one that extracts the satanic verses of the Bible. As I said in another comment although it is in theory mitigated somewhat by doing test/validation splits in practice you keep trying different neural network hyperparameters to finesse your validation performance. > How so? Given a sequence of moves, they can accurately identify which state most of the positions of the board are in just by looking at the network. In order for that to work, the network must be turning a sequence of moves into some representation of a current board state. Assume for the moment they can accurately identify them do you agree with that conclusion? What conclusion? I believe you can probably train a neural network to take in board moves and output board state with some level of board error. So what?

IanCal 2y ago

> And as I said in my original comment they are probably not even able to extract the board state very well, They do measure and report on this, both in summary in the blog post and in more detail in the paper. > otherwise they would depict some kind of direct representation of the state If you can perfectly accurately extract the state the result would be pretty boring to show right? It'd just be a picture of a board state and next to it the same board state with "these are the same". > Note also that the board state is not directly encoded in the neural network: they train another neural network to find weights to approximate the board state if given the internal weights of the Othello network. If you can extract them, they are encoded in the activations. That's pretty much by definition surely. > It's a bit of fishing for the answer you want. How so? Given a sequence of moves, they can accurately identify which state most of the positions of the board are in just by looking at the network. In order for that to work, the network must be turning a sequence of moves into some representation of a current board state. Assume for the moment they can accurately identify them do you agree with that conclusion?

foobarqux 2y ago
> They do measure and report on this, both in summary in the blog post and in more detail in the paper. I didn't see this in the blog post, where is it? Presumably they omitted it from the blog post because the results are bad as I describe below, which is precisely why I cited it as a red flag. > If you can perfectly accurately extract the state the result would be pretty boring to show right? It'd just be a picture of a board state and next to it the same board state with "these are the same". But they don't! They have nearly 10% tile level error on the human-data-trained model. That's nearly 100% board error. It's difficult to understand how bad this is but if you visualize it somehow (for example sampling random boards) it becomes obvious that it is really bad. On average about 6 or 7 tiles are going to be wrong. With nearly 100% probability you get an incorrect board. > If you can extract them, they are encoded in the activations. That's pretty much by definition surely. No, that's silly. For example you can cycle through every algorithm/transformation imaginable until you hit one that extracts the satanic verses of the Bible. As I said in another comment although it is in theory mitigated somewhat by doing test/validation splits in practice you keep trying different neural network hyperparameters to finesse your validation performance. > How so? Given a sequence of moves, they can accurately identify which state most of the positions of the board are in just by looking at the network. In order for that to work, the network must be turning a sequence of moves into some representation of a current board state. Assume for the moment they can accurately identify them do you agree with that conclusion? What conclusion? I believe you can probably train a neural network to take in board moves and output board state with some level of board error. So what?