Mmmh I'm deeply skeptical of some parts. > One explanation for why this game works is that there is only one way in which things are related There is not, thi

Mmmh I'm deeply skeptical of some parts. > One explanation for why this game works is that there is only one way in which things are related There is not, this is a completely non transitive relationship. On another point, suppose you keep the same vocabulary, but permute the signification of the words, the neural network will still learn relationships, completely different ones, but it's representation may converge toward a better compression for that set of words, but I'm dubious that this new compression scheme will ressemble the previous one (?) I would say that given an optimal encoding of the relationships, we can achieve an extreme compression, but not all encodings lead to the same compression at the end. If I add 'bla' between every words in a text, that is easy to compress, but now, if I add an increasing sequence of words between each words, the meaning is still there, but the compression will not be the same, as the network will try to generate the words in-between. (thinking out loud)