Because it is not just reproducing existing moves. An LLM can come to acquire a model which models the structure of moves (in chess for instance using letter an

Because it is not just reproducing existing moves. An LLM can come to acquire a model which models the structure of moves (in chess for instance using letter an number indexing of board positions), the checking of valid moves and the valuation of board positions. It seems surprising that training an LLM would result in layers/weights which contain such representations but it seems that this is indeed an efficient representation of information in the same way that neural nets would learn addition by encoding textual integers into floats.