Language is not necessarily best modelled by predicting the next token in sentence, just like Go is not best modelled by predicting the next move. It makes per

Language is not necessarily best modelled by predicting the next token in sentence, just like Go is not best modelled by predicting the next move. It makes perfect sense that predicting the next token can be improved by going back every so often to reevaluate if the sequence of tokens makes any sense as a whole. If we're conjecturing anyway I reckon the next major step won't come from changes that merely improve training or predicting, but one that fundamentally makes the model capable of learning. Removing the distinction between 'conversing' and 'learning'. This is similar to what you call 'interacting', but I reckon that mere interaction won't be enough if the distinction between letting the model predict and training the model doesn't disappear.