I wouldn't say they've "done it" quite yet. There's definitely an application for imitation learning but that might be it. A translation of the work in sequence

I wouldn't say they've "done it" quite yet. There's definitely an application for imitation learning but that might be it. A translation of the work in sequence-to-sequence to sequence-to-action is something I've also considered researching. A few challenges exist which the author touches on in just one sentence. First, we need data about previous sequences of actions and this is necessarily a challenge in many fields in robotics/learning. A related problem is that of exploration. How exactly should we inform the exploration of new sequences? Also, if our policy is based on the prediction of a Transformer, does it have the traditional desirable properties of a policy in an RL environment? Off the top of my head it seems like a Transformer fed into an MLP would probably be fit but I'm not sure. Transformers do seem promising, but it's a bit early to say they've "done it" :)