Are you familiar with the Gato paper from DeepMind this May? I say this not as a gotcha, but as a genuine question that I'm interested in your response to. > I

Are you familiar with the Gato paper from DeepMind this May? I say this not as a gotcha, but as a genuine question that I'm interested in your response to. > Inspired by progress in large-scale language modelling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. > During the training phase of Gato, data from different tasks and modalities are serialised into a flat sequence of tokens, batched, and processed by a transformer neural network similar to a large language model. The loss is masked so that Gato only predicts action and text targets. https://www.deepmind.com/blog/a-generalist-agent https://www.deepmind.com/blog/a-generalist-agent