Gato and RoboCat[0] are examples of what transformer architectures with environment/action tokens can accomplish. These are pretrained with visual and experien

Gato and RoboCat[0] are examples of what transformer architectures with environment/action tokens can accomplish. These are pretrained with visual and experiential data, but ultimately are "simply generating the next token". This is why it's likely that transformers are capable of AGI, and the top models are no longer strictly LLMs since they are multimodal. [0] https://deepmind.google/research/publications/35829/ https://deepmind.google/research/publications/35829/