In many cases, knowing what to do in very diverse environments could be harder than the physical control of the robots. I've read somewhere that the physical ma

In many cases, knowing what to do in very diverse environments could be harder than the physical control of the robots. I've read somewhere that the physical manipulation of robot body is not as hard as understanding multimodal info (linguistic/visual/auditory), esp in real time. > Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. The last clause is important and suggests there might be further advances from such joint training, incl from interactions in the real world.