3 ms·

They collect data for RL/IL in simulation which _can_ generalize to the real world. Also, being Google, they have the resources to collect data by brute force i

They collect data for RL/IL in simulation which _can_ generalize to the real world. Also, being Google, they have the resources to collect data by brute force i.e. scientists manually collecting that data. The paper says - one main source of the data is internet scale vision/llm data. The second source is 6k trial runs. A principal idea behind this work is that you can collect data in one domain to avoid collecting data in another. Training an LLM can train the 'reasoning' portion of the robot so that it can perform real-world skills with less training.

NalNezumi 3y ago
They use the RT1 data which is (if I remember correctly) 11 month of data using like 6 research engineers sitting everyday, remote controlling the robot. They might even use ROSIE data which is data-augmentation using Stable-diffusion inpainting. And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result. as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =) [1] https://youtu.be/x10964w00zk https://youtu.be/x10964w00zk
- blovescoffee 3y ago
  Sure, I wouldn't say reasoning can guide a robot to do gymnastics zero-shot either. But it is a substantiated approach to generalizing existing capabilities - just like in humans. You or a robot can't reason your way to bike-riding but if you know how to bake cookies or brush your teeth, you can perform much better zero-shot on related tasks like baking a chicken or brushing a mirror.

NalNezumi 3y ago

They use the RT1 data which is (if I remember correctly) 11 month of data using like 6 research engineers sitting everyday, remote controlling the robot. They might even use ROSIE data which is data-augmentation using Stable-diffusion inpainting. And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result. as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =) [1] https://youtu.be/x10964w00zk https://youtu.be/x10964w00zk

blovescoffee 3y ago
Sure, I wouldn't say reasoning can guide a robot to do gymnastics zero-shot either. But it is a substantiated approach to generalizing existing capabilities - just like in humans. You or a robot can't reason your way to bike-riding but if you know how to bake cookies or brush your teeth, you can perform much better zero-shot on related tasks like baking a chicken or brushing a mirror.