5 ms·

While this is very impressive results, they’re not actually having the models do any of the robotic controls. They only have the LLMs output text of what the co

While this is very impressive results, they’re not actually having the models do any of the robotic controls. They only have the LLMs output text of what the controls should be, which then gets translated to actual movements. This works in a very narrow, limited set of experiments. 562 B parameters to output text commands seems excessive

aipushup 4y ago
agreed, to me it's a quite misleading claim.
nopinsight 4y ago
In many cases, knowing what to do in very diverse environments could be harder than the physical control of the robots. I've read somewhere that the physical manipulation of robot body is not as hard as understanding multimodal info (linguistic/visual/auditory), esp in real time. > Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. The last clause is important and suggests there might be further advances from such joint training, incl from interactions in the real world.
karmasimida 4y ago
Using LLM to translate the end result (text) into controls could be just a form of code generation, for which LLM are superb at. I think a critical piece is being figured out as more and more such research emerges.
- andreyk 4y ago
  FYI, translation to code was actually already done last year by Google (Code as Policies - https://lastweekin.ai/p/code-as-policies https://lastweekin.ai/p/code-as-policies) - but this still assumes a set of low level primitives, not tight visuomotor control. You really need reactive policies with tight feedback loops in many robotics domains, and that's kind of hard with these super gigantic models.
eloisius 4y ago
I was showerthinking about this the other day. Would it be possible to use something like ChatGPT to learn a hardware driver protocol? Especially a plain text one like modem AT commands.
- fassssst 4y ago
  Yes, it can already write G-code for CNC machines. A multi-modal model that understands shapes and space would do way better.
robitsT6 4y ago
Reading the paper, they mention the robotic control is handled by RT-1: The low-level policies are from RT-1 (Brohan et al., 2022), a transformer model that takes RGB image and natural language instruction, and outputs end-effector control commands. For those that don't know, RT-1 (Robotic transformer) is previous work from the team that converts natural language to custom control code. You can read more about Rt-1 here: https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-for-real.html?m=1 https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-... Maybe I'm missing something, but this sounds quite generalizable.
- polygamous_bat 4y ago
  RT-1 is trained with very specific (and a lot of) mostly pick-and-place data. That's the domain it is an expert on. Unfortunately, there are only so many things you can build with go to/pick up/place level of instructions. Anything further that goes into the fine manipulation domain that you may need in a real kitchen is still absent. This issue was a general disappointment in the robotics community; they had a LOT of funds to get robotics data with and they spent it on somewhat trivial tasks that we had almost solved already with much smaller and more principled models, instead of getting human demonstration data for more complicated tasks.
  - robitsT6 4y ago
    Right but is there any reason that this architecture won't work with more diverse data? Fundamentally it seems like their research is benchmarked around pick and place, so it makes sense to me that they would want to prove out transformer models could work in this space. Knowing transformers, it's probably safe to say that with more diverse training data, it will be able to scale to more complex controls in more complex embodied robots. While I would love to see them working on robot chefs, I can appreciate that they want to start small. And regardless, it doesn't seem like there are any constraints outside of data for this architecture to work in more and more domains.
    - polygamous_bat 4y ago
brookst 4y ago
Couldn’t this be said of any software controlling a robot? The software isn’t really controlling anything, it’s just telling the motors what to do? IMO the logic making the decision of what to do is by definition controlling the robot, even if it’s intermediated by subsystems and electrical components that translate commands to movement.
- polygamous_bat 4y ago
  Except the jump from robot logic to robot control is a damned difficult one. Even if you have a human teleportating the robot, it's multiple orders of magnitude more expensive to get robot controls than to get robot logic in terms of both time and money.

aipushup 4y ago

agreed, to me it's a quite misleading claim.

nopinsight 4y ago

In many cases, knowing what to do in very diverse environments could be harder than the physical control of the robots. I've read somewhere that the physical manipulation of robot body is not as hard as understanding multimodal info (linguistic/visual/auditory), esp in real time. > Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. The last clause is important and suggests there might be further advances from such joint training, incl from interactions in the real world.

karmasimida 4y ago

Using LLM to translate the end result (text) into controls could be just a form of code generation, for which LLM are superb at. I think a critical piece is being figured out as more and more such research emerges.

andreyk 4y ago
FYI, translation to code was actually already done last year by Google (Code as Policies - https://lastweekin.ai/p/code-as-policies https://lastweekin.ai/p/code-as-policies) - but this still assumes a set of low level primitives, not tight visuomotor control. You really need reactive policies with tight feedback loops in many robotics domains, and that's kind of hard with these super gigantic models.

eloisius 4y ago

I was showerthinking about this the other day. Would it be possible to use something like ChatGPT to learn a hardware driver protocol? Especially a plain text one like modem AT commands.

fassssst 4y ago
Yes, it can already write G-code for CNC machines. A multi-modal model that understands shapes and space would do way better.

robitsT6 4y ago

Reading the paper, they mention the robotic control is handled by RT-1: The low-level policies are from RT-1 (Brohan et al., 2022), a transformer model that takes RGB image and natural language instruction, and outputs end-effector control commands. For those that don't know, RT-1 (Robotic transformer) is previous work from the team that converts natural language to custom control code. You can read more about Rt-1 here: https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-for-real.html?m=1 https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-... Maybe I'm missing something, but this sounds quite generalizable.

polygamous_bat 4y ago
RT-1 is trained with very specific (and a lot of) mostly pick-and-place data. That's the domain it is an expert on. Unfortunately, there are only so many things you can build with go to/pick up/place level of instructions. Anything further that goes into the fine manipulation domain that you may need in a real kitchen is still absent. This issue was a general disappointment in the robotics community; they had a LOT of funds to get robotics data with and they spent it on somewhat trivial tasks that we had almost solved already with much smaller and more principled models, instead of getting human demonstration data for more complicated tasks.
- robitsT6 4y ago
  Right but is there any reason that this architecture won't work with more diverse data? Fundamentally it seems like their research is benchmarked around pick and place, so it makes sense to me that they would want to prove out transformer models could work in this space. Knowing transformers, it's probably safe to say that with more diverse training data, it will be able to scale to more complex controls in more complex embodied robots. While I would love to see them working on robot chefs, I can appreciate that they want to start small. And regardless, it doesn't seem like there are any constraints outside of data for this architecture to work in more and more domains.
  - polygamous_bat 4y ago

brookst 4y ago

Couldn’t this be said of any software controlling a robot? The software isn’t really controlling anything, it’s just telling the motors what to do? IMO the logic making the decision of what to do is by definition controlling the robot, even if it’s intermediated by subsystems and electrical components that translate commands to movement.

polygamous_bat 4y ago
Except the jump from robot logic to robot control is a damned difficult one. Even if you have a human teleportating the robot, it's multiple orders of magnitude more expensive to get robot controls than to get robot logic in terms of both time and money.