4 ms·

This seem like an cool upgrade from RT1, judging from the result. It seems to now also output the delta of the end-effector pose, which was previously handled b

This seem like an cool upgrade from RT1, judging from the result. It seems to now also output the delta of the end-effector pose, which was previously handled by a different motion planner. It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop with a motion planner. This seem to dramatically lower the data requirement but it feel like a fundamental limit to what task we can accomplish. The reason robot manipulation is hard is because we need to take in to the account not just what's happening in the world but also how our interaction alters it and how we need to react to that. Robot Learning right now is either Reinforcement Learning or Imitation Learning (often latter) and I'm not sure how one would collect data that capture this. Edit: I'm surprised Google still do robotics work, I thought Everyday Robotics was shut down [1] [1] https://www.therobotreport.com/alphabet-closes-everyday-robots-among-layoffs/ https://www.therobotreport.com/alphabet-closes-everyday-robo...

empath-nirvana 3y ago
Seems to me that the way to architect this is to have multiple quasi independent embedded controllers at different levels. For example, you might have 3 independent finger controllers, managed by a hand controller, and so on, on up to the highest level LLM that drives everything. So you have an LLM that just says, pick up the green can, issues whatever structured data needs to go to the arm controller and on down the line, going to down to more real-time and less high level processing as you go. As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. The fingers don't need to know how the feet work, for example. There should be semi-autonomous "intelligence" embedded throughout the system, with only necessary feedback being fed back up.
- RecycledEle 3y ago
  I agree. We need feedback loops on each joint that keep doing what they are doing until a higher feedback loop decides to change things. For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it. The low level controllers should also be able to respond to simple things. It's like when I get hurt, my hand yanks back before my brain realizes what has happened. Or when I almost trip and my legs correct before I realized it happened. Low level tasks should be at a low level and not bother the higher professors.
  - RecycledEle 3y ago
    LLMs have terrible spatial awareness. This probably comes from ONLY being trained on text. I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly. It would be like when a video game has a radar display or map to let you know where you are in relation to other things.
    - moffkalast 3y ago
      I actually tried that a while back, giving 3.5-turbo a multishot prompt that consisted of distance readings for ahead, left, right and back in an array, as extracted from lidar data, then giving it movement instructions. It performed rather terribly. You've very much correct that their spatial awareness is terrible. Something as simple as drive forward, then back, turn left, etc. works just fine and they can generally translate it to a specified message format reasonably reliably, but give them something more complex to execute, like drive a robot in a square pattern (an example answer would be go forward, turn right, go forward, turn right, etc.) they start to generate nonsense. I also tested it with the 30B WizardLM at the time which performed almost as well in terms of message format but had even worse awareness. Part of the problem is that the training data contains next to no examples that would teach it how 3D space works. I considered making a dataset of driving a robot around with human movement commands and then logging the aggregated sensor data and commands for fine tuning so the prompt format would be pre-learned, but I'm not entirely sure how much it would help.
- NalNezumi 3y ago
  That is indeed what a lot of Machine Learning turned Robotics researchers/enthusiasts are banking on. A counter argument to that is what Dhruv Batra responded to the question "lol why not use LLM for everything" [1]. >As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. This is true for us ofc, and is encompassed in what is called the "Moravec's paradox" [2]. You don't understand it because it's unconscious process, and It's harder to reverse-engineer an unconscious process (motor movement) than conscious ones (calculating math, playing game, writing text, reading). But the thing is that in the real world we do need to take in to account everything, including noise and time-delay. Evolution gave rise to complex language in the last 100k years compared to millions of year for motor movement. I do agree that there must be some "hierarchical" structure for complex motion, but we currently don't know where and how that hierarchy is. Boston dynamics uses Model Predictive Control for complex movement, which means that at least some model of the world is required, for dexterous motion. Now if that model is part of LLM or not is a hard guess. But if we don't know this it's hard to say what kind of data we need to collect to train a LLM-model applied to embodiment (robotics). Researchers in the past have already made the mistaken assumption of "oh cognition and language is the hard part of intelligence. Perception and motion is easy" [3] and then their work amounted to nothing because turns out the latter was way, wayyyyyy harder. There's an implicit bias in us that think Language, puzzles and logic are harder [2] and therefore models that accomplish this can just be rammed in to the "easier" issues. Edit: I too would like "LLM models will solve these" attitude because otherwise the research I'm doing right now is a dead-end, but the more I try (with my limited compute) the less I'm sure [1] https://imgur.com/eWsH5ui https://imgur.com/eWsH5ui originally https://twitter.com/DhruvBatraDB/status/1641871357020614656 https://twitter.com/DhruvBatraDB/status/1641871357020614656 [2] https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox [3] https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZL8CACYMUqAT&t=3358 https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZL8C...
  - whinenot 3y ago
    >It's harder to reverse-engineer an unconscious process Aside from some basic life support systems, don't almost all movements start with conscious effort? Whether you are deliberate about the exercise or not, you practice and practice until you develop 'muscle memory' where it becomes unconscious: walking, dribbling a basketball, holding a G chord on a guitar, etc.
- jimbokun 3y ago
  Maybe the octopus is a more tractable model for robotic control. I understand that octopus neurons are not as concentrated in a central brain, but spread throughout its limbs that are autonomous compared to humans.
- StackOverlord 3y ago
  This is in fact what happens in the human body. For example, when you reach out to pick up a green can, your brain makes the decision to do the task but it's your spinal cord and peripheral nerves that carry out the detailed work – orienting the hand, managing grasp strength, controlling the arm movements etc. This process is mostly unconscious – you don't need to actively think about how to tense each muscle in the same way that an embedded controller wouldn't need to understanding the working of the entire robotic system to carry out its specific task. Much like the model suggested, the human body communicates feedback across layers — this process is crucial to maintaining balance, coordination and effectively reacting to the environment. For instance, if your fingers touch a hot stove, the sensory receptors in your skin will immediately send a signal to your spinal cord and a reflex action will make you pull your hand back even before you consciously perceive that the stove is hot.
blovescoffee 3y ago
They collect data for RL/IL in simulation which _can_ generalize to the real world. Also, being Google, they have the resources to collect data by brute force i.e. scientists manually collecting that data. The paper says - one main source of the data is internet scale vision/llm data. The second source is 6k trial runs. A principal idea behind this work is that you can collect data in one domain to avoid collecting data in another. Training an LLM can train the 'reasoning' portion of the robot so that it can perform real-world skills with less training.
- NalNezumi 3y ago
  They use the RT1 data which is (if I remember correctly) 11 month of data using like 6 research engineers sitting everyday, remote controlling the robot. They might even use ROSIE data which is data-augmentation using Stable-diffusion inpainting. And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result. as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =) [1] https://youtu.be/x10964w00zk https://youtu.be/x10964w00zk
  - blovescoffee 3y ago
    Sure, I wouldn't say reasoning can guide a robot to do gymnastics zero-shot either. But it is a substantiated approach to generalizing existing capabilities - just like in humans. You or a robot can't reason your way to bike-riding but if you know how to bake cookies or brush your teeth, you can perform much better zero-shot on related tasks like baking a chicken or brushing a mirror.
deleted 3y ago
[deleted]
lucidrains 3y ago
thank you for sharing your thoughts! is there anyway I can get in touch with you, through email or some other means?
- NalNezumi 3y ago
  Sure thing! I'm a bit wary to share my email on public forum but you can DM me on Twitter! https://twitter.com/DanOgawaLil https://twitter.com/DanOgawaLil
  - deleted 3y ago
    [deleted]

empath-nirvana 3y ago

Seems to me that the way to architect this is to have multiple quasi independent embedded controllers at different levels. For example, you might have 3 independent finger controllers, managed by a hand controller, and so on, on up to the highest level LLM that drives everything. So you have an LLM that just says, pick up the green can, issues whatever structured data needs to go to the arm controller and on down the line, going to down to more real-time and less high level processing as you go. As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. The fingers don't need to know how the feet work, for example. There should be semi-autonomous "intelligence" embedded throughout the system, with only necessary feedback being fed back up.

RecycledEle 3y ago
I agree. We need feedback loops on each joint that keep doing what they are doing until a higher feedback loop decides to change things. For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it. The low level controllers should also be able to respond to simple things. It's like when I get hurt, my hand yanks back before my brain realizes what has happened. Or when I almost trip and my legs correct before I realized it happened. Low level tasks should be at a low level and not bother the higher professors.
- RecycledEle 3y ago
  LLMs have terrible spatial awareness. This probably comes from ONLY being trained on text. I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly. It would be like when a video game has a radar display or map to let you know where you are in relation to other things.
  - moffkalast 3y ago
    I actually tried that a while back, giving 3.5-turbo a multishot prompt that consisted of distance readings for ahead, left, right and back in an array, as extracted from lidar data, then giving it movement instructions. It performed rather terribly. You've very much correct that their spatial awareness is terrible. Something as simple as drive forward, then back, turn left, etc. works just fine and they can generally translate it to a specified message format reasonably reliably, but give them something more complex to execute, like drive a robot in a square pattern (an example answer would be go forward, turn right, go forward, turn right, etc.) they start to generate nonsense. I also tested it with the 30B WizardLM at the time which performed almost as well in terms of message format but had even worse awareness. Part of the problem is that the training data contains next to no examples that would teach it how 3D space works. I considered making a dataset of driving a robot around with human movement commands and then logging the aggregated sensor data and commands for fine tuning so the prompt format would be pre-learned, but I'm not entirely sure how much it would help.
NalNezumi 3y ago
That is indeed what a lot of Machine Learning turned Robotics researchers/enthusiasts are banking on. A counter argument to that is what Dhruv Batra responded to the question "lol why not use LLM for everything" [1]. >As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when I want it to happen. I don't think you need to design a robotic AI that understands how every part of it's mechanics work. This is true for us ofc, and is encompassed in what is called the "Moravec's paradox" [2]. You don't understand it because it's unconscious process, and It's harder to reverse-engineer an unconscious process (motor movement) than conscious ones (calculating math, playing game, writing text, reading). But the thing is that in the real world we do need to take in to account everything, including noise and time-delay. Evolution gave rise to complex language in the last 100k years compared to millions of year for motor movement. I do agree that there must be some "hierarchical" structure for complex motion, but we currently don't know where and how that hierarchy is. Boston dynamics uses Model Predictive Control for complex movement, which means that at least some model of the world is required, for dexterous motion. Now if that model is part of LLM or not is a hard guess. But if we don't know this it's hard to say what kind of data we need to collect to train a LLM-model applied to embodiment (robotics). Researchers in the past have already made the mistaken assumption of "oh cognition and language is the hard part of intelligence. Perception and motion is easy" [3] and then their work amounted to nothing because turns out the latter was way, wayyyyyy harder. There's an implicit bias in us that think Language, puzzles and logic are harder [2] and therefore models that accomplish this can just be rammed in to the "easier" issues. Edit: I too would like "LLM models will solve these" attitude because otherwise the research I'm doing right now is a dead-end, but the more I try (with my limited compute) the less I'm sure [1] https://imgur.com/eWsH5ui https://imgur.com/eWsH5ui originally https://twitter.com/DhruvBatraDB/status/1641871357020614656 https://twitter.com/DhruvBatraDB/status/1641871357020614656 [2] https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox [3] https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZL8CACYMUqAT&t=3358 https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZL8C...
- whinenot 3y ago
  >It's harder to reverse-engineer an unconscious process Aside from some basic life support systems, don't almost all movements start with conscious effort? Whether you are deliberate about the exercise or not, you practice and practice until you develop 'muscle memory' where it becomes unconscious: walking, dribbling a basketball, holding a G chord on a guitar, etc.
jimbokun 3y ago
Maybe the octopus is a more tractable model for robotic control. I understand that octopus neurons are not as concentrated in a central brain, but spread throughout its limbs that are autonomous compared to humans.
StackOverlord 3y ago
This is in fact what happens in the human body. For example, when you reach out to pick up a green can, your brain makes the decision to do the task but it's your spinal cord and peripheral nerves that carry out the detailed work – orienting the hand, managing grasp strength, controlling the arm movements etc. This process is mostly unconscious – you don't need to actively think about how to tense each muscle in the same way that an embedded controller wouldn't need to understanding the working of the entire robotic system to carry out its specific task. Much like the model suggested, the human body communicates feedback across layers — this process is crucial to maintaining balance, coordination and effectively reacting to the environment. For instance, if your fingers touch a hot stove, the sensory receptors in your skin will immediately send a signal to your spinal cord and a reflex action will make you pull your hand back even before you consciously perceive that the stove is hot.

blovescoffee 3y ago

They collect data for RL/IL in simulation which _can_ generalize to the real world. Also, being Google, they have the resources to collect data by brute force i.e. scientists manually collecting that data. The paper says - one main source of the data is internet scale vision/llm data. The second source is 6k trial runs. A principal idea behind this work is that you can collect data in one domain to avoid collecting data in another. Training an LLM can train the 'reasoning' portion of the robot so that it can perform real-world skills with less training.

NalNezumi 3y ago
They use the RT1 data which is (if I remember correctly) 11 month of data using like 6 research engineers sitting everyday, remote controlling the robot. They might even use ROSIE data which is data-augmentation using Stable-diffusion inpainting. And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent map. Although the latter is solved with SLAM). it is way, way more versatile (if their claims are correct) so this is cool result. as for sim2real, that's a whole different set of issue.... Simulator have come a long way, but from last time I talked with team-lead of Mujoco sim, it seems like completely realistic ones (at a reasonable compute time) is still far away. Deform-able objects are hard to simulate My point is just that the "reasoning" you mention have a limit;[1] there's no amount of reasoning with language that can zero-shot a robot to do gymnastics or ride a bide. I'd be more than happy to be proven wrong on this point though, then my line of research is not dead-end =) [1] https://youtu.be/x10964w00zk https://youtu.be/x10964w00zk
- blovescoffee 3y ago
  Sure, I wouldn't say reasoning can guide a robot to do gymnastics zero-shot either. But it is a substantiated approach to generalizing existing capabilities - just like in humans. You or a robot can't reason your way to bike-riding but if you know how to bake cookies or brush your teeth, you can perform much better zero-shot on related tasks like baking a chicken or brushing a mirror.

deleted 3y ago

[deleted]

lucidrains 3y ago

thank you for sharing your thoughts! is there anyway I can get in touch with you, through email or some other means?

NalNezumi 3y ago
Sure thing! I'm a bit wary to share my email on public forum but you can DM me on Twitter! https://twitter.com/DanOgawaLil https://twitter.com/DanOgawaLil
- deleted 3y ago
  [deleted]