6 ms·

Learning bimanual mobile manipulation with low-cost whole-body teleoperation

126 points by tristenharr 3y ago

evanjrowley 3y ago
The video recordings give me the impression that piloting this robot to perform mundane household tasks is much more fun than doing so in real life.
- cracrecry 3y ago
  Doing it in real life ONCE is quite fun. The great things about computers is that you can solve a problem once and let the machine repeat it. Having to do it everyday for years and not being able to do other things because of that is not fun at all.
  - jareklupinski 3y ago
    with machine vision, it might be feasible to change the learning step to where instead of using joysticks to manipulate the robot through the steps, it can watch you 'do it once' then extrapolate the kinematics for its body just based on the end states of your hands or something
tristenharr 3y ago
The coolest thing to me was watching it do laundry in the video it takes a jacket and puts it on a hanger and then zips the jacket. Watching it get the zipper was pretty satisfying. It’s also amazing that there’s just instructions and it’s open to anyone. Kudos to the builders. They sauté shrimp, do laundry, and wash dishes with this robot you can build at home. If that isn’t cool I don’t know what is.
- actionfromafar 3y ago
  It needs easy to wash clothes which it then washes when it gets dirty after frying food.
  - greggsy 3y ago
    A smock would be most practical I think.
jamestimmins 3y ago
Can someone with expertise explain why this is so difficult? As an outsider with no knowledge of robotics, I've always been surprised that manipulation tasks (or just smooth robotic movement) are so challenging and seem to progress so slowly, especially when compared to IT.
- talldrinkofwhat 3y ago
  I'm about a decade out, so I'm sure the SOTA has moved the goal-posts, but the work I did in grad(ish) school revolved around making end-effectors (fancy word for robot hands) that could both grab a styrofoam cup, and grab a wet glass (cylindrical, no pint-cheaters). Plenty of cups were crushed. Plenty of glasses were dropped. Nothing that did both happened by the time I graduated. The amount of feedback built into your end-effectors (pedantic word for human hands) is insane. If you're not familiar, proprioception is a good google/wiki hole. Most of the signals that allow you to move your hands don't even hit the brain stem, let alone the boss upstairs. The challenge mostly lies in how we've instrumented these things. Precision requires low tolerances. Low tolerances + unexpected environment == you've just driven your robot through the countertop/pan/coworker or broken a very nicely geared servo.
  - zith 3y ago
    Adding to that, even if we had perfect end-effectors with a good sense of touch, understanding the real world enough to manipulate it is hard. These days we have 3d cameras, but they still only see part of the objects we want to manipulate. The back side is hidden. So you need to either specify and model all objects to interact with, or have some word of a world model where we can predict what the full object, it's weight, center of gravity, surface texture, etc, is like. And before we even decide to manipulate it, we have to detect it, categorize it and segment it (where does the pan stop and the stove begin?). We have to plan out a manipulation task, including finding grasp points, finding movement patterns that do not interfere with the rest of the environment, etc. It's a whole bunch of separate problems that need solving all at once. There's motor control, building the right manipulators with the right sensors, bringing all the sensor data into something where we can make a single decision, understanding of the world and what happens during manipulation, and higher level planning.
canadiantim 3y ago
ah I thought we were this close |---| to "low-cost whole-body teleportation".
- Cilvic 3y ago
  I came for the same reason, but the article is still cool :-)
- ClassyJacket 3y ago
  Oh god, me too. I even started writing a comment asking what teleportation mens in this context.
rattatossk 3y ago
I can just see it now. Massacre. All the oil from the pan in the plate. Set the hot pan down on the counter top.
- 1letterunixname 3y ago
  And an oil fire and an inability for the robot to put it out.
regularfry 3y ago
Well, they got the most critical component correct: an appropriate number of googly eyes is vital for this sort of application.
YeGoblynQueenne 3y ago
That's all kind of pathetically underwhelming. The video is at 6x speed and the robot is shown pushing some chairs forward, placing a pot in an empty cupboard and ... frying one prawn? One prawn? All those are toy tasks that have no application in a real environment, even the constrained and save environment of a home. Most homes don't have such large empty spaces. Most of the time when you need to tidy up a bunch of chairs they're not put in a neat straight line by a RA, they're left in a jumbled mess by a stampede of students and you have to do a lot more pushing and pulling and turning around (and there's tables and possibly empty cups and stuff). Most of the time you need to cook a lot more than one measly prawn. And let's not talk about the primordial chaos of kitchen cupboards. What's worse with all those demonstrations: the robots can only do exactly what you see in the video. Change the parameters even slightly: different shape pan, different height cupboard, different room configuration; and the magick -poof- vanishes, into thin air. That stuff doesn't work. We aren't even close to solving autonomous robotic behaviour. RL doesn't work and the older techniques aren't working either (planning). All that stuff ever does is get published into papers, advertised with fanfare and then forgotten because it never makes it to the real world, because it's all unreliable and unpredictable and costs too much if you want to do anything real, and that's always something trivial. So the state-of-the-art in robotics is in hand-coded industrial robots that do one thing and do it over and over again and nobody asks them to generalise, or to come into your house and cook you a prawn.
- nmca 3y ago
  It's a lot better than last year, and techniques seem very general.
  - YeGoblynQueenne 3y ago
    Not general at all. In all the object manipulation tasks the objects to be manipulated have been placed just so. If you change the placement (or the objects), you need to retrain. And it would be extremely computationally expensive to get the robot to learn to carry out the whole task on its own, from finding and retrieving, e.g. the shrimp, the oil, the plate, the slotted spoon and the pan, one by one, before having to use them all in the right order. That (hierarchical task planning) is a problem that is not yet solved (because it's combinatorially hard) and eliminates any idea of "general" ability: you have to set things up perfectly before the robot can do anything. Exactly like for industrial robots. It just looks more general because it's inside a house rather than a factory.
1letterunixname 3y ago
Baby steps. I wouldn't give a human-operated robot control of dangerous tasks now, but some less dangerous tasks would be okay. Eventually, through iteration, the interfaces, robots, and operators will improve to where full body teleoperation, AI-assisted, and AI-operated robots will be achieved.
btbuildem 3y ago
Look, it's the perfect roommate! Jokes aside -- something like this could be a fantastic breakthrough for people who've lost the use of their limbs. A wearable / chair-attached exo, driven to do complex tasks through simple commands.
- bloopernova 3y ago
  It would be fantastic for my wife, who often cannot stand for longer than a minute or two. If I understand it correctly, it learns from "watching" you during telepresence sessions? If so, and if it learns fast enough, it really would be a massive boon to some disabled people. Although I want to see it peel potatoes and dice onions :)
  - btbuildem 3y ago
    Even if it can't do tasks that require more dexterity, it would still be a great help. The "rinse pan" demo did not impress - but it doesn't need to. You don't have to teach it to "scrub bowls and plates" when you can teach it to "fill and empty the dishwasher"
    - bloopernova 3y ago
      Yeah exactly, or if it can lift things that would otherwise cause her to dislocate joints. I'm definitely excited for this tech.
xg15 3y ago
I was wondering if at some point an evolution of "cloud kitchens" could be "edge kitchens": Imagine a high-end apartment or condo building. In the basement of the building, there is a fully equipped professional kitchen - however, it's not supposed to be used by humans. Instead, the kitchen is fully automated with smart/wired-up devices wherever possible. The actual chopping, cooking etc is performed by robotic arms like the ones in the OP. The kitchen has a human-accessible storeroom on one end (that must be regularly restocked by the condo administration) and is connected to the apartments by a system of service lifts. Residents can order food through a an appliance or an app or something and choose from some menu of predefined meals (or possibly even create their own recipes through some sort of building block system). Once they submitted the order (and all ingredients are available), the kitchen springs into action, prepares the dish and places it in a service elevator. In the apartment, the dish appears, fresh and hot, like right out of a Star Trek replicator. I guess we've just gotten one step closer to that vision :)
deleted 3y ago
[deleted]
Wingman4l7 3y ago
...or they'll just cut out the behavior training part, and we'll have Sleep Dealer: https://en.wikipedia.org/wiki/Sleep_Dealer https://en.wikipedia.org/wiki/Sleep_Dealer
YeGoblynQueenne 3y ago
Just for some context here's a video of "outtakes" from the authors: https://twitter.com/tonyzzhao/status/1743378437174366715 https://twitter.com/tonyzzhao/status/1743378437174366715 In another tweet [1] the authors give a count of the successful (but not the failed) attempts at various tasks: Our robot can consistently handle these tasks, succeeding: - 9 times in a row for Wipe Wine - 5 times for Call Elevator - robust against distractors for Use Cabinet - extrapolate to chairs unseen during training https://twitter.com/zipengfu/status/1742602883256943040 https://twitter.com/zipengfu/status/1742602883256943040 ___________________ [1] How do you call those now? Exes?
- deleted 3y ago
  [deleted]

126 points by tristenharr 3y ago

The video recordings give me the impression that piloting this robot to perform mundane household tasks is much more fun than doing so in real life.

cracrecry 3y ago
Doing it in real life ONCE is quite fun. The great things about computers is that you can solve a problem once and let the machine repeat it. Having to do it everyday for years and not being able to do other things because of that is not fun at all.
- jareklupinski 3y ago
  with machine vision, it might be feasible to change the learning step to where instead of using joysticks to manipulate the robot through the steps, it can watch you 'do it once' then extrapolate the kinematics for its body just based on the end states of your hands or something

tristenharr 3y ago

The coolest thing to me was watching it do laundry in the video it takes a jacket and puts it on a hanger and then zips the jacket. Watching it get the zipper was pretty satisfying. It’s also amazing that there’s just instructions and it’s open to anyone. Kudos to the builders. They sauté shrimp, do laundry, and wash dishes with this robot you can build at home. If that isn’t cool I don’t know what is.

actionfromafar 3y ago
It needs easy to wash clothes which it then washes when it gets dirty after frying food.
- greggsy 3y ago
  A smock would be most practical I think.

jamestimmins 3y ago

Can someone with expertise explain why this is so difficult? As an outsider with no knowledge of robotics, I've always been surprised that manipulation tasks (or just smooth robotic movement) are so challenging and seem to progress so slowly, especially when compared to IT.

talldrinkofwhat 3y ago
I'm about a decade out, so I'm sure the SOTA has moved the goal-posts, but the work I did in grad(ish) school revolved around making end-effectors (fancy word for robot hands) that could both grab a styrofoam cup, and grab a wet glass (cylindrical, no pint-cheaters). Plenty of cups were crushed. Plenty of glasses were dropped. Nothing that did both happened by the time I graduated. The amount of feedback built into your end-effectors (pedantic word for human hands) is insane. If you're not familiar, proprioception is a good google/wiki hole. Most of the signals that allow you to move your hands don't even hit the brain stem, let alone the boss upstairs. The challenge mostly lies in how we've instrumented these things. Precision requires low tolerances. Low tolerances + unexpected environment == you've just driven your robot through the countertop/pan/coworker or broken a very nicely geared servo.
- zith 3y ago
  Adding to that, even if we had perfect end-effectors with a good sense of touch, understanding the real world enough to manipulate it is hard. These days we have 3d cameras, but they still only see part of the objects we want to manipulate. The back side is hidden. So you need to either specify and model all objects to interact with, or have some word of a world model where we can predict what the full object, it's weight, center of gravity, surface texture, etc, is like. And before we even decide to manipulate it, we have to detect it, categorize it and segment it (where does the pan stop and the stove begin?). We have to plan out a manipulation task, including finding grasp points, finding movement patterns that do not interfere with the rest of the environment, etc. It's a whole bunch of separate problems that need solving all at once. There's motor control, building the right manipulators with the right sensors, bringing all the sensor data into something where we can make a single decision, understanding of the world and what happens during manipulation, and higher level planning.

canadiantim 3y ago

ah I thought we were this close |---| to "low-cost whole-body teleportation".

Cilvic 3y ago
I came for the same reason, but the article is still cool :-)
ClassyJacket 3y ago
Oh god, me too. I even started writing a comment asking what teleportation mens in this context.

rattatossk 3y ago

I can just see it now. Massacre. All the oil from the pan in the plate. Set the hot pan down on the counter top.

1letterunixname 3y ago
And an oil fire and an inability for the robot to put it out.

regularfry 3y ago

Well, they got the most critical component correct: an appropriate number of googly eyes is vital for this sort of application.

YeGoblynQueenne 3y ago

That's all kind of pathetically underwhelming. The video is at 6x speed and the robot is shown pushing some chairs forward, placing a pot in an empty cupboard and ... frying one prawn? One prawn? All those are toy tasks that have no application in a real environment, even the constrained and save environment of a home. Most homes don't have such large empty spaces. Most of the time when you need to tidy up a bunch of chairs they're not put in a neat straight line by a RA, they're left in a jumbled mess by a stampede of students and you have to do a lot more pushing and pulling and turning around (and there's tables and possibly empty cups and stuff). Most of the time you need to cook a lot more than one measly prawn. And let's not talk about the primordial chaos of kitchen cupboards. What's worse with all those demonstrations: the robots can only do exactly what you see in the video. Change the parameters even slightly: different shape pan, different height cupboard, different room configuration; and the magick -poof- vanishes, into thin air. That stuff doesn't work. We aren't even close to solving autonomous robotic behaviour. RL doesn't work and the older techniques aren't working either (planning). All that stuff ever does is get published into papers, advertised with fanfare and then forgotten because it never makes it to the real world, because it's all unreliable and unpredictable and costs too much if you want to do anything real, and that's always something trivial. So the state-of-the-art in robotics is in hand-coded industrial robots that do one thing and do it over and over again and nobody asks them to generalise, or to come into your house and cook you a prawn.

nmca 3y ago
It's a lot better than last year, and techniques seem very general.
- YeGoblynQueenne 3y ago
  Not general at all. In all the object manipulation tasks the objects to be manipulated have been placed just so. If you change the placement (or the objects), you need to retrain. And it would be extremely computationally expensive to get the robot to learn to carry out the whole task on its own, from finding and retrieving, e.g. the shrimp, the oil, the plate, the slotted spoon and the pan, one by one, before having to use them all in the right order. That (hierarchical task planning) is a problem that is not yet solved (because it's combinatorially hard) and eliminates any idea of "general" ability: you have to set things up perfectly before the robot can do anything. Exactly like for industrial robots. It just looks more general because it's inside a house rather than a factory.

1letterunixname 3y ago

Baby steps. I wouldn't give a human-operated robot control of dangerous tasks now, but some less dangerous tasks would be okay. Eventually, through iteration, the interfaces, robots, and operators will improve to where full body teleoperation, AI-assisted, and AI-operated robots will be achieved.

btbuildem 3y ago

Look, it's the perfect roommate! Jokes aside -- something like this could be a fantastic breakthrough for people who've lost the use of their limbs. A wearable / chair-attached exo, driven to do complex tasks through simple commands.

bloopernova 3y ago
It would be fantastic for my wife, who often cannot stand for longer than a minute or two. If I understand it correctly, it learns from "watching" you during telepresence sessions? If so, and if it learns fast enough, it really would be a massive boon to some disabled people. Although I want to see it peel potatoes and dice onions :)
- btbuildem 3y ago
  Even if it can't do tasks that require more dexterity, it would still be a great help. The "rinse pan" demo did not impress - but it doesn't need to. You don't have to teach it to "scrub bowls and plates" when you can teach it to "fill and empty the dishwasher"
  - bloopernova 3y ago
    Yeah exactly, or if it can lift things that would otherwise cause her to dislocate joints. I'm definitely excited for this tech.

xg15 3y ago

I was wondering if at some point an evolution of "cloud kitchens" could be "edge kitchens": Imagine a high-end apartment or condo building. In the basement of the building, there is a fully equipped professional kitchen - however, it's not supposed to be used by humans. Instead, the kitchen is fully automated with smart/wired-up devices wherever possible. The actual chopping, cooking etc is performed by robotic arms like the ones in the OP. The kitchen has a human-accessible storeroom on one end (that must be regularly restocked by the condo administration) and is connected to the apartments by a system of service lifts. Residents can order food through a an appliance or an app or something and choose from some menu of predefined meals (or possibly even create their own recipes through some sort of building block system). Once they submitted the order (and all ingredients are available), the kitchen springs into action, prepares the dish and places it in a service elevator. In the apartment, the dish appears, fresh and hot, like right out of a Star Trek replicator. I guess we've just gotten one step closer to that vision :)

deleted 3y ago

[deleted]

Wingman4l7 3y ago

...or they'll just cut out the behavior training part, and we'll have Sleep Dealer: https://en.wikipedia.org/wiki/Sleep_Dealer https://en.wikipedia.org/wiki/Sleep_Dealer

YeGoblynQueenne 3y ago

Just for some context here's a video of "outtakes" from the authors: https://twitter.com/tonyzzhao/status/1743378437174366715 https://twitter.com/tonyzzhao/status/1743378437174366715 In another tweet [1] the authors give a count of the successful (but not the failed) attempts at various tasks: Our robot can consistently handle these tasks, succeeding: - 9 times in a row for Wipe Wine - 5 times for Call Elevator - robust against distractors for Use Cabinet - extrapolate to chairs unseen during training https://twitter.com/zipengfu/status/1742602883256943040 https://twitter.com/zipengfu/status/1742602883256943040 ___________________ [1] How do you call those now? Exes?

deleted 3y ago
[deleted]