3 ms·

More videos: Hanging multiple shirts in a row: https://twitter.com/ayzwah/status/1780263770440491073 https://twitter.com/ayzwah/status/1780263770440491073 Gen

More videos: Hanging multiple shirts in a row: https://twitter.com/ayzwah/status/1780263770440491073 https://twitter.com/ayzwah/status/1780263770440491073 Generalizing to unseen sweater: https://twitter.com/ayzwah/status/1780263771858194809 https://twitter.com/ayzwah/status/1780263771858194809 Struggling to unfold a shirt: https://twitter.com/DannyDriess/status/1780270239185588732 https://twitter.com/DannyDriess/status/1780270239185588732 Assembling gears: https://twitter.com/ayzwah/status/1780263775213629497 https://twitter.com/ayzwah/status/1780263775213629497

godelski 3y ago
I wish more of these were shown at 1x (last one is). Sure, a bit slower, but if you watch the OP link at 1/2 speed it is still impressive. Here's a shoe tying at 1x: https://twitter.com/ayzwah/status/1780263776694182311 https://twitter.com/ayzwah/status/1780263776694182311 Interesting here how it ties the knot. The first knot is already in place and they just do the bow. I don't think how most people would tie their shoes would work well for a robot (bunny around tree method[0]) but I actually tie my shoes like this[1]. This is the same way the robot ties. So I gotta ask, was this a purely learned policy or was this taught or pushed in that direction[^]? I suspect the latter. [0] https://www.youtube.com/watch?v=YwqQvKtmefE https://www.youtube.com/watch?v=YwqQvKtmefE [1] https://www.youtube.com/watch?v=XPIgR89jv3Q https://www.youtube.com/watch?v=XPIgR89jv3Q [^] by pushed in that way, would include watching that video or any other videos like it.
- GuB-42 3y ago
  Here is the source for that knot (as well as many others) https://www.fieggen.com/shoelace/ianknot.htm https://www.fieggen.com/shoelace/ianknot.htm
- polygamous_bat 3y ago
  This is an imitation learning policy running on the robot, which means that the robot was taught by hours and hours of humans controlling it to do the task. The robot didn’t learn to do anything from scratch.
  - godelski 3y ago
    Thanks for confirming. I suspected but hard to tell who is saying they do things from scratch (through sims) or imitation.
fragmede 3y ago
UBTECH and Baidu out of China demoed a clothes folding robot demo two weeks ago (early April 2024), and the video is claimed to be 1x/realtime. https://youtu.be/8MRDF2pkIRs https://youtu.be/8MRDF2pkIRs
- modeless 3y ago
  Cool video. I would say that the clothes folding task in the UBTECH video is much, much easier than the ones in Google's videos. In fact, it could potentially be performed by simple replay of scripted motions with no sensing at all (with low reliability). I have some things I always look for when I'm watching robot demo videos: 1. Are there cuts in the video? If so, the robot may not be able to perform the entire task by itself without help. UBTECH's video has a couple of cuts. Google's videos have none. 2. Is the video sped up? If so, the robot may be very slow. UBTECH's video is 1x which is good, but you can see that the robot does move somewhat slowly and does not switch fluidly between actions. Google posted both 1x and 2x-20x videos so that you can easily see both real time speed and long duration reliability. In the 1x videos Google's robot is also somewhat slow, however it seems to switch more fluidly between actions than UBTECH. 3. Is the initial state at the start of the video realistic? If not, the robot may not be able to start the task without help. UBTECH's video starts with a carefully folded and perfectly flat shirt already in the hands of the robot. Google's videos start with shirts relatively messily placed on tables and somewhat crumpled. 4. Is the task repeated? If not, the robot may be very unreliable at finishing the task. Google's videos show a lot of repetition without cuts. UBTECH's video shows only one instance (with cuts). You could still produce this video even if the UBTECH robot fails 90% of the time. 5. Is there variation in the repeated tasks? If not, the robot may fail if there is any variation. Google shows different colors and initial states of shirts, and also a much larger sweater. That said, almost all the shirts are small polo shirts and the robot would certainly not generalize to anyone's real closet of varied clothes. 6. Does the robot react to mistakes or other unexpected events? If not, it may be mostly playback of pre-recorded motions with little or no sensing influencing the robot's behavior. UBTECH's video shows alleged sensing, but doesn't show any unexpected things or mistakes. Google's videos show the robot recovering from mistakes.
  - 3y ago
hammock 3y ago
At the end of the OP video, why does the right arm grab the hanger then hand it to the left arm? Seems like a highly unnecessary and inefficient step
- pulvinar 3y ago
  There will always need to be a "wasted" step. The hanger comes off the rack with the hook away from the robot and has to go back on that way. Since there are two shoulders of the hanger to put into the shirt, they have to hand the hanger to from one to the other, and that would leave the hook facing the last robot if they don't do this extra step.
  - hammock 3y ago
    Couldn't the left arm just take the hanger off the rod?
    - pulvinar 3y ago
      I don't see that it helps, since the task is symmetrical. Whichever arm takes it off has to give it to the other arm at some point so both sides of the hanger are used, and it needs to be given back either before or after doing the hanging so the hanger's hook is back on the outside (facing away from the arm holding it), before replacing it on the rack. Would be interesting to know if the AI figured this out (the hard way, I'm sure).
      - hammock 3y ago
        Not sure we are talking about the same thing. I'm specifically talking about 1:07-1:10. What is the reason why the left arm could not take the hanger off directly, without involvement of the right arm?
        pulvinar 3y ago
        It could, and then the hanger would be facing the other direction, and everything would simply be mirrored left-right. The last arm to hold the hanger still has to replace it on the rack, which means it needs the hanger's hook facing away from it, which only happens when there have been an even number of exchanges between arms. Thus the need for that extra exchange. That extra exchange could have been done after putting the shirt on the hanger, but that would have been more risk for it falling off again.

godelski 3y ago

I wish more of these were shown at 1x (last one is). Sure, a bit slower, but if you watch the OP link at 1/2 speed it is still impressive. Here's a shoe tying at 1x: https://twitter.com/ayzwah/status/1780263776694182311 https://twitter.com/ayzwah/status/1780263776694182311 Interesting here how it ties the knot. The first knot is already in place and they just do the bow. I don't think how most people would tie their shoes would work well for a robot (bunny around tree method[0]) but I actually tie my shoes like this[1]. This is the same way the robot ties. So I gotta ask, was this a purely learned policy or was this taught or pushed in that direction[^]? I suspect the latter. [0] https://www.youtube.com/watch?v=YwqQvKtmefE https://www.youtube.com/watch?v=YwqQvKtmefE [1] https://www.youtube.com/watch?v=XPIgR89jv3Q https://www.youtube.com/watch?v=XPIgR89jv3Q [^] by pushed in that way, would include watching that video or any other videos like it.

GuB-42 3y ago
Here is the source for that knot (as well as many others) https://www.fieggen.com/shoelace/ianknot.htm https://www.fieggen.com/shoelace/ianknot.htm
polygamous_bat 3y ago
This is an imitation learning policy running on the robot, which means that the robot was taught by hours and hours of humans controlling it to do the task. The robot didn’t learn to do anything from scratch.
- godelski 3y ago
  Thanks for confirming. I suspected but hard to tell who is saying they do things from scratch (through sims) or imitation.

fragmede 3y ago

UBTECH and Baidu out of China demoed a clothes folding robot demo two weeks ago (early April 2024), and the video is claimed to be 1x/realtime. https://youtu.be/8MRDF2pkIRs https://youtu.be/8MRDF2pkIRs

modeless 3y ago
Cool video. I would say that the clothes folding task in the UBTECH video is much, much easier than the ones in Google's videos. In fact, it could potentially be performed by simple replay of scripted motions with no sensing at all (with low reliability). I have some things I always look for when I'm watching robot demo videos: 1. Are there cuts in the video? If so, the robot may not be able to perform the entire task by itself without help. UBTECH's video has a couple of cuts. Google's videos have none. 2. Is the video sped up? If so, the robot may be very slow. UBTECH's video is 1x which is good, but you can see that the robot does move somewhat slowly and does not switch fluidly between actions. Google posted both 1x and 2x-20x videos so that you can easily see both real time speed and long duration reliability. In the 1x videos Google's robot is also somewhat slow, however it seems to switch more fluidly between actions than UBTECH. 3. Is the initial state at the start of the video realistic? If not, the robot may not be able to start the task without help. UBTECH's video starts with a carefully folded and perfectly flat shirt already in the hands of the robot. Google's videos start with shirts relatively messily placed on tables and somewhat crumpled. 4. Is the task repeated? If not, the robot may be very unreliable at finishing the task. Google's videos show a lot of repetition without cuts. UBTECH's video shows only one instance (with cuts). You could still produce this video even if the UBTECH robot fails 90% of the time. 5. Is there variation in the repeated tasks? If not, the robot may fail if there is any variation. Google shows different colors and initial states of shirts, and also a much larger sweater. That said, almost all the shirts are small polo shirts and the robot would certainly not generalize to anyone's real closet of varied clothes. 6. Does the robot react to mistakes or other unexpected events? If not, it may be mostly playback of pre-recorded motions with little or no sensing influencing the robot's behavior. UBTECH's video shows alleged sensing, but doesn't show any unexpected things or mistakes. Google's videos show the robot recovering from mistakes.
- 3y ago

hammock 3y ago

At the end of the OP video, why does the right arm grab the hanger then hand it to the left arm? Seems like a highly unnecessary and inefficient step

pulvinar 3y ago
There will always need to be a "wasted" step. The hanger comes off the rack with the hook away from the robot and has to go back on that way. Since there are two shoulders of the hanger to put into the shirt, they have to hand the hanger to from one to the other, and that would leave the hook facing the last robot if they don't do this extra step.
- hammock 3y ago
  Couldn't the left arm just take the hanger off the rod?
  - pulvinar 3y ago
    I don't see that it helps, since the task is symmetrical. Whichever arm takes it off has to give it to the other arm at some point so both sides of the hanger are used, and it needs to be given back either before or after doing the hanging so the hanger's hook is back on the outside (facing away from the arm holding it), before replacing it on the rack. Would be interesting to know if the AI figured this out (the hard way, I'm sure).
    - hammock 3y ago
      Not sure we are talking about the same thing. I'm specifically talking about 1:07-1:10. What is the reason why the left arm could not take the hanger off directly, without involvement of the right arm?
      - pulvinar 3y ago
        It could, and then the hanger would be facing the other direction, and everything would simply be mirrored left-right. The last arm to hold the hanger still has to replace it on the rack, which means it needs the hanger's hook facing away from it, which only happens when there have been an even number of exchanges between arms. Thus the need for that extra exchange. That extra exchange could have been done after putting the shirt on the hanger, but that would have been more risk for it falling off again.