8 ms·
Google DeepMind's Aloha Unleashed is pushing the boundaries of robot dexterity
- RobotToaster 3y agoI assume, being google, none of this is going to be open source?
- krasin 3y agoIt's already open-source; most of it anyway: 1. https://github.com/tonyzhaozh/aloha https://github.com/tonyzhaozh/aloha 2. https://aloha-2.github.io/ https://aloha-2.github.io/ 3. https://github.com/tonyzhaozh/aloha/tree/main/aloha2 https://github.com/tonyzhaozh/aloha/tree/main/aloha2
- RobotToaster 3y agoThanks, that's helpful[0], I must have missed that. [0]Unlike the people who downvoted me for asking a question.
- we_love_idf 3y agoGoogle's days are numbered. OpenAI showed that AI is about delivering AGI, not playing some board games and doing PR stunts. Unfortunately Google hasn't learned its lessons. It's still doing PR stunts and people are falling for it.
- modeless 3y agoMore videos: Hanging multiple shirts in a row: https://twitter.com/ayzwah/status/1780263770440491073 https://twitter.com/ayzwah/status/1780263770440491073 Generalizing to unseen sweater: https://twitter.com/ayzwah/status/1780263771858194809 https://twitter.com/ayzwah/status/1780263771858194809 Struggling to unfold a shirt: https://twitter.com/DannyDriess/status/1780270239185588732 https://twitter.com/DannyDriess/status/1780270239185588732 Assembling gears: https://twitter.com/ayzwah/status/1780263775213629497 https://twitter.com/ayzwah/status/1780263775213629497
- godelski 3y agoI wish more of these were shown at 1x (last one is). Sure, a bit slower, but if you watch the OP link at 1/2 speed it is still impressive. Here's a shoe tying at 1x: https://twitter.com/ayzwah/status/1780263776694182311 https://twitter.com/ayzwah/status/1780263776694182311 Interesting here how it ties the knot. The first knot is already in place and they just do the bow. I don't think how most people would tie their shoes would work well for a robot (bunny around tree method[0]) but I actually tie my shoes like this[1]. This is the same way the robot ties. So I gotta ask, was this a purely learned policy or was this taught or pushed in that direction[^]? I suspect the latter. [0] https://www.youtube.com/watch?v=YwqQvKtmefE https://www.youtube.com/watch?v=YwqQvKtmefE [1] https://www.youtube.com/watch?v=XPIgR89jv3Q https://www.youtube.com/watch?v=XPIgR89jv3Q [^] by pushed in that way, would include watching that video or any other videos like it.
- GuB-42 3y agoHere is the source for that knot (as well as many others) https://www.fieggen.com/shoelace/ianknot.htm https://www.fieggen.com/shoelace/ianknot.htm
- polygamous_bat 3y agoThis is an imitation learning policy running on the robot, which means that the robot was taught by hours and hours of humans controlling it to do the task. The robot didn’t learn to do anything from scratch.
- godelski 3y agoThanks for confirming. I suspected but hard to tell who is saying they do things from scratch (through sims) or imitation.
- fragmede 3y agoUBTECH and Baidu out of China demoed a clothes folding robot demo two weeks ago (early April 2024), and the video is claimed to be 1x/realtime. https://youtu.be/8MRDF2pkIRs https://youtu.be/8MRDF2pkIRs
- riidom 3y agoReminds me about "Foxes in Love" somehow.
- occasionallybit 3y ago[flagged]
- throwup238 3y agoFinally, a robot that can tie my shoes for me!
- linsomniac 3y agoIn the last year I've started using that knot that those robots used, the "Ian Knot", to tie my shoes, and I'm loving it. https://www.fieggen.com/shoelace/ianknot.htm https://www.fieggen.com/shoelace/ianknot.htm
- mjamesaustin 3y agoYeah these robots tie a better shoelace than most humans!
- p1mrx 3y agoDo most humans leave their laces dragging on the ground? Though to be fair, those laces are really long. The robot needs to unlace the shoes, cut some length from the middle, tie a double fisherman's knot, and relace them.
- ozten 3y agoThe speed that these arms/hands move at is incredible compared to 4 months ago.
- chabons 3y agoThe videos are all shown at 2x speed, but your point stands, this is still pretty quick.
- lyapunova 3y agoSorry, but this is a lot of marketing for the same thing over and over again. I'm not against Aloha as an _affordable_ platform, but skimping on hardware is kind of a bug not a feature. Moreover it's not even _lowcost_, its BoM is still like 20k and collecting all the data is labor intensive and not cheap. And if we're focusing on the idea, it has existed since the 1950s and they were doing it relatively well then: https://www.youtube.com/watch?v=LcIKaKsf4cM https://www.youtube.com/watch?v=LcIKaKsf4cM
- modeless 3y agoThese videos are all autonomous. They didn't have that in the 1950s.
- lyapunova 3y agoI can appreciate that, but also they are recording and replaying motor signals from specific teleoperation demonstrations. Something that _was_ possible in the 1950s. You might say that it is challenging to replay demonstrations well on lower-quality hardware. And so there is academic value in trying to make it work on worse hardware, but it would not be my goto solution for real industry problems. E.g. this is not a route I would fund for a startup, for example.
- modeless 3y agoThey do not replay recorded motor signals. They use recorded motor signals only to train neural policies, which then run autonomously on the robot and can generalize to new instances of a task (such as the above video generalizing to an adult size sweater when it was only ever trained on child size polo shirts). Obviously some amount of generalization is required to fold a shirt, as no two shirts will ever be in precisely the same configuration after being dropped on a table by a human. Playback of recorded motor signals could never solve this task.
- adolph 3y ago> recorded motor signals only to train neural policies Is interesting that they are using "Leader Arms" [0] to encode tasks instead of motion capture. Is it just a matter of reduced complexity to get off the ground? I suppose the task of mapping human arm motion to what a robot can do is tough. 0. https://www.trossenrobotics.com/widowx-aloha-set https://www.trossenrobotics.com/widowx-aloha-set
- taylorfinley 3y agoI wonder if this unfortunate naming choice will cause a stir similar to: https://kawaiola.news/cover/aloha-not-for-sale-cultural-in-appropriation/ https://kawaiola.news/cover/aloha-not-for-sale-cultural-in-a...
- math_dandy 3y agoHopefully DeepMind will think twice before sending cease-and-desist orders to any Hawaiian AI robotics businesses with aloha in the name!
- taylorfinley 3y agoI should definitely hope so! Though I think the name would cause a stir in local circles even without any legal actions. Tech companies in general are deeply unpopular here (see: Larry Ellison, Mark Zuckerburg, and Marc Benioff buying up big chunks of land, AirBnB and digital nomads driving up rental prices so high such that more native Hawaiians now live on the mainland than in Hawai`i, and perceived lack of cultural respect from projects like the Thirty Meter Telescope leading to major protests). The other thing is that words have a lot of power in the cultural frame, even just the concept of aloha being something that could be "unleashed" is likely to offend. All to say nothing off the palpable fear people have here of robots taking hospitality industry jobs like housekeeping (which are unionized in many hotels out here, and are actually one of the few low-barrier-to-entry jobs out here that can support a reasonable quality of life) I'm sure I'll get a ton of downvotes for bringing up cultural sensitivity and pointing out these concerns -- I don't mean to imply they're all 100% rational nor that no one should say "aloha" unless they're Hawaiian, but if anyone at DeepMind had a Hawaiian cultural frame I think they likely would have flagged these concerns and recommended a different name.
- 1024core 3y ago> Tech companies in general are deeply unpopular here Which is such a shame, as Univ of Hawaii was one of the pioneers of the Internet: https://en.wikipedia.org/wiki/ALOHAnet https://en.wikipedia.org/wiki/ALOHAnet
- im3w1l 3y agoTo me the most impressive thing is the arms servicing each other. When they can self replicate it could potentially have big consequences. I have a dream that we put self-replicating robots on Mars and let them build a mostly by-robots for-robots civilization that can potentially export stuff to earth, do various science projects and build spacecraft.
- btbuildem 3y agoI guess this is obvious in retrospect.. but having two arms vs one greatly expands the range of possible tasks.
- baron816 3y agoSkin is also incredible when you think about it. Each square cm is able to sense temperature, texture, whether it’s wet, sticky, etc. And it’s self healing. It’s hard to imagine robots getting very far without artificial skin.
- yakz 3y agoSkin can’t sense “wet” can it? I thought it was mostly just temperature, but also in combination with a few other properties you perceive it as moisture but it can be easily fooled because there’s not a direct sense for it.
- CooCooCaCha 3y agoI think you answered your own question.
- reaperman 3y agoYour point is entirely valid despite my critiques. Technically, skin can't sense whether something is wet, and isn't particularly great at sensing temperature. Skin senses pressure and heat flow (derived via sensing temperature change of the flesh itself, rather than the temperature of the object it is touching), and perhaps can sense shear (there is a unique sensation when skin is stretched/pulled apart), as well as the weight of an object (if it is absorbent and more wet than damp). This distinction about what skin can directly sense manifests itself to deceive the human brain about wetness and temperature, specifically. Wetness is a perception derived from feeling higher-than-expected heat loss and unusual pressure/sheer, and even through the sound made when squeezing an absorbent material or the sensation of water pooling around the finger (broadening the area of heat loss) when you squeeze into the material. Damp laundry at room temperature is perceived as obviously wet because it feels colder than it should if it were dry, but when we're pulling laundry out of a dryer we often can't tell if it's dry vs. still a bit damp -- the higher temperature of the object removes the sensation of heat flowing away from our fingers, so there's nothing our fingers can sense to tell us the clothes aren't dry until the clothes finally cool down to room temperature. Our skin also doesn't sense the temperature of an object well if that object has a particularly high or low heat transfer coefficient of conduction. I recently bought a 6-pack of beer cans which have a moderately thick plastic vinyl label shrunk around the can. When I reach in my fridge, I can't convince myself to perceive it as chilled no matter how hard I try. Even though the vinyl is the same temperature as everything else in the fridge, it doesn't pull heat out of my finger tissue, so my brain cannot perceive that it isn't "room temperature". Conversely, picking up a normal metal can of beer that is just barely below room temperature, my brain perceives it to be much colder than it actually is because the metal draws heat away from my fingers so quickly compared to other objects. If wood is cooled 5 degrees below room temperature, it doesn't feel cold, but a can of beer certainly does! It is absolutely incredible that our skin can sense things to such a high resolution that it seems like we have a lot more abilities than we actually have. It is also amazing how our brain integrates this into a rich perception. But there actually aren't many physical properties actually being measured, and this distinction matters sometimes for edge cases, some of which are quite common.
- m3kw9 3y agoDon’t look impressive because this is what you see a lot in factories anyways, maybe a little better then Sota
- danpalmer 3y agoThe difference with factories is that every movement is programmed by someone in quite intricate detail. Factory robots aren't "smart" in any sense.
- YeGoblynQueenne 3y agoSo is this, except the movements are "programmed" by demonstration and with a very small tolerance for variation in the initial placement of objects. It's not that different from factory robots, and it won't be folding your laundry any time soon.
- adrr 3y agoFactories/Distribution Centers are doing hard goods not soft goods.
- dghughes 3y agoIf I ever move and end up living in a giant concrete warehouse devoid of furniture I'll keep this robot in mind.
- deleted 3y ago[deleted]
- julienreszka 3y agoI am skeptical. Lots of fake demos out there. Can we rule out that it turns out it’s actually remotely controlled by some dude in india, just like the amazon’s « Just walk out »?
- InPanthera 3y agoAmazons was remote controlled?
- sp332 3y agohttps://gizmodo.com/amazon-reportedly-ditches-just-walk-out-grocery-stores-1851381116 https://gizmodo.com/amazon-reportedly-ditches-just-walk-out-... "Just Walk Out relied on more than 1,000 people in India watching and labeling videos to ensure accurate checkouts."
- jjjjjjjkjjjjjj 3y agoI've worked in humanoid robots and manipulation for the past decade and this is mind blowing. For robots. Still pathetic compared to any human, but mind blowing for robots. I remember when we were hoping one humanoid would someday be able to replace a broken limb on another humanoid and we were designing super easy quick disconnects to make that possible. This is already way beyond that. Very impressive.
- fragmede 3y agohave you seen https://youtu.be/8MRDF2pkIRs https://youtu.be/8MRDF2pkIRs ?
- adolph 3y agoIt isn't clear how "Aloha Unleashed" is different from "Mobile ALOHA" Paper: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation: https://arxiv.org/abs/2401.02117 https://arxiv.org/abs/2401.02117 Video set: https://mobile-aloha.github.io/ https://mobile-aloha.github.io/ Tutorial: https://docs.google.com/document/d/1_3yhWjodSNNYlpxkRCPIlvIAaQ76Nqk2wsqhnEVM6Dc/edit https://docs.google.com/document/d/1_3yhWjodSNNYlpxkRCPIlvIA... Kits for sale: https://www.trossenrobotics.com/aloha-kits https://www.trossenrobotics.com/aloha-kits
- ingend88 3y agoIs there ready made low-cost arm that is available ?
- adolph 3y agoYou have to think of this as an entire system. The arm is necessary but not sufficient. An "arm" could be as simple as small servos and popsicle sticks [0]. In the case of ALOHA, below is an outline of the basic components. * arms (aka follower arms) - effector (i.e. gripper) - sensors (i.e. cameras, depth sensors, specced Intel RealSense D405) - gravity compensation (so the relatively delicate servos aren't overloaded) * controller - runs Robot Operating System (ROS [1]) plus other software (i.e. arm, gripper interfaces [2]) - runs ALOHA model in inference to tell ROS what to do based on task and sensor input - trains ALOHA models using arm motion encoder and ACT: Action Chunking with Transformers [4] * leader arms - motion encoders (essentialy an arm in reverse that can be used by a human to telecontrol the arm to encode motions into model training) The system at this point is "research grade" which is at once expensive due to custom/nice materials/units and not super user friendly--you must know a lot. See the build instructions [5]. 0. https://github.com/evildmp/BrachioGraph https://github.com/evildmp/BrachioGraph 1. https://www.ros.org/ https://www.ros.org/ 2. https://github.com/interbotix https://github.com/interbotix 3. https://www.trossenrobotics.com/aloha-kits https://www.trossenrobotics.com/aloha-kits 4. https://github.com/tonyzhaozh/act https://github.com/tonyzhaozh/act 5. https://docs.google.com/document/d/1sgRZmpS7HMcZTPfGy3kAxDrqFMtNNzmK-yVtX5cKYME/edit https://docs.google.com/document/d/1sgRZmpS7HMcZTPfGy3kAxDrq...
- InPanthera 3y agoWas expecting more from google than a robot that can tie shoe laces, wats the use case for this? Toys for the 1%?
- mikepurvis 3y agoI don't think the point is lace-tying; it's to demonstrate what is possible in terms of analogous tasks requiring a similar level of dexterity and environmental adaptability. In any case, the real start of this show is clearly the shirt hanging.
- yosito 3y agoThere's something odd about the way the arms move, like they are two distinct entities cooperating rather than being part of one coordinated mind. Maybe this is an example of the uncanny valley, or maybe it's because they are two physically separate arms, but it seems to me like one arm moves while the other waits for its turn. It's as if engineers programmed them to work sequentially. I wonder if it might be beneficial for engineers to study videos of humans doing these tasks and try to mimic those movements rather than trying to program a sequential procedure.
- patcon 3y agoNow I'm trying to imagine how our limb movements might be perceived by a creature that natively evolved the style of coordination in the video :) it would be "weird" but how might they describe that weirdness and what might underlie it in us..?
- dylan604 3y agoLook, it moves its mouth while it reads. Like it can't do one thing without doing the other thing moving at the same time
- lachlan_gray 3y agoSometimes I feel this about myself... I don't think much to walk or do something with both hands, they work stuff out on their own. How much do my legs or hands understand about each other?
- smusamashah 3y agoThis is very exciting but because its from Google this tech won't get out of their quarters.
- macromaniac 3y agoIt's impressive that transformers, diffusion, and human generated data can go so far in robotics. I would have expected simulation would be needed to achieve such results. My fear is that we see a similar problem with other generative AI in that it gets stuck in loops on complex problems and is unable to correct itself because the training data covers the problem but not the failure modes.
- 3327 3y ago[dead]
- visarga 3y agoThat's because most models have been trained on data created by humans for humans, it needs data created by AI for itself. Better learn from your mistakes than from the mistakes of others, they are more efficient and informative. When an AI is set up to learn from its own mistakes it might turn out like AlphaZero, who rediscovered the strategy of Go from scratch. LLMs are often incapable of solving complex tasks, but they are greatly helped by evolutionary algorithms. If you combine LLMs with EA you get black box optimization and intuition. It's all based on learning from the environment, interactivity & play. LLMs can provide the mutation operation, or function as judge to select surviving agents, or act as the agents themselves.
- netcan 3y agoOoh! it can almost fold a shirt. Shade aside, robotics is so damned hard. The current under/over for godlike superintelligence before a robot that can make sandwiches and work the laundry machine... So unintuitive.
- nabla9 3y agoLook at the grapples it has to work with. If you would have to work with two chopsticks, or two spanners as hands, you would not do any better.
- tho23i43423434 3y agoActually, you typically don't use all the dexterity of a five-fingered hand for most tasks. Opposing-digits gets you 90% of the way there, which is why when you finally start using fingers independently (say while playing the violin), a large amount of time is spent getting around the learnt 'synergies'. Unclear why we actually evolved fingers in the first place (balance ?).
- nabla9 3y agoIt's not the number of fingers I'm talking about. Human two fingers are much better gapplers than what that machine has. Fingers can feel pressure, have multiple joints, have soft pads.
- moffkalast 3y agoThe neat part about working in robotics is that nobody can tell if you're a genius or a moron because neither of the two can get the damn thing working properly.
- n0us 3y agoCan't wait for someone to turn this into a product and make this available to the public!
- a_wild_dandan 3y agoThe Laundry Folding Helping Hands will sell so goddamn hard. When the tech gets there, I'll be first in line. I'll even buy the Vegetable Chopping DLC.
- quux 3y agoUnexpected Ian Knot https://www.fieggen.com/shoelace/ianknot.htm https://www.fieggen.com/shoelace/ianknot.htm
- bastawhiz 3y agoThis makes me wonder whether Google regrets spinning Boston Dynamics back off as its own entity.
- throwaway29303 3y agohttps://www.theverge.com/2017/6/8/15766434/alphabet-google-boston-dynamics-softbank-sale-acquisition-robotics https://www.theverge.com/2017/6/8/15766434/alphabet-google-b...
- modeless 3y agoThey should. If they don't yet, I bet they will in a few years. One of the most shortsighted decisions Google has ever made.
- margorczynski 3y agoQuestion is how well it handles unseen variations and environments? This is a well controlled lab setup, with probably cherry-picked videos showing positive results.
- sheepscreek 3y agoWhat makes this special to me is their claim of using cheap robots. So, how much would one of those robo arms cost? In other words, what would it set me back to recreate this?
- krasin 3y agoHere is the tutorial how to recreate an ALOHA cell: https://docs.google.com/document/d/1sgRZmpS7HMcZTPfGy3kAxDrqFMtNNzmK-yVtX5cKYME/edit https://docs.google.com/document/d/1sgRZmpS7HMcZTPfGy3kAxDrq... The robot arm kits are available commercially: https://www.trossenrobotics.com/aloha-kits https://www.trossenrobotics.com/aloha-kits It's my belief that 10x cheaper arms with the same performance are possible, and the only reason they don't exist is because nobody needs them in sufficient quantities.
- YeGoblynQueenne 3y agoAs usual, people get over-excited about videos that seem to be showing robots doing something useful, when in truth they're doing something that shows how far away we are from robotic autonomy. It's like the dancing robot videos by Boston Dynamics all over again. The Aloha systems in this video are just as scripted as Boston Dynamics robots. The difference is in how the robots' behaviours are scripted: instead of hard-coding Finite State Controllers, the behaviours are programmed by demonstration, through teleoperation. The end result is the same: the final system can do one thing, and do it well enough to put in a video, but it can't deal with changes to the environment or to the initial setup of the task. For instance, in the shoe-lace tying part of the video, the shoe is already placed neatly in between the two arms and only at a small angle away from the vertical, the laces are pulled to the side and of course the knot is already half-tied. If you changed those initial parameters by a few units, placed the shoe further towards the top or bottom of the table, had both laces on one side, or turned the shoe in a right angle to the arms, the system would fail. Despite the "generalisation to sweater" video, there's very little flexiblity and very little generalisation, and that's still a system that can perform a handful of discrete tasks (swap gripper, tie shoelaces, hang shirt). That's not a system that can function autonomously in the real world. No robot maids or android valets coming in a shop near you any time soon, I'm afraid. In the foreseeable future the most successful autonomous systems will remain self-guided munition, which are made to be destroyed on impact and cause maximum damage.