4 ms·

Can anyone that works on this technology provide an honest assessment of where this technology actually stands? How much instrumentation is actually required, w

by aabhay 2mo ago

Can anyone that works on this technology provide an honest assessment of where this technology actually stands? How much instrumentation is actually required, what the interaction quality is, how much trouble do humanoids have with in the wild daily tasks like turning doorknobs, recovering from falls, avoiding knocking into things, etc.

utopiah 2mo ago
IMHO the real test is if any robotic startup currently selling (or planning to sell) robots as a service for homes not just use it but gets returning users from it. I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks. FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dexterity/ https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.
p1esk 2mo ago
It’s still at GPT-1 level, but GPT-2 moment feels imminent.
- tavavex 2mo ago
  What signs point to an inevitable GPT-2 moment in this field?
  - p1esk 2mo ago
    This blog post
adityashankar 2mo ago
I work in this field, and I wrote my bachelor's thesis here, not with humanoids but with VLAs (think chatgpt connected to a robot arm) It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here. Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous. There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife. Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine. Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)
- dyauspitr 2mo ago
  This is absolutely not true check out Sunday robotics and their robot that folds clothes. Sunday robotics says it does holding of clothes correctly 99% of the time that is real world usage today. They even have three hour video of the robot folding the clothes.
  - shawabawa3 2mo ago
    They didn't mention anything about folding clothes so I suspect this comment is just an ad
    - cyrux004 2mo ago
      they considered it solved https://www.sunday.ai/blog/act-2-preview https://www.sunday.ai/blog/act-2-preview
    - dyauspitr 2mo ago
      It’s a video on their YouTube channel
  - YeGoblynQueenne 2mo ago
    No, the above is absolutely exactly true. Robots folding clothes and tying shoelaces etc is nothing but a tech demo at this point. It's difficult to grok this because if you watch a human folding a t-shirt, you can reliably predict that the same human will fold a different t-shirt just as well, and in fact be perfectly capable of folding a wide variety of other clothes items as well. Not so for robots. With robots, what you see is precisely what you get. If you see a robot folding a t-shirt, all that means is that that particular robot can fold that particular t-shirt. The state of the art today is that the same robot cannot be expected to be able to fold a different t-shirt. For example, see this article about Mobile ALOHA at Google. There's a passage where the visiting, awe-struck, journalist asks whether the robot he's just seen tying up a pair of shoelaces can tie up his own shoe. “If I gave it my shoe,” I ventured, “would it just totally fail?” “We could try,” Tompson said. I removed my right sneaker, with apologies to anyone forced to handle it. Tompson gamely placed it on the table, while Driess reloaded the policy. “To set expectations,” Driess said, “this is a task that is thought of as being impossible.” Tompson eyed his new experimental subject with some trepidation. “Very short shoelaces,” he said. The policy booted up, and the claws set to work. This time, they poked at the shoelace without getting a grip. “Do you give consent for your shoe to be destroyed?” Driess joked, as the hands grabbed at the tongue. Tompson let them try for a few more seconds before hitting the Failure pedal. https://archive.ph/CiJJG#selection-1887.0-1915.291 https://archive.ph/CiJJG#selection-1887.0-1915.291 (Original: https://www.newyorker.com/magazine/2024/12/02/a-revolution-in-how-robots-learn https://www.newyorker.com/magazine/2024/12/02/a-revolution-i...) As to ACT-2 which basically uses the same techniques as ALOHA (imitation learning) far as I can tell, that's a commercial product and the information they give on their site is difficult to parse. E.g. they say they have 99.1% ±0.3 success rate, 778 successful folds and 9 garment types which is low enough to engender some trust they're not trying to inflate their numbers, but they don't say whether they trained on the garments used in evaluation or not. Chances are they did, because that's the current limit of the technology, i.e. if the garment being folded is unseen (as opposed to the environment, which they tout) then performance is basically random. So either ACT-2 have a major breakthrough that is a few leaps and bounds away from the current state of the art, or you've just watched a tech demo. The fact that they only advertise "9 garment types" though is a big hint: they have the same problem with generalisation as everybody else at this point in time.
tamimio 2mo ago
There is a LOT of monkey business when it comes to robotics industry, a lot of demos you see are well crafted for that specific demo, a lot of times there’s even actual fraud “like claiming it’s an autonomous but it was actually teleoperated during the demo”, a lot robots that make most hypes are practically useless (dog robots or qugv for example) no matter how they try to create a usecase, when it comes to practicality, most fail. Robotics as an industry is niche (not to be confused with cobotics) the only exception is drones which is something I said in a workshop back in 2020 that drones are the only potential and will dominate if we solved the endurance issue, which is also why they immediately got weaponized because humans are usually evil and will utilize useful stuff in bad things. Humanoids are far from being remotely useful in real life scenarios, yet. Fun fact: most (if not all) qugv can’t go reverse on a stairway.
probablyabot 2mo ago
Unfortunately I can't go into too much depth but I've seen many first hand evaluations of these robots- I'm a robot technician with first hand experience working on these robots. There is an incredible amount of effort and money going into training on many different tasks, including a few you mentioned and many more. They're getting pretty good, but don't expect to see this generation of bots doing their thing in public for a while, if ever. Generally they're pretty safe but they're a long way from safe enough to be sold to individuals in a household environment. I expect to see an incredible amount of progress in the next couple years.

by aabhay 2mo ago

utopiah 2mo ago

IMHO the real test is if any robotic startup currently selling (or planning to sell) robots as a service for homes not just use it but gets returning users from it. I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks. FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dexterity/ https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.

p1esk 2mo ago

It’s still at GPT-1 level, but GPT-2 moment feels imminent.

tavavex 2mo ago
What signs point to an inevitable GPT-2 moment in this field?
- p1esk 2mo ago
  This blog post

adityashankar 2mo ago

I work in this field, and I wrote my bachelor's thesis here, not with humanoids but with VLAs (think chatgpt connected to a robot arm) It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here. Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous. There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife. Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine. Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)

dyauspitr 2mo ago
This is absolutely not true check out Sunday robotics and their robot that folds clothes. Sunday robotics says it does holding of clothes correctly 99% of the time that is real world usage today. They even have three hour video of the robot folding the clothes.
- shawabawa3 2mo ago
  They didn't mention anything about folding clothes so I suspect this comment is just an ad
  - cyrux004 2mo ago
    they considered it solved https://www.sunday.ai/blog/act-2-preview https://www.sunday.ai/blog/act-2-preview
  - dyauspitr 2mo ago
    It’s a video on their YouTube channel
- YeGoblynQueenne 2mo ago
  No, the above is absolutely exactly true. Robots folding clothes and tying shoelaces etc is nothing but a tech demo at this point. It's difficult to grok this because if you watch a human folding a t-shirt, you can reliably predict that the same human will fold a different t-shirt just as well, and in fact be perfectly capable of folding a wide variety of other clothes items as well. Not so for robots. With robots, what you see is precisely what you get. If you see a robot folding a t-shirt, all that means is that that particular robot can fold that particular t-shirt. The state of the art today is that the same robot cannot be expected to be able to fold a different t-shirt. For example, see this article about Mobile ALOHA at Google. There's a passage where the visiting, awe-struck, journalist asks whether the robot he's just seen tying up a pair of shoelaces can tie up his own shoe. “If I gave it my shoe,” I ventured, “would it just totally fail?” “We could try,” Tompson said. I removed my right sneaker, with apologies to anyone forced to handle it. Tompson gamely placed it on the table, while Driess reloaded the policy. “To set expectations,” Driess said, “this is a task that is thought of as being impossible.” Tompson eyed his new experimental subject with some trepidation. “Very short shoelaces,” he said. The policy booted up, and the claws set to work. This time, they poked at the shoelace without getting a grip. “Do you give consent for your shoe to be destroyed?” Driess joked, as the hands grabbed at the tongue. Tompson let them try for a few more seconds before hitting the Failure pedal. https://archive.ph/CiJJG#selection-1887.0-1915.291 https://archive.ph/CiJJG#selection-1887.0-1915.291 (Original: https://www.newyorker.com/magazine/2024/12/02/a-revolution-in-how-robots-learn https://www.newyorker.com/magazine/2024/12/02/a-revolution-i...) As to ACT-2 which basically uses the same techniques as ALOHA (imitation learning) far as I can tell, that's a commercial product and the information they give on their site is difficult to parse. E.g. they say they have 99.1% ±0.3 success rate, 778 successful folds and 9 garment types which is low enough to engender some trust they're not trying to inflate their numbers, but they don't say whether they trained on the garments used in evaluation or not. Chances are they did, because that's the current limit of the technology, i.e. if the garment being folded is unseen (as opposed to the environment, which they tout) then performance is basically random. So either ACT-2 have a major breakthrough that is a few leaps and bounds away from the current state of the art, or you've just watched a tech demo. The fact that they only advertise "9 garment types" though is a big hint: they have the same problem with generalisation as everybody else at this point in time.

tamimio 2mo ago

There is a LOT of monkey business when it comes to robotics industry, a lot of demos you see are well crafted for that specific demo, a lot of times there’s even actual fraud “like claiming it’s an autonomous but it was actually teleoperated during the demo”, a lot robots that make most hypes are practically useless (dog robots or qugv for example) no matter how they try to create a usecase, when it comes to practicality, most fail. Robotics as an industry is niche (not to be confused with cobotics) the only exception is drones which is something I said in a workshop back in 2020 that drones are the only potential and will dominate if we solved the endurance issue, which is also why they immediately got weaponized because humans are usually evil and will utilize useful stuff in bad things. Humanoids are far from being remotely useful in real life scenarios, yet. Fun fact: most (if not all) qugv can’t go reverse on a stairway.

probablyabot 2mo ago

Unfortunately I can't go into too much depth but I've seen many first hand evaluations of these robots- I'm a robot technician with first hand experience working on these robots. There is an incredible amount of effort and money going into training on many different tasks, including a few you mentioned and many more. They're getting pretty good, but don't expect to see this generation of bots doing their thing in public for a while, if ever. Generally they're pretty safe but they're a long way from safe enough to be sold to individuals in a household environment. I expect to see an incredible amount of progress in the next couple years.