4 ms·

Can someone with expertise explain why this is so difficult? As an outsider with no knowledge of robotics, I've always been surprised that manipulation tasks (

Can someone with expertise explain why this is so difficult? As an outsider with no knowledge of robotics, I've always been surprised that manipulation tasks (or just smooth robotic movement) are so challenging and seem to progress so slowly, especially when compared to IT.

talldrinkofwhat 3y ago
I'm about a decade out, so I'm sure the SOTA has moved the goal-posts, but the work I did in grad(ish) school revolved around making end-effectors (fancy word for robot hands) that could both grab a styrofoam cup, and grab a wet glass (cylindrical, no pint-cheaters). Plenty of cups were crushed. Plenty of glasses were dropped. Nothing that did both happened by the time I graduated. The amount of feedback built into your end-effectors (pedantic word for human hands) is insane. If you're not familiar, proprioception is a good google/wiki hole. Most of the signals that allow you to move your hands don't even hit the brain stem, let alone the boss upstairs. The challenge mostly lies in how we've instrumented these things. Precision requires low tolerances. Low tolerances + unexpected environment == you've just driven your robot through the countertop/pan/coworker or broken a very nicely geared servo.
- zith 3y ago
  Adding to that, even if we had perfect end-effectors with a good sense of touch, understanding the real world enough to manipulate it is hard. These days we have 3d cameras, but they still only see part of the objects we want to manipulate. The back side is hidden. So you need to either specify and model all objects to interact with, or have some word of a world model where we can predict what the full object, it's weight, center of gravity, surface texture, etc, is like. And before we even decide to manipulate it, we have to detect it, categorize it and segment it (where does the pan stop and the stove begin?). We have to plan out a manipulation task, including finding grasp points, finding movement patterns that do not interfere with the rest of the environment, etc. It's a whole bunch of separate problems that need solving all at once. There's motor control, building the right manipulators with the right sensors, bringing all the sensor data into something where we can make a single decision, understanding of the world and what happens during manipulation, and higher level planning.
  - imiric 3y ago
    I realize these are difficult problems, but couldn't we simulate how the human brain approaches these situations? That is, we don't model the entire 3D world in our head, but make decisions in real-time mostly by intuition and previous knowledge. We perceive depth of objects visually, and loosely map out their position and dimensions that way. We don't need to know the center of mass of every object, but have general intuition for where to grab it (if it has a handle, etc.). We have touch sensors to determine if something is hot or cold, and thus safe to handle, but a robot could have actual temperature sensors, making this easier. I'm far removed from this field, and speaking as a layperson, so pardon my ignorance.
    - MaxikCZ 3y ago
      The thing is that you take intuition for granted, but machine parts just have none. Programming intuition is exceedingly hard, but we are getting closer with neural networks. I'd say its easier to program machine calculating predicted centre of mass of an object than algorithmic sense of intuition outputting suitable spot to grab the item effectively.
      - imiric 3y ago
        I get that, but yeah, with ML it would be a matter of training it on raw data: objects, materials, physical properties and behaviors, etc. And then "intuition" would arise from this knowledge, and its own experience from reinforced learning. It's the same problem as implementing self-driving in vehicles, just applied to a different domain. I'm not downplaying the difficulty, of course, but pointing out that this type of automation wouldn't be feasible if we'd have to classically program every scenario the robot is likely to encounter.
        NalNezumi 3y ago
        I don't think you're downplaying the difficulty but just completely unaware of the depth of it. We don't even know if "intuition" would arise from the knowledge you claim, we don't know how that model would work, and even before that, collecting all the data (not to speak of availability of all the sensors) is a vastly more complex than even what ChatGPT or any LLM model data collection would ever be. >it's own experience from reinforcement learning This is a common mistake often heard from CS -> ML(RL) -> robotics transition folks. Reward function is given for free in RL, but in the real world, estimating the reward is a complex problem in its self. That's why RL on robotics have mostly seen success in quadrupedal locomotion; the reward function is simple (forward velocity, calculated from IMU), but how would you calculate a reward function in 30Hz+ for a simple task such as "chop onion and put it in the pan"? If you can construct the reward function for that task, most likely, you already have all the world-states available and might as well skip RL and do something else with that, such as Model-predictive control. As for intuition, see: https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox
Roark66 3y ago
Just some reasons as a robotics enthusiast: Hardware: - lack of off the shelf parts suitable for such projects. For example sensitive enough and compact at the same time pressure & force sensing. Also actuators, most that we have off the shelf is too slow, too bulky, too weak, or too strong/heavy to the point of being unsafe in direct contact with humans. Software: - one requires very low latency control loops and communications. If this is to go beyond local network use it will also have to include some software that will maintain the "feel" while connection gets dropped briefly, packets are delayed etc. - whenever force feedback is used one requires robust safety protocols. If the manipulator is strong enough to simulate a force of lifting a heavy object it might be capable of breaking your arm or a finger. Software has to prevent this even when it "glitches" or if there is hardware failure. No doubt there is lots of much more advanced reasons. These are just from the top of my head.
- regularfry 3y ago
  Also, the hardware is too damn expensive. They will have spent thousands of dollars just on actuators for those rigs. It makes it very hard for anyone outside a research or industrial setting to sketch out ideas.
  - imtringued 3y ago
    For reference. A hobbyist building a quadruped with dynamixels, the most accessible robot servos, is going to spend 12x$60 = $720 just on motors alone for a measily torque of 1.5Nm. Basically, such a motor can lift two chocolate bars (2x100g) at arms length.
flor1s 3y ago
I think these days we can quite reliably train robots to do single tasks from a few demonstrations. The problem is that "quite reliably" simply is not good enough for the real world. Also a "low-cost" system as presented in this work is still 32k (ignoring hardware and software integration costs).
iandanforth 3y ago
Muscles are pretty amazing. They have a higher strength to weight ratio than pretty much any small actuators we have. That strength is essential for smooth dynamic movement (the forces you encounter trying to pick up a gallon of milk, open a jar, get yourself out of bed are surprisingly large). In addition we don't just have muscles that go forward and back, or up and down, like many actuators. We have dozens of muscles engaged during pretty much any task that allow for flexible 3D force application. This is then coupled with reflexes and the brain's ability to accurately predict the body's motion, and that of things you interact with. Robot actuators are almost all reactive making their processing speed a limitation for control.* Humans all use long range predictive control to apply forces preemptively to smooth out motions. Finally we're highly optimized for an effort minimization, we don't just choose motions that work, we choose motions that are efficient. That objective and ability goes all the way back to evolutionary influences. So yeah, smooth motion feels easy, but is a gd miracle of biology :) * Robots can make up for a lack of prediction through really really fast control. This is how Boston Dynamics robots operate at a basic level.
- JoeAltmaier 3y ago
  Here's one! https://interestingengineering.com/science/a-new-artificial-muscle-is-stronger-more-flexible-than-natural-ones https://interestingengineering.com/science/a-new-artificial-...
- bemmu 3y ago
  It's also amazing how silent they are vs. any actuators we have.
- yowlingcat 3y ago
  Agree with all of that, and will add one more -- the power to weight ratio of muscles is truly remarkable. Really comes into focus when you compare to even current SOTA actuators.
  - panabee 3y ago
    interesting. could you elaborate on the mechanical/physical limitations that cause SOTA actuators to lag behind muscles, and if there's an equivalent "moore's law" that might predict when this gap closes appreciably, if ever?

talldrinkofwhat 3y ago

I'm about a decade out, so I'm sure the SOTA has moved the goal-posts, but the work I did in grad(ish) school revolved around making end-effectors (fancy word for robot hands) that could both grab a styrofoam cup, and grab a wet glass (cylindrical, no pint-cheaters). Plenty of cups were crushed. Plenty of glasses were dropped. Nothing that did both happened by the time I graduated. The amount of feedback built into your end-effectors (pedantic word for human hands) is insane. If you're not familiar, proprioception is a good google/wiki hole. Most of the signals that allow you to move your hands don't even hit the brain stem, let alone the boss upstairs. The challenge mostly lies in how we've instrumented these things. Precision requires low tolerances. Low tolerances + unexpected environment == you've just driven your robot through the countertop/pan/coworker or broken a very nicely geared servo.

zith 3y ago
Adding to that, even if we had perfect end-effectors with a good sense of touch, understanding the real world enough to manipulate it is hard. These days we have 3d cameras, but they still only see part of the objects we want to manipulate. The back side is hidden. So you need to either specify and model all objects to interact with, or have some word of a world model where we can predict what the full object, it's weight, center of gravity, surface texture, etc, is like. And before we even decide to manipulate it, we have to detect it, categorize it and segment it (where does the pan stop and the stove begin?). We have to plan out a manipulation task, including finding grasp points, finding movement patterns that do not interfere with the rest of the environment, etc. It's a whole bunch of separate problems that need solving all at once. There's motor control, building the right manipulators with the right sensors, bringing all the sensor data into something where we can make a single decision, understanding of the world and what happens during manipulation, and higher level planning.
- imiric 3y ago
  I realize these are difficult problems, but couldn't we simulate how the human brain approaches these situations? That is, we don't model the entire 3D world in our head, but make decisions in real-time mostly by intuition and previous knowledge. We perceive depth of objects visually, and loosely map out their position and dimensions that way. We don't need to know the center of mass of every object, but have general intuition for where to grab it (if it has a handle, etc.). We have touch sensors to determine if something is hot or cold, and thus safe to handle, but a robot could have actual temperature sensors, making this easier. I'm far removed from this field, and speaking as a layperson, so pardon my ignorance.
  - MaxikCZ 3y ago
    The thing is that you take intuition for granted, but machine parts just have none. Programming intuition is exceedingly hard, but we are getting closer with neural networks. I'd say its easier to program machine calculating predicted centre of mass of an object than algorithmic sense of intuition outputting suitable spot to grab the item effectively.
    - imiric 3y ago
      I get that, but yeah, with ML it would be a matter of training it on raw data: objects, materials, physical properties and behaviors, etc. And then "intuition" would arise from this knowledge, and its own experience from reinforced learning. It's the same problem as implementing self-driving in vehicles, just applied to a different domain. I'm not downplaying the difficulty, of course, but pointing out that this type of automation wouldn't be feasible if we'd have to classically program every scenario the robot is likely to encounter.
      - NalNezumi 3y ago
        I don't think you're downplaying the difficulty but just completely unaware of the depth of it. We don't even know if "intuition" would arise from the knowledge you claim, we don't know how that model would work, and even before that, collecting all the data (not to speak of availability of all the sensors) is a vastly more complex than even what ChatGPT or any LLM model data collection would ever be. >it's own experience from reinforcement learning This is a common mistake often heard from CS -> ML(RL) -> robotics transition folks. Reward function is given for free in RL, but in the real world, estimating the reward is a complex problem in its self. That's why RL on robotics have mostly seen success in quadrupedal locomotion; the reward function is simple (forward velocity, calculated from IMU), but how would you calculate a reward function in 30Hz+ for a simple task such as "chop onion and put it in the pan"? If you can construct the reward function for that task, most likely, you already have all the world-states available and might as well skip RL and do something else with that, such as Model-predictive control. As for intuition, see: https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox

Roark66 3y ago

Just some reasons as a robotics enthusiast: Hardware: - lack of off the shelf parts suitable for such projects. For example sensitive enough and compact at the same time pressure & force sensing. Also actuators, most that we have off the shelf is too slow, too bulky, too weak, or too strong/heavy to the point of being unsafe in direct contact with humans. Software: - one requires very low latency control loops and communications. If this is to go beyond local network use it will also have to include some software that will maintain the "feel" while connection gets dropped briefly, packets are delayed etc. - whenever force feedback is used one requires robust safety protocols. If the manipulator is strong enough to simulate a force of lifting a heavy object it might be capable of breaking your arm or a finger. Software has to prevent this even when it "glitches" or if there is hardware failure. No doubt there is lots of much more advanced reasons. These are just from the top of my head.

regularfry 3y ago
Also, the hardware is too damn expensive. They will have spent thousands of dollars just on actuators for those rigs. It makes it very hard for anyone outside a research or industrial setting to sketch out ideas.
- imtringued 3y ago
  For reference. A hobbyist building a quadruped with dynamixels, the most accessible robot servos, is going to spend 12x$60 = $720 just on motors alone for a measily torque of 1.5Nm. Basically, such a motor can lift two chocolate bars (2x100g) at arms length.

flor1s 3y ago

I think these days we can quite reliably train robots to do single tasks from a few demonstrations. The problem is that "quite reliably" simply is not good enough for the real world. Also a "low-cost" system as presented in this work is still 32k (ignoring hardware and software integration costs).

iandanforth 3y ago

Muscles are pretty amazing. They have a higher strength to weight ratio than pretty much any small actuators we have. That strength is essential for smooth dynamic movement (the forces you encounter trying to pick up a gallon of milk, open a jar, get yourself out of bed are surprisingly large). In addition we don't just have muscles that go forward and back, or up and down, like many actuators. We have dozens of muscles engaged during pretty much any task that allow for flexible 3D force application. This is then coupled with reflexes and the brain's ability to accurately predict the body's motion, and that of things you interact with. Robot actuators are almost all reactive making their processing speed a limitation for control.* Humans all use long range predictive control to apply forces preemptively to smooth out motions. Finally we're highly optimized for an effort minimization, we don't just choose motions that work, we choose motions that are efficient. That objective and ability goes all the way back to evolutionary influences. So yeah, smooth motion feels easy, but is a gd miracle of biology :) * Robots can make up for a lack of prediction through really really fast control. This is how Boston Dynamics robots operate at a basic level.

JoeAltmaier 3y ago
Here's one! https://interestingengineering.com/science/a-new-artificial-muscle-is-stronger-more-flexible-than-natural-ones https://interestingengineering.com/science/a-new-artificial-...
bemmu 3y ago
It's also amazing how silent they are vs. any actuators we have.
yowlingcat 3y ago
Agree with all of that, and will add one more -- the power to weight ratio of muscles is truly remarkable. Really comes into focus when you compare to even current SOTA actuators.
- panabee 3y ago
  interesting. could you elaborate on the mechanical/physical limitations that cause SOTA actuators to lag behind muscles, and if there's an equivalent "moore's law" that might predict when this gap closes appreciably, if ever?