4 ms·

Ugh. This is making a hash of it. What is true: The understanding of laymen about what robots can do is very much out of touch with reality. Making real world

Ugh. This is making a hash of it. What is true: The understanding of laymen about what robots can do is very much out of touch with reality. Making real world robots is very hard. I think it's movies which made people think that any 6 year old can just plop together a C-3PO. What is the hash then? (I omit the points where I lack experience.) 1; Motion planning "Even things like a model of where the robot is, with respect to the surroundings" That's not motion planing. Motion planing starts from when you have a model of yourself and your surrounding. There are theoretical challenges (imagine a flytrap, or maze) but the real world challenges in my experience come from that under the inaccuracies and glitchyness of perception you are expected to make okay-ish decisions. 3; SLAM "there’s always going to be new obstacles (a pair of shoes, a book)" SLAM is about localisation. It gives you a 6dof pose from some fixed coordinate system. It does not deal with tripping hazards. That's obstacle detection. It does indeed deal with "mapping" but only in as much as to get you that pose estimate. "not turn-key to where there would be a SLAM module you can buy for your robot." Sure you can. The "Intel Realsense t265" is for example one such a module. The inside-out tracking of the Oculus Quest is an other (albeit one you will have hard time buying for your robot.) 6; Depth estimation. That one is odd. True, getting full fidelity depth maps out of monocular images is a research problem. But if in reality you want to estimate the distance to your beer bottle you will use stereo images, or a Kinect like depth sensor. It is obviously a hard engineering challenge to make it four 9 robust, but not an unheard of challenge. If his definition of an "open problem" is that there are people writing research papers about it then it will remain an open problem for a while. But if he just wants to depth estimate stuff, then there are already working good ways. Maybe cost prohibitive, maybe not robust enough for his liking. 9; Scene understanding Humanity made insane leaps and bounds on this one. Again there are engineering tradeoffs. How much accuracy you get for how much watts. What kind of training data you need, etc etc. Our systems are nowhere as good as a 5 year old human child, but I think we can handle that "beer bottle obscured by ketchup bottle" challenge if we try.

cpgxiii 6y ago
Depth estimation I wish it were as solved as vendors would like to believe it is. If you want a sensor for medium-range applications, say order 0.5m to 10m depth with better than 1cm accuracy, that works indoors and outdoors, with reflective and untextured objects, doesn't interfere with other sensors of its kind, it simply does not exist. Traditional active and passive stereo is OK provided you have texture, ToF so long as the surfaces aren't reflective and you don't have sunlight to worry about, structured light if you can control lighting and can otherwise control/avoid interference from other sensors. There's some promise in learned stereo matching, but collecting enough data and running fast enough inference are big challenges to practical use. Existing depth sensors for manipulation tasks, even indoors in reasonably controlled lighting, are still mostly insufficient given that the objects they struggle to see are often the objects you want to manipulate.
- fxtentacle 6y ago
  I strongly disagree! I have seen fantastic results with stereo cameras, colored lights, and self-calibration. For most use cases, it's no problem if your robot will stop for a few seconds, rotate the camera axis around a bit, and then continue. But that appears to be good enough to calibrate the features for tracking things like a reflective and transparent glass jar. As for the precision, I agree that 1cm at 10m distance doesn't work. But 1cm precision at 50cm distance is doable. And for a robot arm, you mainly need the precision when you're close to the object. And yes, I am talking about what you probably meant with learned stereo matching. I would call it close to solved because we can by now do unsupervised training and achieve usable results. While I had trouble reproducing this specific paper, the general idea is valid: https://github.com/google-research/google-research/tree/master/depth_from_video_in_the_wild https://github.com/google-research/google-research/tree/mast... https://arxiv.org/abs/1904.04998 https://arxiv.org/abs/1904.04998 We are also seeing good results from using random YouTube videos to train AI vision. But given that you are so sure that this is unsolved, I wonder if I should start a company to sell my depth estimate pipeline. Would you have any example image pairs that are causing problems, so that I can see visually what fails?
  - cpgxiii 6y ago
    If you have a vision system that "just works" and produces high-quality pointclouds in an actual kitchen environment, with reflective appliances, silverware, shiny countertops, shiny ceramic dishware, and glasses, we would absolutely use it. We have in-house research work on both learned monocular depth (so-so for robotics tasks) and learned stereo disparity (much more promising), so progress here isn't impossible, but none of the off-the-shelf products really solve the problem.
    - zippy5 6y ago
      Hypothetically, it seems that your problem has to do surfaces being reflective in the visible light spectrum. Do you have any luck with ultrasonic, ultraviolet, or IR sensors? Also I was also thinking about how light source estimation might be viable work around for reflective surfaces. http://www.thomaswhelan.ie/Whelan16ijrr.pdf http://www.thomaswhelan.ie/Whelan16ijrr.pdf
tuatoru 6y ago
> SLAM ... does not deal with tripping hazards. Do we need to add kinaesthesia to the list of open problems then?
Symmetry 6y ago
In SLAM a pair of shoes constitutes both an obstacle to be avoided as well as an unexpected landmark that might confuse your position estimation. I wouldn't call this an unsolved problem, even textbooks will give techniques to mitigate it, but it is a problem that roboticists need to be aware of.

cpgxiii 6y ago

Depth estimation I wish it were as solved as vendors would like to believe it is. If you want a sensor for medium-range applications, say order 0.5m to 10m depth with better than 1cm accuracy, that works indoors and outdoors, with reflective and untextured objects, doesn't interfere with other sensors of its kind, it simply does not exist. Traditional active and passive stereo is OK provided you have texture, ToF so long as the surfaces aren't reflective and you don't have sunlight to worry about, structured light if you can control lighting and can otherwise control/avoid interference from other sensors. There's some promise in learned stereo matching, but collecting enough data and running fast enough inference are big challenges to practical use. Existing depth sensors for manipulation tasks, even indoors in reasonably controlled lighting, are still mostly insufficient given that the objects they struggle to see are often the objects you want to manipulate.

fxtentacle 6y ago
I strongly disagree! I have seen fantastic results with stereo cameras, colored lights, and self-calibration. For most use cases, it's no problem if your robot will stop for a few seconds, rotate the camera axis around a bit, and then continue. But that appears to be good enough to calibrate the features for tracking things like a reflective and transparent glass jar. As for the precision, I agree that 1cm at 10m distance doesn't work. But 1cm precision at 50cm distance is doable. And for a robot arm, you mainly need the precision when you're close to the object. And yes, I am talking about what you probably meant with learned stereo matching. I would call it close to solved because we can by now do unsupervised training and achieve usable results. While I had trouble reproducing this specific paper, the general idea is valid: https://github.com/google-research/google-research/tree/master/depth_from_video_in_the_wild https://github.com/google-research/google-research/tree/mast... https://arxiv.org/abs/1904.04998 https://arxiv.org/abs/1904.04998 We are also seeing good results from using random YouTube videos to train AI vision. But given that you are so sure that this is unsolved, I wonder if I should start a company to sell my depth estimate pipeline. Would you have any example image pairs that are causing problems, so that I can see visually what fails?
- cpgxiii 6y ago
  If you have a vision system that "just works" and produces high-quality pointclouds in an actual kitchen environment, with reflective appliances, silverware, shiny countertops, shiny ceramic dishware, and glasses, we would absolutely use it. We have in-house research work on both learned monocular depth (so-so for robotics tasks) and learned stereo disparity (much more promising), so progress here isn't impossible, but none of the off-the-shelf products really solve the problem.
  - zippy5 6y ago
    Hypothetically, it seems that your problem has to do surfaces being reflective in the visible light spectrum. Do you have any luck with ultrasonic, ultraviolet, or IR sensors? Also I was also thinking about how light source estimation might be viable work around for reflective surfaces. http://www.thomaswhelan.ie/Whelan16ijrr.pdf http://www.thomaswhelan.ie/Whelan16ijrr.pdf

tuatoru 6y ago

> SLAM ... does not deal with tripping hazards. Do we need to add kinaesthesia to the list of open problems then?

Symmetry 6y ago

In SLAM a pair of shoes constitutes both an obstacle to be avoided as well as an unexpected landmark that might confuse your position estimation. I wouldn't call this an unsolved problem, even textbooks will give techniques to mitigate it, but it is a problem that roboticists need to be aware of.