4 ms·

Depth estimation I wish it were as solved as vendors would like to believe it is. If you want a sensor for medium-range applications, say order 0.5m to 10m dep

Depth estimation I wish it were as solved as vendors would like to believe it is. If you want a sensor for medium-range applications, say order 0.5m to 10m depth with better than 1cm accuracy, that works indoors and outdoors, with reflective and untextured objects, doesn't interfere with other sensors of its kind, it simply does not exist. Traditional active and passive stereo is OK provided you have texture, ToF so long as the surfaces aren't reflective and you don't have sunlight to worry about, structured light if you can control lighting and can otherwise control/avoid interference from other sensors. There's some promise in learned stereo matching, but collecting enough data and running fast enough inference are big challenges to practical use. Existing depth sensors for manipulation tasks, even indoors in reasonably controlled lighting, are still mostly insufficient given that the objects they struggle to see are often the objects you want to manipulate.

fxtentacle 6y ago
I strongly disagree! I have seen fantastic results with stereo cameras, colored lights, and self-calibration. For most use cases, it's no problem if your robot will stop for a few seconds, rotate the camera axis around a bit, and then continue. But that appears to be good enough to calibrate the features for tracking things like a reflective and transparent glass jar. As for the precision, I agree that 1cm at 10m distance doesn't work. But 1cm precision at 50cm distance is doable. And for a robot arm, you mainly need the precision when you're close to the object. And yes, I am talking about what you probably meant with learned stereo matching. I would call it close to solved because we can by now do unsupervised training and achieve usable results. While I had trouble reproducing this specific paper, the general idea is valid: https://github.com/google-research/google-research/tree/master/depth_from_video_in_the_wild https://github.com/google-research/google-research/tree/mast... https://arxiv.org/abs/1904.04998 https://arxiv.org/abs/1904.04998 We are also seeing good results from using random YouTube videos to train AI vision. But given that you are so sure that this is unsolved, I wonder if I should start a company to sell my depth estimate pipeline. Would you have any example image pairs that are causing problems, so that I can see visually what fails?
- cpgxiii 6y ago
  If you have a vision system that "just works" and produces high-quality pointclouds in an actual kitchen environment, with reflective appliances, silverware, shiny countertops, shiny ceramic dishware, and glasses, we would absolutely use it. We have in-house research work on both learned monocular depth (so-so for robotics tasks) and learned stereo disparity (much more promising), so progress here isn't impossible, but none of the off-the-shelf products really solve the problem.
  - zippy5 6y ago
    Hypothetically, it seems that your problem has to do surfaces being reflective in the visible light spectrum. Do you have any luck with ultrasonic, ultraviolet, or IR sensors? Also I was also thinking about how light source estimation might be viable work around for reflective surfaces. http://www.thomaswhelan.ie/Whelan16ijrr.pdf http://www.thomaswhelan.ie/Whelan16ijrr.pdf
    - cpgxiii 6y ago
      Near-IR, which many of the available ToF (Kinect2, K4A, Basler/Lucid/Sony), active stereo (Realsense D4xx, Mynt, plenty more), and many some structured light (all Primesense derivatives, some older Realsense) cameras use, as well as close UV frequencies, suffers much the same issues as visible light. Farther wavelengths in shortwave IR start to have different/better properties, but the costs of cameras are just astronomical (~$20K/camera). Farther UV is also possible, but eventually we're dealing with frequencies that are actually harmful. I do think there's a lot of open possibility in using more than the traditional visible spectrum. Using polarized sensors to remove reflections, or hyperspectral cameras to reduce noise that only appears in certain bands are good ideas. The problem is that we've got lots of cheap imagers that are really well built for visible spectra, and not a lot outside that. Small hyperspectral cameras for things like agricultural drones do exist, but they tend to work by combining multiple sensors (one for each band) which works fine for the ranges a drone operates at but not very useful at the close range indoor robots need. Ultrasonic has a rather bad name in the field, a lot of people having used absolutely terrible robots with basically useless ultrasonic rangefinders early in their careers. In theory sound could be really useful, and there are some very nice (and very expensive) imaging sonars for underwater robotics use, but I'm not aware of any high-resolution ultrasonic sensors for land robots. One minor challenge with ultrasonic sensors in real products is ensuring that they are inaudible to people and pets - when they are almost-audible they are extremely annoying or even painful.
snovv_crash 6y ago
I'm in the same boat. I'm now leaning toward the idea that single-shot depth estimation isn't the way to go. Throwing away all of the rich motion and flow information from previous frames seems like such a waste when we could be using it to update a dense model instead.

fxtentacle 6y ago

I strongly disagree! I have seen fantastic results with stereo cameras, colored lights, and self-calibration. For most use cases, it's no problem if your robot will stop for a few seconds, rotate the camera axis around a bit, and then continue. But that appears to be good enough to calibrate the features for tracking things like a reflective and transparent glass jar. As for the precision, I agree that 1cm at 10m distance doesn't work. But 1cm precision at 50cm distance is doable. And for a robot arm, you mainly need the precision when you're close to the object. And yes, I am talking about what you probably meant with learned stereo matching. I would call it close to solved because we can by now do unsupervised training and achieve usable results. While I had trouble reproducing this specific paper, the general idea is valid: https://github.com/google-research/google-research/tree/master/depth_from_video_in_the_wild https://github.com/google-research/google-research/tree/mast... https://arxiv.org/abs/1904.04998 https://arxiv.org/abs/1904.04998 We are also seeing good results from using random YouTube videos to train AI vision. But given that you are so sure that this is unsolved, I wonder if I should start a company to sell my depth estimate pipeline. Would you have any example image pairs that are causing problems, so that I can see visually what fails?

cpgxiii 6y ago
If you have a vision system that "just works" and produces high-quality pointclouds in an actual kitchen environment, with reflective appliances, silverware, shiny countertops, shiny ceramic dishware, and glasses, we would absolutely use it. We have in-house research work on both learned monocular depth (so-so for robotics tasks) and learned stereo disparity (much more promising), so progress here isn't impossible, but none of the off-the-shelf products really solve the problem.
- zippy5 6y ago
  Hypothetically, it seems that your problem has to do surfaces being reflective in the visible light spectrum. Do you have any luck with ultrasonic, ultraviolet, or IR sensors? Also I was also thinking about how light source estimation might be viable work around for reflective surfaces. http://www.thomaswhelan.ie/Whelan16ijrr.pdf http://www.thomaswhelan.ie/Whelan16ijrr.pdf
  - cpgxiii 6y ago
    Near-IR, which many of the available ToF (Kinect2, K4A, Basler/Lucid/Sony), active stereo (Realsense D4xx, Mynt, plenty more), and many some structured light (all Primesense derivatives, some older Realsense) cameras use, as well as close UV frequencies, suffers much the same issues as visible light. Farther wavelengths in shortwave IR start to have different/better properties, but the costs of cameras are just astronomical (~$20K/camera). Farther UV is also possible, but eventually we're dealing with frequencies that are actually harmful. I do think there's a lot of open possibility in using more than the traditional visible spectrum. Using polarized sensors to remove reflections, or hyperspectral cameras to reduce noise that only appears in certain bands are good ideas. The problem is that we've got lots of cheap imagers that are really well built for visible spectra, and not a lot outside that. Small hyperspectral cameras for things like agricultural drones do exist, but they tend to work by combining multiple sensors (one for each band) which works fine for the ranges a drone operates at but not very useful at the close range indoor robots need. Ultrasonic has a rather bad name in the field, a lot of people having used absolutely terrible robots with basically useless ultrasonic rangefinders early in their careers. In theory sound could be really useful, and there are some very nice (and very expensive) imaging sonars for underwater robotics use, but I'm not aware of any high-resolution ultrasonic sensors for land robots. One minor challenge with ultrasonic sensors in real products is ensuring that they are inaudible to people and pets - when they are almost-audible they are extremely annoying or even painful.

snovv_crash 6y ago

I'm in the same boat. I'm now leaning toward the idea that single-shot depth estimation isn't the way to go. Throwing away all of the rich motion and flow information from previous frames seems like such a waste when we could be using it to update a dense model instead.