5 ms·

If you have a vision system that "just works" and produces high-quality pointclouds in an actual kitchen environment, with reflective appliances, silverware, sh

If you have a vision system that "just works" and produces high-quality pointclouds in an actual kitchen environment, with reflective appliances, silverware, shiny countertops, shiny ceramic dishware, and glasses, we would absolutely use it. We have in-house research work on both learned monocular depth (so-so for robotics tasks) and learned stereo disparity (much more promising), so progress here isn't impossible, but none of the off-the-shelf products really solve the problem.

zippy5 6y ago
Hypothetically, it seems that your problem has to do surfaces being reflective in the visible light spectrum. Do you have any luck with ultrasonic, ultraviolet, or IR sensors? Also I was also thinking about how light source estimation might be viable work around for reflective surfaces. http://www.thomaswhelan.ie/Whelan16ijrr.pdf http://www.thomaswhelan.ie/Whelan16ijrr.pdf
- cpgxiii 6y ago
  Near-IR, which many of the available ToF (Kinect2, K4A, Basler/Lucid/Sony), active stereo (Realsense D4xx, Mynt, plenty more), and many some structured light (all Primesense derivatives, some older Realsense) cameras use, as well as close UV frequencies, suffers much the same issues as visible light. Farther wavelengths in shortwave IR start to have different/better properties, but the costs of cameras are just astronomical (~$20K/camera). Farther UV is also possible, but eventually we're dealing with frequencies that are actually harmful. I do think there's a lot of open possibility in using more than the traditional visible spectrum. Using polarized sensors to remove reflections, or hyperspectral cameras to reduce noise that only appears in certain bands are good ideas. The problem is that we've got lots of cheap imagers that are really well built for visible spectra, and not a lot outside that. Small hyperspectral cameras for things like agricultural drones do exist, but they tend to work by combining multiple sensors (one for each band) which works fine for the ranges a drone operates at but not very useful at the close range indoor robots need. Ultrasonic has a rather bad name in the field, a lot of people having used absolutely terrible robots with basically useless ultrasonic rangefinders early in their careers. In theory sound could be really useful, and there are some very nice (and very expensive) imaging sonars for underwater robotics use, but I'm not aware of any high-resolution ultrasonic sensors for land robots. One minor challenge with ultrasonic sensors in real products is ensuring that they are inaudible to people and pets - when they are almost-audible they are extremely annoying or even painful.
  - hwillis 6y ago
    > but I'm not aware of any high-resolution ultrasonic sensors for land robots. Sound cameras are very cool- passive sonar, but they're audio frequency and not usually depth-sensing. There's a lot of room for improvement there, but its a relatively unknown area and definitely hasn't found its niche. High resolution air sonar is necessarily fairly large, which makes it decidedly unsexy. It also puts a lot of strain on the potential for a low cost BOM when the pcb and chassis are ~>12" wide. > One minor challenge with ultrasonic sensors in real products is ensuring that they are inaudible to people and pets - when they are almost-audible they are extremely annoying or even painful. One thing that really surprised me when I got into it was that this isnt a trivial problem. Part of it is practical- the cheapest and most common sensors are around 60-80 kHz (mechanical size makes them easy to produce, with the drawback of a high-ish time constant). Getting above the range of domestic animals isn't too hard or inconvenient, but it's surprising how extensively evolution has adapted for sound in bats. If you want to avoid harming them you need to keep above ~150 kHz, which gives you a fairly uncomfortably small bandwidth before you're into the highly-attenuating frequency ranges. Underwater it's a different story (biological structures have difficulty resonating at MHz+) but the frequencies that would give you sub-millimeter precision (350 kHz) are way more inconvenient than 40 or 60 kHz. You need totally different types of piezos and the range is sharply limited unless you're projecting volumes that would deafen at audio ranges. Im still very optimistic about the role of sonar even as cheap lidar and depth cameras crowd the space, but its pretty difficult today to make a truly good sensor.
fxtentacle 6y ago
It "just works" for our use case. We put in 4K @ 60fps and receive 960x540 @ 20fps of stereo correspondence pairs. So every matched pixel is averaged over 3 frames in time and 4 pixels in every direction in space, meaning 9x9 convolution kernels. In other words, we make the video super clean by area sampling in space and time. The specific part about our system that makes it usable for me is that for pixels that cannot be matched with a predetermined quality, it'll return a gap marker instead of guessing. For SfM, that means you can just skip those pixels that are affected by reflections moving around. Cooking pots and plates tend to have enough scratches, design, or shape markers to work OK. Wet white floor will usually be flagged as "unknown" except for the grey gaps in between tiles. As for glass, our system can return up to 2 flows per pixel, meaning for a glass mug we get both the mug and a see-through estimate. If you look at Sintel Clean+"s0-10", you'll find that there are some learned matching algorithms that perform quite well under those conditions: http://sintel.is.tue.mpg.de/quant?metric_id=6&selected_pass=1 http://sintel.is.tue.mpg.de/quant?metric_id=6&selected_pass=... We're H-v3 (2nd place) and that 0.284 EPE for s0-10 (slow movements, small disparities) is quite workable, because that means you have on average less than 0.1 pixels of disparity error on the 960x540 depth maps. As for monocular depth, I see that as mostly of a memorization task again. You train a good stereo matching, then do unsupervised learning on stereo data to get the monocular AI. I'm at hajo.me and I currently work on stereo matching and depth mapping with the goal of improving VR. Can you disclose which company or what in general you're working on? Also, do you know any discussion groups where optical flow / stereo matching people usually hang out?
- cpgxiii 6y ago
  Yes, in general if you can capture multiple views you can make up for many dropouts and artifacts. The more you can move, the more likely that you'll get a complete reconstruction. Single-view artifacts are more of a problem when reaching into confined areas or during visual servoing where you don't necessarily have the freedom to move around to get a better view. The results for the Sintel benchmark do look interesting. Do you have a report or some sort of overview of your approach? It would be nice to see a similar benchmark on real recorded scenes, especially if that provided a way to compare learned matching algorithms with available sensor hardware. I think there's reasonable promise in learned stereo matching, especially if we include more information than just visible spectra. Human eyes are so much more than two RGB imagers, so we shouldn't limit our robots to that either. Monocular depth, I agree, seems to be mostly a memorization problem. In cases where you can effectively memorize everything, it will work quite well and the savings in hardware complexity (and physical sensor size) will be well worth the training complexity. I actually think it has much more promise as a backup depth perception method on cars, since objects in a driving context are mostly consistent in size. I have doubts about how useful it will be in manipulation tasks where you may encounter similar objects at a range of sizes. I'm part of a research group at TRI working on home manipulation tasks although I have something of a hobby interest in outdoor robotics as well - in general I'd say the depth sensors for outdoor tasks are often better, but much more expensive and out of reach for hobbyist users.

zippy5 6y ago

Hypothetically, it seems that your problem has to do surfaces being reflective in the visible light spectrum. Do you have any luck with ultrasonic, ultraviolet, or IR sensors? Also I was also thinking about how light source estimation might be viable work around for reflective surfaces. http://www.thomaswhelan.ie/Whelan16ijrr.pdf http://www.thomaswhelan.ie/Whelan16ijrr.pdf

cpgxiii 6y ago
Near-IR, which many of the available ToF (Kinect2, K4A, Basler/Lucid/Sony), active stereo (Realsense D4xx, Mynt, plenty more), and many some structured light (all Primesense derivatives, some older Realsense) cameras use, as well as close UV frequencies, suffers much the same issues as visible light. Farther wavelengths in shortwave IR start to have different/better properties, but the costs of cameras are just astronomical (~$20K/camera). Farther UV is also possible, but eventually we're dealing with frequencies that are actually harmful. I do think there's a lot of open possibility in using more than the traditional visible spectrum. Using polarized sensors to remove reflections, or hyperspectral cameras to reduce noise that only appears in certain bands are good ideas. The problem is that we've got lots of cheap imagers that are really well built for visible spectra, and not a lot outside that. Small hyperspectral cameras for things like agricultural drones do exist, but they tend to work by combining multiple sensors (one for each band) which works fine for the ranges a drone operates at but not very useful at the close range indoor robots need. Ultrasonic has a rather bad name in the field, a lot of people having used absolutely terrible robots with basically useless ultrasonic rangefinders early in their careers. In theory sound could be really useful, and there are some very nice (and very expensive) imaging sonars for underwater robotics use, but I'm not aware of any high-resolution ultrasonic sensors for land robots. One minor challenge with ultrasonic sensors in real products is ensuring that they are inaudible to people and pets - when they are almost-audible they are extremely annoying or even painful.
- hwillis 6y ago
  > but I'm not aware of any high-resolution ultrasonic sensors for land robots. Sound cameras are very cool- passive sonar, but they're audio frequency and not usually depth-sensing. There's a lot of room for improvement there, but its a relatively unknown area and definitely hasn't found its niche. High resolution air sonar is necessarily fairly large, which makes it decidedly unsexy. It also puts a lot of strain on the potential for a low cost BOM when the pcb and chassis are ~>12" wide. > One minor challenge with ultrasonic sensors in real products is ensuring that they are inaudible to people and pets - when they are almost-audible they are extremely annoying or even painful. One thing that really surprised me when I got into it was that this isnt a trivial problem. Part of it is practical- the cheapest and most common sensors are around 60-80 kHz (mechanical size makes them easy to produce, with the drawback of a high-ish time constant). Getting above the range of domestic animals isn't too hard or inconvenient, but it's surprising how extensively evolution has adapted for sound in bats. If you want to avoid harming them you need to keep above ~150 kHz, which gives you a fairly uncomfortably small bandwidth before you're into the highly-attenuating frequency ranges. Underwater it's a different story (biological structures have difficulty resonating at MHz+) but the frequencies that would give you sub-millimeter precision (350 kHz) are way more inconvenient than 40 or 60 kHz. You need totally different types of piezos and the range is sharply limited unless you're projecting volumes that would deafen at audio ranges. Im still very optimistic about the role of sonar even as cheap lidar and depth cameras crowd the space, but its pretty difficult today to make a truly good sensor.

fxtentacle 6y ago

It "just works" for our use case. We put in 4K @ 60fps and receive 960x540 @ 20fps of stereo correspondence pairs. So every matched pixel is averaged over 3 frames in time and 4 pixels in every direction in space, meaning 9x9 convolution kernels. In other words, we make the video super clean by area sampling in space and time. The specific part about our system that makes it usable for me is that for pixels that cannot be matched with a predetermined quality, it'll return a gap marker instead of guessing. For SfM, that means you can just skip those pixels that are affected by reflections moving around. Cooking pots and plates tend to have enough scratches, design, or shape markers to work OK. Wet white floor will usually be flagged as "unknown" except for the grey gaps in between tiles. As for glass, our system can return up to 2 flows per pixel, meaning for a glass mug we get both the mug and a see-through estimate. If you look at Sintel Clean+"s0-10", you'll find that there are some learned matching algorithms that perform quite well under those conditions: http://sintel.is.tue.mpg.de/quant?metric_id=6&selected_pass=1 http://sintel.is.tue.mpg.de/quant?metric_id=6&selected_pass=... We're H-v3 (2nd place) and that 0.284 EPE for s0-10 (slow movements, small disparities) is quite workable, because that means you have on average less than 0.1 pixels of disparity error on the 960x540 depth maps. As for monocular depth, I see that as mostly of a memorization task again. You train a good stereo matching, then do unsupervised learning on stereo data to get the monocular AI. I'm at hajo.me and I currently work on stereo matching and depth mapping with the goal of improving VR. Can you disclose which company or what in general you're working on? Also, do you know any discussion groups where optical flow / stereo matching people usually hang out?

cpgxiii 6y ago
Yes, in general if you can capture multiple views you can make up for many dropouts and artifacts. The more you can move, the more likely that you'll get a complete reconstruction. Single-view artifacts are more of a problem when reaching into confined areas or during visual servoing where you don't necessarily have the freedom to move around to get a better view. The results for the Sintel benchmark do look interesting. Do you have a report or some sort of overview of your approach? It would be nice to see a similar benchmark on real recorded scenes, especially if that provided a way to compare learned matching algorithms with available sensor hardware. I think there's reasonable promise in learned stereo matching, especially if we include more information than just visible spectra. Human eyes are so much more than two RGB imagers, so we shouldn't limit our robots to that either. Monocular depth, I agree, seems to be mostly a memorization problem. In cases where you can effectively memorize everything, it will work quite well and the savings in hardware complexity (and physical sensor size) will be well worth the training complexity. I actually think it has much more promise as a backup depth perception method on cars, since objects in a driving context are mostly consistent in size. I have doubts about how useful it will be in manipulation tasks where you may encounter similar objects at a range of sizes. I'm part of a research group at TRI working on home manipulation tasks although I have something of a hobby interest in outdoor robotics as well - in general I'd say the depth sensors for outdoor tasks are often better, but much more expensive and out of reach for hobbyist users.