4 ms·

It "just works" for our use case. We put in 4K @ 60fps and receive 960x540 @ 20fps of stereo correspondence pairs. So every matched pixel is averaged over 3 fra

It "just works" for our use case. We put in 4K @ 60fps and receive 960x540 @ 20fps of stereo correspondence pairs. So every matched pixel is averaged over 3 frames in time and 4 pixels in every direction in space, meaning 9x9 convolution kernels. In other words, we make the video super clean by area sampling in space and time. The specific part about our system that makes it usable for me is that for pixels that cannot be matched with a predetermined quality, it'll return a gap marker instead of guessing. For SfM, that means you can just skip those pixels that are affected by reflections moving around. Cooking pots and plates tend to have enough scratches, design, or shape markers to work OK. Wet white floor will usually be flagged as "unknown" except for the grey gaps in between tiles. As for glass, our system can return up to 2 flows per pixel, meaning for a glass mug we get both the mug and a see-through estimate. If you look at Sintel Clean+"s0-10", you'll find that there are some learned matching algorithms that perform quite well under those conditions: http://sintel.is.tue.mpg.de/quant?metric_id=6&selected_pass=1 http://sintel.is.tue.mpg.de/quant?metric_id=6&selected_pass=... We're H-v3 (2nd place) and that 0.284 EPE for s0-10 (slow movements, small disparities) is quite workable, because that means you have on average less than 0.1 pixels of disparity error on the 960x540 depth maps. As for monocular depth, I see that as mostly of a memorization task again. You train a good stereo matching, then do unsupervised learning on stereo data to get the monocular AI. I'm at hajo.me and I currently work on stereo matching and depth mapping with the goal of improving VR. Can you disclose which company or what in general you're working on? Also, do you know any discussion groups where optical flow / stereo matching people usually hang out?

cpgxiii 6y ago
Yes, in general if you can capture multiple views you can make up for many dropouts and artifacts. The more you can move, the more likely that you'll get a complete reconstruction. Single-view artifacts are more of a problem when reaching into confined areas or during visual servoing where you don't necessarily have the freedom to move around to get a better view. The results for the Sintel benchmark do look interesting. Do you have a report or some sort of overview of your approach? It would be nice to see a similar benchmark on real recorded scenes, especially if that provided a way to compare learned matching algorithms with available sensor hardware. I think there's reasonable promise in learned stereo matching, especially if we include more information than just visible spectra. Human eyes are so much more than two RGB imagers, so we shouldn't limit our robots to that either. Monocular depth, I agree, seems to be mostly a memorization problem. In cases where you can effectively memorize everything, it will work quite well and the savings in hardware complexity (and physical sensor size) will be well worth the training complexity. I actually think it has much more promise as a backup depth perception method on cars, since objects in a driving context are mostly consistent in size. I have doubts about how useful it will be in manipulation tasks where you may encounter similar objects at a range of sizes. I'm part of a research group at TRI working on home manipulation tasks although I have something of a hobby interest in outdoor robotics as well - in general I'd say the depth sensors for outdoor tasks are often better, but much more expensive and out of reach for hobbyist users.
- fxtentacle 6y ago
  One trick that I have seen for confined environments is to rotate the plate with both stereo cameras around its forward axis. That way, you can convert the left-right stereo to up-down stereo and especially with highly reflective stuff, there's a chance that that will be enough of a change to reduce reflections.

cpgxiii 6y ago

Yes, in general if you can capture multiple views you can make up for many dropouts and artifacts. The more you can move, the more likely that you'll get a complete reconstruction. Single-view artifacts are more of a problem when reaching into confined areas or during visual servoing where you don't necessarily have the freedom to move around to get a better view. The results for the Sintel benchmark do look interesting. Do you have a report or some sort of overview of your approach? It would be nice to see a similar benchmark on real recorded scenes, especially if that provided a way to compare learned matching algorithms with available sensor hardware. I think there's reasonable promise in learned stereo matching, especially if we include more information than just visible spectra. Human eyes are so much more than two RGB imagers, so we shouldn't limit our robots to that either. Monocular depth, I agree, seems to be mostly a memorization problem. In cases where you can effectively memorize everything, it will work quite well and the savings in hardware complexity (and physical sensor size) will be well worth the training complexity. I actually think it has much more promise as a backup depth perception method on cars, since objects in a driving context are mostly consistent in size. I have doubts about how useful it will be in manipulation tasks where you may encounter similar objects at a range of sizes. I'm part of a research group at TRI working on home manipulation tasks although I have something of a hobby interest in outdoor robotics as well - in general I'd say the depth sensors for outdoor tasks are often better, but much more expensive and out of reach for hobbyist users.

fxtentacle 6y ago
One trick that I have seen for confined environments is to rotate the plate with both stereo cameras around its forward axis. That way, you can convert the left-right stereo to up-down stereo and especially with highly reflective stuff, there's a chance that that will be enough of a change to reduce reflections.