6 ms·

The monodepth space is full of people insisting that their models can produce metric depth with no explanation other than "NN does magic" for why metric depth i

The monodepth space is full of people insisting that their models can produce metric depth with no explanation other than "NN does magic" for why metric depth is possible from generic mono images. If you provide a single arbitrary image, you can't generate depth that is immune from scale error (e.g. produce accurate depth for a both an image of a real car and a scale model of the same car). Plausibly, you can train a model that encodes sufficient information about a specific set of imager+lens combinations such that the lens distortion behavior of images captured by those imagers+lenses provides the necessary information to resolve the scale of objects, but that is a much weaker claim than what monodepth researchers generally make. Two notable cases where something like monodepth does reliably work are actually ones where considerably more information is present: in animal eyes there is considerable information about focus available, let alone the fact that eyes are nothing like a planar imager; and phase-detection autofocus uses an entirely different set of data (phase offsets via special lenses) than is used by monodepth models (and, arguably, is mostly a relative incremental process rather than something that produces absolute depth).

EnigmaFlare 2y ago
[dead]
reissbaker 2y ago
You wouldn't use monodepth for self-driving, but I think it's useful for producing natural-looking image transformations; after all, you're working with the same image onscreen that a human eye sees (both the computer and the human eye looking at an onscreen photo have access to the same number of bits of information), so if you can get an accurate read on what a human would visually process the image depth to be, you can use that for reasonable-looking changes. I'm a little surprised there isn't more research on producing depth from short videos, though, since iPhones take Live Photos and presumably could use more of the natural movement and shakiness to interpret more of the true depth in the scene... Presumably much better than processing a single still.
- cpgxiii 2y ago
  > You wouldn't use monodepth for self-driving... If you look where a lot of the money has gone into monodepth, self-driving or driver assistance isn't too far away... Actually, I tend to think self-driving is one of the few places you can make a case for monodepth, as a backup for failures in the rest of your depth sensing suite. You wouldn't want to use it as a primary sensor, but if you're a vehicle driving on a highway and you take critical damage to some of your sensors you do still have to keep driving, if only long enough to get safely off the road, and having something that only needs a single camera is very valuable.
- robotresearcher 2y ago
  There’s a huge literature on ‘structure from motion’ which uses image sequences from a moving camera.
- Someone 2y ago
  >> The monodepth space is full of people insisting that their models can produce metric depth with no explanation other than "NN does magic" for why metric depth is possible from generic mono images. If you provide a single arbitrary image, you can't generate depth that is immune from scale error (e.g. produce accurate depth for a both an image of a real car and a scale model of the same car) > You wouldn't use monodepth for self-driving I don’t think you need accurate metric depth for self-driving. “seconds to impact” plus a reasonable estimate of one’s speed (one second from impact at 100km/hour is more dangerous than at 1km/hour both because braking distances grow faster than linear and because a hit at lower speed is less damaging) is way more useful than “meters away” for that.
robotresearcher 2y ago
The ‘magic’ of the monodepth DNNs is exactly in their ability to learn what scale objects and scenery tends to be in the training data, rather than having to be modeled one object at a time. Use them in inlier situations and they work great. Outlier situations, of course not. You don’t expect them to work in the pitch dark or in featureless white boxes. Same with strangely-scaled environments. So, while scale models are indeed a fundamental problem for monodepth, this is not often a problem in practice. The human visual system has the same limitation. A self driving car camera will never in practice find itself peering into a little box of toy cars. And a phone camera will very rarely be looking into a dollhouse instead of a real house. In the latter case, processing can usually be done based on relative depth. The former does need absolute depth, but it might be safe enough to ignore the case where all the cars and surrounding scenery have been shrunk. And if you don’t want to do that, you have wheel odometry with absolute scale you can compare with the imagery.
- cpgxiii 2y ago
  > The ‘magic’ of the monodepth DNNs is exactly in their ability to learn what scale objects and scenery tends to be in the training data, rather than having to be modeled one object at a time. Yes, and for controlled industrial environments where the domain can be effectively captured in the training dataset, this is plausible. Although a fair number of industrial applications do contain very similar objects at multiple scales, and I don't think much of the existing monodepth work has been meaningfully trained or tested on these cases. > The human visual system has the same limitation Only when looking at picture, not when looking directly at the scene even if you only have one eye. Human vision (and that of many animals, even very small ones like jumping spiders) uses information about the focusing of the eye itself to effectively recover real depth (this mechanism, like structure-from-focus on cameras, is obviously more effective at close distances with shallow depth of field and less effective as the distance and depth of field increase). And, of course, the scanning + focusing behavior of the eye is very different than a perfectly-exposed global-shutter planar imager with deep depth-of-field (which is what the "ideal" car camera would be). > A self driving car camera will never in practice find itself peering into a little box of toy cars. Ignoring the case of someone deliberately trying to confuse the car, it will find itself looking at a large variety of children, of different ages, body shapes and sizes, and clothing - and a scale error there can easily have fatal results. Lest this be considered far-fetched, one particular application of monodepth that has been considered by automakers is car backup/360-view cameras, an application area that exists largely because drivers back over small children with alarming frequency.

EnigmaFlare 2y ago

[dead]

reissbaker 2y ago

You wouldn't use monodepth for self-driving, but I think it's useful for producing natural-looking image transformations; after all, you're working with the same image onscreen that a human eye sees (both the computer and the human eye looking at an onscreen photo have access to the same number of bits of information), so if you can get an accurate read on what a human would visually process the image depth to be, you can use that for reasonable-looking changes. I'm a little surprised there isn't more research on producing depth from short videos, though, since iPhones take Live Photos and presumably could use more of the natural movement and shakiness to interpret more of the true depth in the scene... Presumably much better than processing a single still.

cpgxiii 2y ago
> You wouldn't use monodepth for self-driving... If you look where a lot of the money has gone into monodepth, self-driving or driver assistance isn't too far away... Actually, I tend to think self-driving is one of the few places you can make a case for monodepth, as a backup for failures in the rest of your depth sensing suite. You wouldn't want to use it as a primary sensor, but if you're a vehicle driving on a highway and you take critical damage to some of your sensors you do still have to keep driving, if only long enough to get safely off the road, and having something that only needs a single camera is very valuable.
robotresearcher 2y ago
There’s a huge literature on ‘structure from motion’ which uses image sequences from a moving camera.
Someone 2y ago
>> The monodepth space is full of people insisting that their models can produce metric depth with no explanation other than "NN does magic" for why metric depth is possible from generic mono images. If you provide a single arbitrary image, you can't generate depth that is immune from scale error (e.g. produce accurate depth for a both an image of a real car and a scale model of the same car) > You wouldn't use monodepth for self-driving I don’t think you need accurate metric depth for self-driving. “seconds to impact” plus a reasonable estimate of one’s speed (one second from impact at 100km/hour is more dangerous than at 1km/hour both because braking distances grow faster than linear and because a hit at lower speed is less damaging) is way more useful than “meters away” for that.

robotresearcher 2y ago

The ‘magic’ of the monodepth DNNs is exactly in their ability to learn what scale objects and scenery tends to be in the training data, rather than having to be modeled one object at a time. Use them in inlier situations and they work great. Outlier situations, of course not. You don’t expect them to work in the pitch dark or in featureless white boxes. Same with strangely-scaled environments. So, while scale models are indeed a fundamental problem for monodepth, this is not often a problem in practice. The human visual system has the same limitation. A self driving car camera will never in practice find itself peering into a little box of toy cars. And a phone camera will very rarely be looking into a dollhouse instead of a real house. In the latter case, processing can usually be done based on relative depth. The former does need absolute depth, but it might be safe enough to ignore the case where all the cars and surrounding scenery have been shrunk. And if you don’t want to do that, you have wheel odometry with absolute scale you can compare with the imagery.

cpgxiii 2y ago
> The ‘magic’ of the monodepth DNNs is exactly in their ability to learn what scale objects and scenery tends to be in the training data, rather than having to be modeled one object at a time. Yes, and for controlled industrial environments where the domain can be effectively captured in the training dataset, this is plausible. Although a fair number of industrial applications do contain very similar objects at multiple scales, and I don't think much of the existing monodepth work has been meaningfully trained or tested on these cases. > The human visual system has the same limitation Only when looking at picture, not when looking directly at the scene even if you only have one eye. Human vision (and that of many animals, even very small ones like jumping spiders) uses information about the focusing of the eye itself to effectively recover real depth (this mechanism, like structure-from-focus on cameras, is obviously more effective at close distances with shallow depth of field and less effective as the distance and depth of field increase). And, of course, the scanning + focusing behavior of the eye is very different than a perfectly-exposed global-shutter planar imager with deep depth-of-field (which is what the "ideal" car camera would be). > A self driving car camera will never in practice find itself peering into a little box of toy cars. Ignoring the case of someone deliberately trying to confuse the car, it will find itself looking at a large variety of children, of different ages, body shapes and sizes, and clothing - and a scale error there can easily have fatal results. Lest this be considered far-fetched, one particular application of monodepth that has been considered by automakers is car backup/360-view cameras, an application area that exists largely because drivers back over small children with alarming frequency.