3 ms·

False color depth maps are extremely misleading. The way to judge the quality of a depth map is to use it to reproject the image into 3D and rotate it around a

False color depth maps are extremely misleading. The way to judge the quality of a depth map is to use it to reproject the image into 3D and rotate it around a bit. Papers almost never do this because it makes their artifacts extremely obvious. I'd bet that if you did that on these examples you'd see that the hair, rather than being attached to the animal, is floating halfway between the animal and the background. Of course, depth mapping is an ill-posed problem. The hair is not completely opaque and the pixels in that region have contributions from both the hair and the background, so the neural net is doing the best it can. To really handle hair correctly you would have to output a list of depths (and colors) per pixel, rather than a single depth, so pixels with contributions from multiple objects could be accurately represented.

jonas21 2y ago
> The way to judge the quality of a depth map is to use it to reproject the image into 3D and rotate it around a bit. They do this. See figure 4 in the paper. Are the results cherry-picked to look good? Probably. But so is everything else.
- threeseed 2y ago
  > Figure 4 We plug depth maps produced by Depth Pro, Marigold, Depth Anything v2, and Metric3D v2 into a recent publicly available novel view synthesis system. We demonstrate results on images from AM-2k. Depth Pro produces sharper and more accurate depth maps, yielding cleaner synthesized views. Depth Anything v2 and Metric3D v2 suffer from misalignment between the input images and estimated depth maps, resulting in foreground pixels bleeding into the background. Marigold is considerably slower than Depth Pro and produces less accurate boundaries, yielding artifacts in synthesized images.
incrudible 2y ago
> I'd bet that if you did that on these examples you'd see that the hair, rather than being attached to the animal, is floating halfway between the animal and the background. You're correct about that, but for something like matte/depth-treshold that's exactly what you want to get a smooth and controllable transition within the limited amount of resolution you have. For that use case, especially with the fuzzy hair, it's pretty good.
- modeless 2y ago
  It's not exactly what you want because you will get both background bleeding into the foreground and clipping of the parts of the foreground that fall under your threshold. What you want is for the neural net to estimate the different color contributions of background and foreground at each pixel so you can separate them without bleeding or clipping.
  - incrudible 2y ago
    It's what you'd want out of a depth map used for that purpose. What you're describing is not a depth map.
    - zardo 2y ago
      Maybe the depthMap should only accept images that have been typed as hairless.
  - Stedag 2y ago
    I work on Time-of-flight camera's that need to handle the kind of data that you're referring too. Each pixel takes a multiple measurements over time of the intensity of reflected light that matches the emission pulse encodings. The result is essentially a vector of intensity over a set of distances. A low depth resolution example of reflected intensity by time (distance): i: _ _ ^ _ ^ - _ _ d: 0 1 2 3 4 5 6 7 In the above example, the pixel would exhibit an ambiguity between distances of 2 and 4. The simplest solution is to select the weighted average or median distance, which results in "flying pixels" or "mixed pixels" for which there are existing efficient techniques for filtration. The bottom line is that for applications like low-latency obstacle detection on a cost-constrained mobile robot, there's some compression of depth information required. For the sake of inferring a highly realistic model from an image, Neural radiance fields or gaussian splats may best generate the representation that you might be envisioning, where there would be a volumetric representation of material properties like hair. This comes with higher compute costs however and doesn't factor in semantic interpretation of a scene. The Top performing results in photogrammetry have tended to use a combination of less expensive techniques like this one to better handle sparsity of scene coverage, and then refining the a result using more expensive techniques [1]. 1: https://arxiv.org/pdf/2404.08252 https://arxiv.org/pdf/2404.08252

jonas21 2y ago

> The way to judge the quality of a depth map is to use it to reproject the image into 3D and rotate it around a bit. They do this. See figure 4 in the paper. Are the results cherry-picked to look good? Probably. But so is everything else.

threeseed 2y ago
> Figure 4 We plug depth maps produced by Depth Pro, Marigold, Depth Anything v2, and Metric3D v2 into a recent publicly available novel view synthesis system. We demonstrate results on images from AM-2k. Depth Pro produces sharper and more accurate depth maps, yielding cleaner synthesized views. Depth Anything v2 and Metric3D v2 suffer from misalignment between the input images and estimated depth maps, resulting in foreground pixels bleeding into the background. Marigold is considerably slower than Depth Pro and produces less accurate boundaries, yielding artifacts in synthesized images.

incrudible 2y ago

> I'd bet that if you did that on these examples you'd see that the hair, rather than being attached to the animal, is floating halfway between the animal and the background. You're correct about that, but for something like matte/depth-treshold that's exactly what you want to get a smooth and controllable transition within the limited amount of resolution you have. For that use case, especially with the fuzzy hair, it's pretty good.

modeless 2y ago
It's not exactly what you want because you will get both background bleeding into the foreground and clipping of the parts of the foreground that fall under your threshold. What you want is for the neural net to estimate the different color contributions of background and foreground at each pixel so you can separate them without bleeding or clipping.
- incrudible 2y ago
  It's what you'd want out of a depth map used for that purpose. What you're describing is not a depth map.
  - zardo 2y ago
    Maybe the depthMap should only accept images that have been typed as hairless.
- Stedag 2y ago
  I work on Time-of-flight camera's that need to handle the kind of data that you're referring too. Each pixel takes a multiple measurements over time of the intensity of reflected light that matches the emission pulse encodings. The result is essentially a vector of intensity over a set of distances. A low depth resolution example of reflected intensity by time (distance): i: _ _ ^ _ ^ - _ _ d: 0 1 2 3 4 5 6 7 In the above example, the pixel would exhibit an ambiguity between distances of 2 and 4. The simplest solution is to select the weighted average or median distance, which results in "flying pixels" or "mixed pixels" for which there are existing efficient techniques for filtration. The bottom line is that for applications like low-latency obstacle detection on a cost-constrained mobile robot, there's some compression of depth information required. For the sake of inferring a highly realistic model from an image, Neural radiance fields or gaussian splats may best generate the representation that you might be envisioning, where there would be a volumetric representation of material properties like hair. This comes with higher compute costs however and doesn't factor in semantic interpretation of a scene. The Top performing results in photogrammetry have tended to use a combination of less expensive techniques like this one to better handle sparsity of scene coverage, and then refining the a result using more expensive techniques [1]. 1: https://arxiv.org/pdf/2404.08252 https://arxiv.org/pdf/2404.08252