4 ms·

The example images look convincing, but the sharp hairs of the llama and the cat are pictured against an out-of-focus background... In real life, you'd use the

The example images look convincing, but the sharp hairs of the llama and the cat are pictured against an out-of-focus background... In real life, you'd use these models for synthetic depth-of-field, adding fake bokeh to a very sharp image that's in focus everywhere. so this seems too easy? Impressive latency tho.

JBorrow 2y ago
I don't think the only utility of a depth model is to provide synthetic blurring of backgrounds. There are many things you'd like to use them for, including feeding into object detection pipelines.
dagmx 2y ago
On visionOS 2, there’s functionality to convert 2D images to 3D images for stereo viewing. https://youtu.be/pLfCdI0mjkI?si=8K7rPHu558P-Hf-Z https://youtu.be/pLfCdI0mjkI?si=8K7rPHu558P-Hf-Z I assume the first pass is the depth inference here.
- deleted 2y ago
  [deleted]
amluto 2y ago
I’m not convinced that this type of model is the right solution to fake bokeh, at least not if you use it as a black box. Imagine you have the letter A in the background behind some hair. You should end up with a blurry A and most in-focus hair. Instead you end up with an erratic mess, because a fuzzy depth map doesn’t capture the relevant information. Of course, lots of text-to-image models generate a mess, because their training sets are highly contaminated by the messes produced by “Portrait mode”.
- qingcharles 2y ago
  I use the fake bokeh "lens blur" tool in Adobe Camera Raw every day, and 99% of the time it gets these kinds of problems correct. Every now and again I have to click the tool and adjust the depth map, but for the most part it is amazingly good. I don't know what ML model they use and how far behind this Apple SOTA research Adobe's product is.
  - amluto 2y ago
    From (very) brief searching, Adobe Camera Raw produces a depth map (and lets you edit it), and gets situations wheee the same pixel contains hair and background wrong, exactly the way one would expect if one trying to use a depth map-aware blue. I would expect an ML algorithm actually trained to blur the background behind hair to be able to do better (where the resulting image is the output), and maybe a radiance field or splatting algorithm could also do better. In the latter cases, the hair would be separately represented as a foreground object/source. I could easily be missing something, though. And I did not find any good full-resolution examples.
    - qingcharles 2y ago
      I don't know, TBH. It definitely spools up my GPU hard, but that could just be standard maths, not ML inference. It works very, very well for me. Adobe actually has a definite ML version in Photoshop under their "Neural Filters" named "Depth Blur (beta)", which is actual AI, but I have slightly poorer results from that than I do with their mainstream "Lens Blur" on Camera Raw.

JBorrow 2y ago

I don't think the only utility of a depth model is to provide synthetic blurring of backgrounds. There are many things you'd like to use them for, including feeding into object detection pipelines.

dagmx 2y ago

On visionOS 2, there’s functionality to convert 2D images to 3D images for stereo viewing. https://youtu.be/pLfCdI0mjkI?si=8K7rPHu558P-Hf-Z https://youtu.be/pLfCdI0mjkI?si=8K7rPHu558P-Hf-Z I assume the first pass is the depth inference here.

deleted 2y ago
[deleted]

amluto 2y ago

I’m not convinced that this type of model is the right solution to fake bokeh, at least not if you use it as a black box. Imagine you have the letter A in the background behind some hair. You should end up with a blurry A and most in-focus hair. Instead you end up with an erratic mess, because a fuzzy depth map doesn’t capture the relevant information. Of course, lots of text-to-image models generate a mess, because their training sets are highly contaminated by the messes produced by “Portrait mode”.

qingcharles 2y ago
I use the fake bokeh "lens blur" tool in Adobe Camera Raw every day, and 99% of the time it gets these kinds of problems correct. Every now and again I have to click the tool and adjust the depth map, but for the most part it is amazingly good. I don't know what ML model they use and how far behind this Apple SOTA research Adobe's product is.
- amluto 2y ago
  From (very) brief searching, Adobe Camera Raw produces a depth map (and lets you edit it), and gets situations wheee the same pixel contains hair and background wrong, exactly the way one would expect if one trying to use a depth map-aware blue. I would expect an ML algorithm actually trained to blur the background behind hair to be able to do better (where the resulting image is the output), and maybe a radiance field or splatting algorithm could also do better. In the latter cases, the hair would be separately represented as a foreground object/source. I could easily be missing something, though. And I did not find any good full-resolution examples.
  - qingcharles 2y ago
    I don't know, TBH. It definitely spools up my GPU hard, but that could just be standard maths, not ML inference. It works very, very well for me. Adobe actually has a definite ML version in Photoshop under their "Neural Filters" named "Depth Blur (beta)", which is actual AI, but I have slightly poorer results from that than I do with their mainstream "Lens Blur" on Camera Raw.