3 ms·

I’m not convinced that this type of model is the right solution to fake bokeh, at least not if you use it as a black box. Imagine you have the letter A in the b

by amluto 2y ago

I’m not convinced that this type of model is the right solution to fake bokeh, at least not if you use it as a black box. Imagine you have the letter A in the background behind some hair. You should end up with a blurry A and most in-focus hair. Instead you end up with an erratic mess, because a fuzzy depth map doesn’t capture the relevant information. Of course, lots of text-to-image models generate a mess, because their training sets are highly contaminated by the messes produced by “Portrait mode”.

qingcharles 2y ago
I use the fake bokeh "lens blur" tool in Adobe Camera Raw every day, and 99% of the time it gets these kinds of problems correct. Every now and again I have to click the tool and adjust the depth map, but for the most part it is amazingly good. I don't know what ML model they use and how far behind this Apple SOTA research Adobe's product is.
- amluto 2y ago
  From (very) brief searching, Adobe Camera Raw produces a depth map (and lets you edit it), and gets situations wheee the same pixel contains hair and background wrong, exactly the way one would expect if one trying to use a depth map-aware blue. I would expect an ML algorithm actually trained to blur the background behind hair to be able to do better (where the resulting image is the output), and maybe a radiance field or splatting algorithm could also do better. In the latter cases, the hair would be separately represented as a foreground object/source. I could easily be missing something, though. And I did not find any good full-resolution examples.
  - qingcharles 2y ago
    I don't know, TBH. It definitely spools up my GPU hard, but that could just be standard maths, not ML inference. It works very, very well for me. Adobe actually has a definite ML version in Photoshop under their "Neural Filters" named "Depth Blur (beta)", which is actual AI, but I have slightly poorer results from that than I do with their mainstream "Lens Blur" on Camera Raw.

by amluto 2y ago

qingcharles 2y ago

I use the fake bokeh "lens blur" tool in Adobe Camera Raw every day, and 99% of the time it gets these kinds of problems correct. Every now and again I have to click the tool and adjust the depth map, but for the most part it is amazingly good. I don't know what ML model they use and how far behind this Apple SOTA research Adobe's product is.

amluto 2y ago
From (very) brief searching, Adobe Camera Raw produces a depth map (and lets you edit it), and gets situations wheee the same pixel contains hair and background wrong, exactly the way one would expect if one trying to use a depth map-aware blue. I would expect an ML algorithm actually trained to blur the background behind hair to be able to do better (where the resulting image is the output), and maybe a radiance field or splatting algorithm could also do better. In the latter cases, the hair would be separately represented as a foreground object/source. I could easily be missing something, though. And I did not find any good full-resolution examples.
- qingcharles 2y ago
  I don't know, TBH. It definitely spools up my GPU hard, but that could just be standard maths, not ML inference. It works very, very well for me. Adobe actually has a definite ML version in Photoshop under their "Neural Filters" named "Depth Blur (beta)", which is actual AI, but I have slightly poorer results from that than I do with their mainstream "Lens Blur" on Camera Raw.