5 ms·

I believe the algorithm could be improved by applying the blur to certain areas/depths of the image without including pixels from very distant depths, and inste

I believe the algorithm could be improved by applying the blur to certain areas/depths of the image without including pixels from very distant depths, and instead blurring/feathering edges with an alpha channel over those distant (large depth separation) pixels. For example, if you look at the left example photo by Rachel Been[1], the hair is blurred together with the distant tree details. If instead the algorithm detected the large depth separation there and applied the foreground blur edge against an alpha mask, I believe the results would look a lot more natural. [1] http://4.bp.blogspot.com/-bZJNDZGLS_U/U03bQE2VzKI/AAAAAAAAARI/yQwRovcDWRQ/s1600/image3.png http://4.bp.blogspot.com/-bZJNDZGLS_U/U03bQE2VzKI/AAAAAAAAAR...

gomox 13y ago
Yes! It shouldn't be too hard to implement either, compared to depth detection.
- Theodores 13y ago
  Actually I would say it is a hard problem. Look at anything filmed in a studio against a blue/green screen. This is with perfect lighting, a perfectly racked camera, a proper camera tripod/dolly, a perfectly even background screen, the presenter wearing safe colours and anything else I forgot to mention controlled. Now look at the fine hair detail. Does it resolve too well in front of the keyed in <weathermap>? Probably not. You need a very highly skilled operator to get the fine hair detail right. Hollywood can afford the pros, a regional news station in Papua New Guinea maybe not so. I would not say fine hair detail is out of the question, however, we might have to live without it when using this faux depth affect for now.
  - gomox 13y ago
    It doesn't seem to be the same league of problem. Because you are going to blur the stuff in the background anyway, you can be agressive with the masking and the blurring will make it imperceptible. With blue/green screens you are making changes much more severe than just blurring (in terms of the colors, hues and shades involved). Source: Hardly an expert, but I am quite a photography geek, and I've done some work on this type of graphical algorithms in the past (not this type of fancy 3D stuff though).
aray 13y ago
I'd love to see some test images with that (just get a side-by-side with the app and a larger aperture camera). As I understand what you're proposing, I'm not sure it would actually be closer to what a large-aperture camera would capture. The light field from the farther depth field should be convolving with the light field from the near depth field. Still, side-by-side would be the best way to view these :) I'll do it later this weekend if I get the chance.
- alanh 13y ago
  What do you mean, "the light field from the farther depth should be convolving with the … near"? Intuitively, grandparent post is right, the hair should be sharp.
  - darkmighty 13y ago
    Nope, in the left example the hair should be blurred even in the edges. But the blur should be such that the details in the background are preserved (it seems sightly off). The math behind this afaik is a variant convolution with depth dependent kernel + occlusion. So in the right the hair is sharp because the blurred background is occluded by the sharp hair, but in the left the frontier isn't sharp because the sharp background isn't occluding the blurry foreground.
- kbrower 13y ago
  Side by side comparison http://onionpants.s3.amazonaws.com/IMG_0455.jpg http://onionpants.s3.amazonaws.com/IMG_0455.jpg
  - aray 13y ago
    Thanks! Great example. Does it look like the large camera is focusing a few inches back from the front of the ball and the Google Camera is sharpest at the center of the ball? Edit: I was trying to look at the sharpness of features on the ball -- specifically the lettering on the left side.
    - 13y ago
darkmighty 13y ago
I think that case was just an imperfection of the depth map (the low depth area seems sightly larger than it should be)
devindotcom 13y ago
Getting a good alpha channel around objects is pretty hard live and with high megapixel counts. If you want to do this in-camera or within half a second, it's going to be sloppy for now. Essentially you have to check out a bunch of different 'magic wand' settings and compare the results, and there's no validation of those results until the user sees them. But yeah, it should definitely be possible to scoot the blur point "backwards" a bit.
SandB0x 13y ago
Occlusion boundaries are always a challenge for vision algorithms. The defects you see are probably due to incorrect depth estimates in these regions. My guess is that for efficiency on a mobile device, they are using a simple window to aggregate the SAD measurements rather than a more complex term which weights errors based on color similarity. The simpler method will not perform well at these boundaries. Getting correct depth and foreground/background separation is a chicken and egg problem, though there are papers which aim for a joint optimization.
- dperfect 13y ago
  True. This may just be due to inaccuracies in the depth map. And therein lies the problem with synthetic aperture algorithms like this: to make it look (close to) natural, you need very clean depth boundaries. There will always be problems/artifacts when trying to calculate accurate boundaries from something as limited as a few seconds of 2D camera data. EDIT: Hard to say if this really is a problem with the depth map. In the right image (foreground in focus), the background blur seems to end pretty close to the hair boundary, if not somewhat inside the boundary of hair - the edge of the hair appears slightly blurred. If the depth map were simply inaccurate (with the same inaccuracy applied in both images), I'm guessing you'd see a portion of sharper background outside the hair boundary. I suppose another explanation would be some biased expansion/feathering of the blur mask.

gomox 13y ago

Yes! It shouldn't be too hard to implement either, compared to depth detection.

Theodores 13y ago
Actually I would say it is a hard problem. Look at anything filmed in a studio against a blue/green screen. This is with perfect lighting, a perfectly racked camera, a proper camera tripod/dolly, a perfectly even background screen, the presenter wearing safe colours and anything else I forgot to mention controlled. Now look at the fine hair detail. Does it resolve too well in front of the keyed in <weathermap>? Probably not. You need a very highly skilled operator to get the fine hair detail right. Hollywood can afford the pros, a regional news station in Papua New Guinea maybe not so. I would not say fine hair detail is out of the question, however, we might have to live without it when using this faux depth affect for now.
- gomox 13y ago
  It doesn't seem to be the same league of problem. Because you are going to blur the stuff in the background anyway, you can be agressive with the masking and the blurring will make it imperceptible. With blue/green screens you are making changes much more severe than just blurring (in terms of the colors, hues and shades involved). Source: Hardly an expert, but I am quite a photography geek, and I've done some work on this type of graphical algorithms in the past (not this type of fancy 3D stuff though).

aray 13y ago

I'd love to see some test images with that (just get a side-by-side with the app and a larger aperture camera). As I understand what you're proposing, I'm not sure it would actually be closer to what a large-aperture camera would capture. The light field from the farther depth field should be convolving with the light field from the near depth field. Still, side-by-side would be the best way to view these :) I'll do it later this weekend if I get the chance.

alanh 13y ago
What do you mean, "the light field from the farther depth should be convolving with the … near"? Intuitively, grandparent post is right, the hair should be sharp.
- darkmighty 13y ago
  Nope, in the left example the hair should be blurred even in the edges. But the blur should be such that the details in the background are preserved (it seems sightly off). The math behind this afaik is a variant convolution with depth dependent kernel + occlusion. So in the right the hair is sharp because the blurred background is occluded by the sharp hair, but in the left the frontier isn't sharp because the sharp background isn't occluding the blurry foreground.
kbrower 13y ago
Side by side comparison http://onionpants.s3.amazonaws.com/IMG_0455.jpg http://onionpants.s3.amazonaws.com/IMG_0455.jpg
- aray 13y ago
  Thanks! Great example. Does it look like the large camera is focusing a few inches back from the front of the ball and the Google Camera is sharpest at the center of the ball? Edit: I was trying to look at the sharpness of features on the ball -- specifically the lettering on the left side.
  - 13y ago

darkmighty 13y ago

I think that case was just an imperfection of the depth map (the low depth area seems sightly larger than it should be)

devindotcom 13y ago

Getting a good alpha channel around objects is pretty hard live and with high megapixel counts. If you want to do this in-camera or within half a second, it's going to be sloppy for now. Essentially you have to check out a bunch of different 'magic wand' settings and compare the results, and there's no validation of those results until the user sees them. But yeah, it should definitely be possible to scoot the blur point "backwards" a bit.

SandB0x 13y ago

Occlusion boundaries are always a challenge for vision algorithms. The defects you see are probably due to incorrect depth estimates in these regions. My guess is that for efficiency on a mobile device, they are using a simple window to aggregate the SAD measurements rather than a more complex term which weights errors based on color similarity. The simpler method will not perform well at these boundaries. Getting correct depth and foreground/background separation is a chicken and egg problem, though there are papers which aim for a joint optimization.

dperfect 13y ago
True. This may just be due to inaccuracies in the depth map. And therein lies the problem with synthetic aperture algorithms like this: to make it look (close to) natural, you need very clean depth boundaries. There will always be problems/artifacts when trying to calculate accurate boundaries from something as limited as a few seconds of 2D camera data. EDIT: Hard to say if this really is a problem with the depth map. In the right image (foreground in focus), the background blur seems to end pretty close to the hair boundary, if not somewhat inside the boundary of hair - the edge of the hair appears slightly blurred. If the depth map were simply inaccurate (with the same inaccuracy applied in both images), I'm guessing you'd see a portion of sharper background outside the hair boundary. I suppose another explanation would be some biased expansion/feathering of the blur mask.