17 ms·

Lens Blur in the new Google Camera app

644 points by cleverjake 13y ago

spyder 13y ago
But it can be used only on static subjects because it needs series of frames for depth.
bckrasnow 13y ago
Well, the Lytro guys are screwed now. They're selling a $400 camera with this feature as the main selling point.
- sjtrny 13y ago
  Except you don't have to move the Lytro
guardian5x 13y ago
I guess that is exactly the same as Nokias Refocus that is on the Lumia Phones for quite some time: https://refocus.nokia.com/ https://refocus.nokia.com/
- refrigerator 13y ago
  Refocus is awesome and I owe many but not as powerful as this at all - Refocus takes like 10 photos at different levels of focus, so you're limited to the amount of bokeh that the small phone sensor can produce, i.e you only get a nice blurred background when you're taking a really close up picture of something. The new Google Camera feature lets you take a portrait at a normal distance, and simulate f1.8, which is a lot better than what Refocus can do.
goatslacker 13y ago
On iOS you can customize your DoF with an app called Big Lens. Normally apps like Instagram and Fotor let you pick one point in the picture or a vertical/horizontal segment and apply focus there while blurring the background. Big Lens is more advanced since it lets you draw with your finger what you'd like to be in focus. They also include various apertures you can set (as low as f/1.8) as well as some filters -- although I personally find the filters to be overdone but others might find them tasteful.
scep12 13y ago
Impressive feat. Took a few snaps on my Nexus 4 and it seems to work really well given a decent scene.
- ctingom 13y ago
  Agreed. This has been one of my biggest gripes with Instagram's blur feature as well.
- jevinskie 13y ago
  I tried it out as well. Good results, especially from their first public iteration.
deleted 13y ago
[deleted]
jawns 13y ago
Regarding the technology (achieving shallow depth of field through an algorithm), not Google's specific implementation ... Up until now, a decently shallow depth of field was pretty much only achievable in DSLR cameras (and compacts with sufficiently large sensor sizes, which typically cost as much as a DSLR). You can simulate it in Photoshop, but generally it takes a lot of work and the results aren't great. The "shallow depth of field" effect was one of the primary reasons why I bought a DSLR. (Yeah, yeah, yeah, quality of the lens and sensor are important too.) Being able to achieve a passable blur effect, even if it's imperfect, on a cellphone camera is really pretty awesome, considering the convenience factor. And if you wanted to be able to change the focus after you take the picture, you had to get a Lytro light field camera -- again, as expensive as a DSLR, but with a more limited feature set. Regarding Google's specific implementation ... I've got a Samsung Galaxy S4 Zoom, which hasn't yet gotten the Android 4.4 update, so I can't use the app itself to evaluate the Lens Blur feature, but based on the examples in the blog post, it's pretty good. It's clearly not indistinguishable from optical shallow depth of field, but it's not so bad that it's glaring. That you can adjust the focus after you shoot is icing on the cake, but tremendously delicious icing. The S4 Zoom is a really terrific point-and-shoot that happens to have a phone, so I'm excited to try it out. Even if I can use it in just 50% of the cases where I now lean on my DLSR, it'll save me from having to lug a bulky camera around AND be easier to share over wifi/data.
- theoh 13y ago
  I don't know for sure but the code may be derived from or related to Marc Levoy's SynthCam: https://sites.google.com/site/marclevoy/ https://sites.google.com/site/marclevoy/ It does similar things, and in fact I could believe it if the Google app was just a dumbed down version of the functionality intended to be usable by a wider audience.
  - aray 13y ago
    There have been multiple implementations of this class of algorithm (collectively "synthetic aperture") using different techniques. Edit: below mentions the HTC "double camera" phones, and in fact it's also possible to create synthetic aperture photography with multiple cameras instead of moving a single camera to multiple positions. Then you have the added advantage of not needing to assume a static scene! But a single-camera algorithm is great for rolling out to the common devices most of us have in our pockets. Marc Levoy's SynthCam used a circular wiping motion, and attempted to "paint" the 3d space occupied by your theoretical processed aperture. This Google Camera only requires a single linear move, and processes the rest!
gamesurgeon 13y ago
One of the greatest features is the ability to change your focus point AFTER you shoot. This is huge.
- wmf 13y ago
  As long as the original photo was sharp.
- Panoramix 13y ago
  Kind of like an all-software Lytro https://www.lytro.com/ https://www.lytro.com/
jnevelson 13y ago
So Google basically took what Lytro has been using hardware to achieve, and did it entirely in software. Pretty impressive.
- ISL 13y ago
  The difference is that Lytro actually does it, instead of simulating it. It'd be fun to play around with the software to see in which cases it breaks (perhaps taking a photo of a framed landscape photo with another landscape behind, for example)
  - apu 13y ago
    Lytro is also "just" simulating it, but with slightly more/different data. They capture a light field but that doesn't magically give them depth values; they have to estimate them using an optimization, and then render the final image using a very similar algorithm.
    - ISL 13y ago
      Edit: Whoops: I mis-read the article; 'Lens Blur' uses multiple frames. That lets you get a lot more information.
      - ISL 13y ago
        Well, The Lytro will still win on moving subjects where a sequence convolves subject motion with the depth map.
  - frozenport 13y ago
    In reality the Lytro has abysmal performance, which questions the value of lightfield technology as a whole. Instead of capturing the 'light field' you may as well capture most of the image. Secondly the loss of the 2nd derivative of the k-vector makes lightfield currently unsuitable for use in a microscope (although they are working on it...). Although there might be better techniques for refocusing, such as division in the spatial frequency domain. see http://www.cnet.com.au/nokia-vs-lytro-the-refocusing-challenge-339346170.htm http://www.cnet.com.au/nokia-vs-lytro-the-refocusing-challen...
frenchman_in_ny 13y ago
Does this pretty much blow Lytro out of the water, and mean that you no longer need dedicated hardware to do this?
- gtremper 13y ago
  This doesn't capture the whole light field, which is the advantage of a light field camera. Changing the focus is just a fun application if you have access to the whole light field. http://en.wikipedia.org/wiki/Light_field http://en.wikipedia.org/wiki/Light_field
- sp332 13y ago
  The Lytro images have a lot more data in them (although probably fewer pixels). You can move around slightly in a Lytro photo because it samples light from all directions. Also, Google's version doesn't seem to use the multiple exposures in the final photo. It only uses them to determine which pixels to blur in the first photo, and uses normal gaussian blur instead of simulating a lens.
  - avaku 13y ago
    It does simulate the lens, it says so in the article
    - sp332 13y ago
      Oh... well it doesn't do a very good job. The last example has her hair blurred where it should be sharp. That's not very lens-like.
      - avaku 13y ago
        Yeah, in the first example too (hair at the bottom). But it's still much better than Gaussian blur...
bitJericho 13y ago
If you couple this with instagram does it break the cosmological fabric?
themgt 13y ago
Is looking at the examples giving anyone else a headache? It's like the software blur falls into some kind of uncanny valley for reality.
- zippergz 13y ago
  It doesn't give me a headache, but to my eyes it isn't remotely convincing. There's no way I'd look at any of those example pictures and not realize the blurring was done with postprocessing.
  - JTon 13y ago
    I'm a photography layman and I really can't tell it was done with postprocessing. What gives it away for you?
    - bronson 13y ago
      The blur is far too uniform and boring. Actual bokeh has more intrigue to it: lens aberrations, crazy highlights, wonderful patterns, even when the depth of field isn't very extreme. Check out some real examples on google image search: https://www.google.com/search?q=bokeh+examples&espv=210&es_sm=91&tbm=isch&imgil=4yWA5_ormTgB0M%253A%253Bhttps%253A%252F%252Fencrypted-tbn0.gstatic.com%252Fimages%253Fq%253Dtbn%253AANd9GcRLNLx7njG3LMIaGPxX-biipqDXL1X8ayqBMerER-ru4kInUwuR%253B500%253B498%253B5PGfCFJ_2qPyJM%253Bhttp%25253A%25252F%25252Fwww.slrphotographyguide.com%25252Ftips%25252Fbokeh-DOF.shtml&source=iu&usg=__4dcsDgtrRNOwCo3ZfbA8svWIPJM%3D&sa=X&ei=cNtOU8_UGIrn8gHHyYCYDA&ved=0CDEQ9QEwAw#facrc=_&imgdii=_&imgrc=4yWA5_ormTgB0M%253A%3B5PGfCFJ_2qPyJM%3Bhttp%253A%252F%252Fwww.slrphotographyguide.com%252Ftips%252Fimages%252Fcirclular-bokeh.jpg%3Bhttp%253A%252F%252Fwww.slrphotographyguide.com%252Ftips%252Fbokeh-DOF.shtml%3B500%3B498 https://www.google.com/search?q=bokeh+examples&espv=210&es_s... I'm impressed! Looks like Google solved the hard problem. Implementing a more pleasing blur than plain gaussian doesn't sound too hard.
      - justincormack 13y ago
        Especially in large format petzval and similar lenses eg https://www.flickr.com/photos/richard314159/galleries/72157624453174055/ https://www.flickr.com/photos/richard314159/galleries/721576...
Spittie 13y ago
I find it funny that this was one of the "exclusive features" of the HTC One M8 thanks to the double camera, and days after it's release Google is giving the same ability to every Android phones. I'm sure the HTC implementation works better, but this is still impressive.
- dannyr 13y ago
  Well, only Android 4.4.x phones. But coming to other Android version.
  - pjmlp 13y ago
    Where do they state < 4.4 users would ever get it? I only find mentions of 4.4.
- kbrower 13y ago
  From what I have seen the HTC implementation does not work better. It looks really synthetic. Google is doing a much better job.
grecy 13y ago
We had an interesting discussion about this a few nights ago at a Photojournalism talk. In that field, digital edits are seriously banned, to the point multiple very well known photo journalists have been fired for one little use of the clone tool [1] and other minor edits. It's interesting to think I can throw an f/1.8 lens on my DSLR and take a very shallow depth of field photo, which is OK, even though it's not very representative of what my eyes saw. If I take the photo at f/18 then use an app like the one linked, producing extremely similar results, that's banned. Fascinating what's allowed and what's not. I find even more interesting is the allowance of changing color photos to B/W, or that almost anything that "came straight off the camera" no matter how far it strays from what your eyes saw. [1] http://www.toledoblade.com/frontpage/2007/04/15/A-basic-rule-Newspaper-photos-must-tell-the-truth.html http://www.toledoblade.com/frontpage/2007/04/15/A-basic-rule...
- VikingCoder 13y ago
  Radiologists are the same... You take a Computed Tomography series of images, doing all kinds of back-projection (duh) and volumetric anisotropic edge-preserving smoothing, and then you apply segmentation and false-color, semi-transparent transfer functions and lighting... And then you apply one more post-processing effect to try to highlight something, and they freak out that you're no longer showing them the raw data! That does not mean what you think it does.
- headShrinker 13y ago
  Photojournalism is a weird cross over of history/evidence collection and drifting in to art. They fit together like oil and water, for all the reasons you pointed out.
  - bowerbird 13y ago
    on the other hand, i know photographers in art/advertising -- working for some of the biggest companies, like apple -- who brag that "not one pixel goes out without being touched". -bowerbird
- aray 13y ago
  Could you speak to how familiar photojournalists are with the math and transformations that take place inside a digital camera? For example, almost certainly your cameras have dead pixels, which are processed away during the demosaicing stage, but showing them would be a "straight off the camera image" that I doubt any photojournalist would desire. Additionally many scene-wide process steps (like lens shading map estimation) can be changed in post-processing if the cameras automatic algorithms (3a, etc) "decided" wrong.
the_cat_kittles 13y ago
Isn't it interesting how, by diminishing the overall information content of the image by blurring it, it actually communicates more (in some ways, particularly depth) to the viewer?
- selmnoo 13y ago
  Well, usually it's for a specific purpose (drawing attention to something) or aesthetic that you use a shallow depth of field. There have been artists in the past that tried to go extreme in the other direction, trying to capture as wide DOF as they could. This is a good example: http://gallery.realitydesign.com/dof.jpg http://gallery.realitydesign.com/dof.jpg Both of these are good individually. But what is best depends totally on what exactly you're going for. Stanley Kubrick did very interesting experimentation with this.
sib 13y ago
If only they had not confused shallow depth of field with Bokeh (which is not the shallowness of the depth of field, but, rather, how out-of-focus areas are rendered), this writeup would have been much better. http://en.wikipedia.org/wiki/Bokeh http://en.wikipedia.org/wiki/Bokeh Cool technology, though.
- jcampbell1 13y ago
  Bokeh = shallow depth of field effects The author isn't the one confused.
  - sib 13y ago
    Bokeh is the quality of the rendering of out of focus areas. It's not just making something out of focus (or creating shallow depth of field). 'Bokeh has been defined as "the way the lens renders out-of-focus points of light".'
  - chernand 13y ago
    Hi, I am the author of the Lens Blur blog post and the sentence was indeed wrong as a result of multiple edits. Bokeh and shallow depth of field are indeed two different things. By Bokeh we mean that the blur is synthesized using a disk kernel, e.g. as opposed to a Gaussian Blur. The blog is now fixed.
    - josephagoss 13y ago
      Hey, is there any chance we can download the RAW data alongside the JPG file?
      - chernand 13y ago
        You can extract the computed depthmap, all-in-focus image, and focus settings from the XMP data of the jpeg. See https://developers.google.com/depthmap-metadata/ https://developers.google.com/depthmap-metadata/ for the depthmap format.
mcescalante 13y ago
I may be wrong because I don't know much about image based algorithms, but this seems to be a pretty successful new approach to achieving this effect. Are there any other existing "lens blur" or depth of field tricks that phone makers or apps are using? I'd love to see their code open sourced.
- matthiasv 13y ago
  The novelty I see here, is the combination and integration into an app. Creating blurred images from a photograph and its depth map can be achieved with a GIMP plugin since at least 2008 [1]. Combining a series of photos to create a 3D model is also widely implemented, e.g. in the Photosynth application developed at U Washington/Microsoft which according to Wikipedia exists as a tech preview since 2006. [1] http://registry.gimp.org/node/1444 http://registry.gimp.org/node/1444
  - mcescalante 13y ago
    Super, this is exactly what I was wondering - whether they really had done something brand new or just put it into an app that lets you choose the blur point yourself after.
- theoh 13y ago
  SynthCam is one, free but not open source. You might be interested in the various bits of lightfield rendering code released by Stanford: http://lightfield.stanford.edu/ http://lightfield.stanford.edu/
  - aray 13y ago
    Synthetic aperture-like algorithms are also common at SIGGRAPH if you go through the past two decades. A quick search didn't unearth any, but there is open source software to do parallax depth inferencing, and you could just apply proportional gaussian blur kernels to each depth segment to get a very similar effect.
    - 13y ago
dperfect 13y ago
I believe the algorithm could be improved by applying the blur to certain areas/depths of the image without including pixels from very distant depths, and instead blurring/feathering edges with an alpha channel over those distant (large depth separation) pixels. For example, if you look at the left example photo by Rachel Been[1], the hair is blurred together with the distant tree details. If instead the algorithm detected the large depth separation there and applied the foreground blur edge against an alpha mask, I believe the results would look a lot more natural. [1] http://4.bp.blogspot.com/-bZJNDZGLS_U/U03bQE2VzKI/AAAAAAAAARI/yQwRovcDWRQ/s1600/image3.png http://4.bp.blogspot.com/-bZJNDZGLS_U/U03bQE2VzKI/AAAAAAAAAR...
- gomox 13y ago
  Yes! It shouldn't be too hard to implement either, compared to depth detection.
  - Theodores 13y ago
    Actually I would say it is a hard problem. Look at anything filmed in a studio against a blue/green screen. This is with perfect lighting, a perfectly racked camera, a proper camera tripod/dolly, a perfectly even background screen, the presenter wearing safe colours and anything else I forgot to mention controlled. Now look at the fine hair detail. Does it resolve too well in front of the keyed in <weathermap>? Probably not. You need a very highly skilled operator to get the fine hair detail right. Hollywood can afford the pros, a regional news station in Papua New Guinea maybe not so. I would not say fine hair detail is out of the question, however, we might have to live without it when using this faux depth affect for now.
    - gomox 13y ago
      It doesn't seem to be the same league of problem. Because you are going to blur the stuff in the background anyway, you can be agressive with the masking and the blurring will make it imperceptible. With blue/green screens you are making changes much more severe than just blurring (in terms of the colors, hues and shades involved). Source: Hardly an expert, but I am quite a photography geek, and I've done some work on this type of graphical algorithms in the past (not this type of fancy 3D stuff though).
fidotron 13y ago
Doesn't look totally convincing, but it's good for a first version. The real problem with things like this is the effect became cool by virtue of the fact it needed dedicated equipment. Take that away and the desire people will have to apply the effect will be greatly diminished.
defdac 13y ago
Is this related to the point cloud generation feature modern compositing programs use, like Nuke? Example/tutorial video: http://vimeo.com/61463556 http://vimeo.com/61463556 (skip to 10:27 for magic)
kingnight 13y ago
I'd like to see an example of a evening/night shot using this. I can't imagine the results are anything like the examples here, but would love to be surprised. Are there more samples somewhere?
- avaku 13y ago
  Next they should introduce an infrared laser for measuring depth like in Xbox, then is will totally work in the dark :)
  - spyder 13y ago
    Google Project Tango: https://www.google.com/atap/projecttango/ https://www.google.com/atap/projecttango/
nostromo 13y ago
I sure wish you could buy a DSLR that just plugs into your iPhone. I don't want any of that terrible DSLR software -- just the hardware. I think many devices should become BYOD (bring your own device) soon, including big things like cars. edit: I don't just want my pictures to be saved on my phone. I'd like the phone to have full control of the camera's features -- so I can use apps (like timelapse, hdr, etc.) directly within the camera.
- DocG 13y ago
  There is something similar like this available already! Although, I use it on android. DSLR Controller(BETA) with OTG supported Android phone and almost all of the digital DSLR cameras. I use it with my 600D to take timelapse (it does not have it built in). You can use it with tablet to have bigger screen for example. And it supports almost every setting. EDIT: Free "does it work with your device" version: https://play.google.com/store/apps/details?id=us.zig.dslr https://play.google.com/store/apps/details?id=us.zig.dslr Full version: https://play.google.com/store/apps/details?id=eu.chainfire.dslrcontroller https://play.google.com/store/apps/details?id=eu.chainfire.d...
  - EricaJoy 13y ago
    You might want to have a look at DSLRDashboard (https://play.google.com/store/apps/details?id=com.dslr.dashboard https://play.google.com/store/apps/details?id=com.dslr.dashb...) which is free, frequently updated, has high reviews, works with any PTP compatible camera, and is also open source.
- aray 13y ago
  They have large-aperture digital cameras that are similar already: http://store.sony.com/smartphone-attachable-lens-style-camera-zid27-DSCQX100/B/cat-27-catid-All-Cyber-shot-Cameras http://store.sony.com/smartphone-attachable-lens-style-camer... I'm interested why you'd want DSLR, though, because if it attaches to my phone i'd probably be happy to use the phone screen as the viewfinder and save the depth and weight that would otherwise go to a moving mirror assembly.
techaddict009 13y ago
Just installed it. Frankly speaking I loved the new app!
avaku 13y ago
So glad I did the Coursera course on Probabilistic Graphical Models, so I totally have an understanding of how this is done when they mention Markov Random Field...
salimmadjd 13y ago
Is the app taking more than one photo? It wasn't clear in the blog post. AFAIU to have any depth perception you need to take more than one photo. Calculate the pupil distance (the distance the phone moved) then match image features between the two or more images. Calculate the amount of movement between the matching features to then calculate the depth. As described you then map the depth into an alpha transparency and then apply the blurred image with various blur strength over the original image. Since you're able to apply the blur after the image, it would mean the google camera always takes more than one photo. Also a Cool feature would be to animate the transition from no blur to DOF blur as a short clip or use the depth perception to apply different effect than just blur, like selective coloring, or other filters.
- mikeleung 13y ago
  Yes, the app requires you to move the camera after your initial shot which takes another picture to calculate depth
  - kordless 13y ago
    Using the app it feels like it knows how much you are moving it, which would indicate it's not taking more 'pictures' but actually building a depth map for the first image it takes.
- slaven 13y ago
  Yes - you have to move the camera upwards as it takes a series of photos. It is not working from a single photo.
  - JimmaDaRustla 13y ago
    I just used it, it seems to use the movement to calculate the depth, but the initial image is not blurred or mutated in any way other than how it has calculated the depth.
- rasz_pl 13y ago
  >AFAIU to have any depth perception you need to take more than one photo no you dont, its very tricky, but doable with one: http://www.cs.cornell.edu/~asaxena/learningdepth/ http://www.cs.cornell.edu/~asaxena/learningdepth/ http://www.cs.cornell.edu/~asaxena/reconstruction3d/ http://www.cs.cornell.edu/~asaxena/reconstruction3d/
DangerousPie 13y ago
Isn't this just a copy of Nokia's Refocus? https://refocus.nokia.com/ https://refocus.nokia.com/ edit - better link: http://www.engadget.com/2014/03/14/nokia-refocus-camera-app-lumia-update/ http://www.engadget.com/2014/03/14/nokia-refocus-camera-app-...
- deleted 13y ago
  [deleted]
- piyush_soni 13y ago
  Wow. I didn't know about this.
- timbre 13y ago
  The method at least is very different. The Google app is doing structure from motion, which essentially uses parallax to get the 3D shape of the scene. From there, you can blur/deblur according to depth. The Nokia app uses focal sweep, i.e. it just takes lots of pictures of the same scene focussed at different depths. I'm not sure what the pros and cons of each approach are.
  - Orangeair 13y ago
    The Nokia app requires a longer exposure time, encompassing multiple shots, which basically means that anything moving is a no-go. The quality of its pictures is theoretically superior, though, because the depth effect comes from the photos themselves, and not a simulation. Google's version, on the other hand, works with only a single photo, so it's more versatile. It may suffer issues with quality, though (arising from the difficulties in accurately extracting depth values from a 2D image). That has yet to be seen.
    - zmmmmm 13y ago
      > Google's version, on the other hand, works with only a single photo, so it's more versatile. This doesn't seem to be what everyone else is saying. Most people here are saying it takes multiple shots as you move the camera.
seba_dos1 13y ago
Looks exactly like "shallow" mode of BlessN900 app for Nokia N900 from few years ago. It's funny to see how most of the "innovations" in mobile world presented today either by Apple or Google was already implemented on open or semi-open platforms like Openmoko or Maemo few years before. Most of them only as experiments, granted, but still shows what the community is capable of on its own when not putting unnecessary restrictions on it.
kbrower 13y ago
I did a quick comparison of a full frame slr vs moto x with this lens blur effect. I tried to match the blur amount, but made no other adjustments. Work really well compared to everything else I have seen! http://onionpants.s3.amazonaws.com/IMG_0455.jpg http://onionpants.s3.amazonaws.com/IMG_0455.jpg
- piyush_soni 13y ago
  Humm ... Looks only 'ok-ish' to me ... Though I've done some experiments (no DSLR here) just with people, and it looks good there.
servowire 13y ago
I'm no photographer, but I was tought this was called bokeh not blur. Blur is more because of motion during open shutter.
- yellow 13y ago
  "Bokeh" is specific to photo characteristics. "blur" is more general but still applies: "a thing that cannot be seen clearly".
- randyrand 13y ago
  I think it's more accurate to call bokeh a type of blur. But I still agree with you in that the name Google Lens Bokeh is more accurate, but it sounds worse so...
- sib 13y ago
  This is actually not bokeh at all, it is just simulating being out of focus.
insickness 13y ago
> First, we pick out visual features in the scene and track them over time, across the series of images. Does this mean it needs to take multiple shots for this to work?
- sjtrny 13y ago
  Yes it says so in the previous paragraph
DanielBMarkham 13y ago
Lately I've been watching various TV shows that are using green screen/composite effects. At times, I felt there was some kind of weird DOF thing going on that just didn't look right. Now I know what that is. Computational DOF. Interesting. Along these lines, wasn't there a camera technology that came out last year that allowed total focus/DOF changes post-image-capture? It looked awesome, but IIRC, the tech was going to be several years until released. ADD: Here it is. Would love to see this in stereo 4K: http://en.wikipedia.org/wiki/Lytro http://en.wikipedia.org/wiki/Lytro The nice thing about this tech is that in stereo, you should be able to eliminate the eyeball-focus strain that drives users crazy.
CSDude 13y ago
I wonder what is the exact reason that my country is not included. It is just a fricking camera app.
sneilan 13y ago
Nobody mentioned that today is google's earning date.
mauricesvay 13y ago
The interesting part is not that it can blur a part of the image. The interesting part is that it can generate a depth map automatically from a series of images taken from different points of view, using techniques used in photogrammetry.
nileshtrivedi 13y ago
With these algorithms, will it become feasible to make a driverless car that doesn't need a LIDAR and can run with just a few cameras? Currently, the cost of LIDARs are prohibitive to make (or even experiment with) a DIY self-driving car.
- sjtrny 13y ago
  In the future maybe. Computing the depth maps in this simple case is not very costly because it only requires a relatively sparse amount of key points. To recreate the full scene geometry around a car while moving at 100kmph is a lot harder. Not only do you need to use a massive amount of key points but you need to produce a depth map in millisecond time frames.
- Houshalter 13y ago
  Yes. Machine vision has advanced a lot in recent years and it might just be possible. There is at least one startup trying to make self-driving cars with just machine vision. Computing the differences between several cameras can be a judge of distance, but you can also see how much the object moves as the car moves, and get an estimate based on normal machine vision (how big objects like that normally are, objects nearby it, where it's shadow is, etc.)
  - nileshtrivedi 12y ago
    Do you happen to remember the name of this startup? I have been thinking of experimenting with automated driving. Have started reading up on computer vision and found some encouragement from things like these: https://www.youtube.com/watch?v=dcm9NpMNi68 https://www.youtube.com/watch?v=dcm9NpMNi68
- rjdagost 13y ago
  These algorithms can allow you to have a self-driving car with only cameras. But, there would be a lot of problems if you tried to make a camera-only system for consumer vehicle navigation. Vision systems need distinct "features" in images to find and track across frames to allow you to compute distance, speed, etc. If you don't have many features, pure vision approaches won't work. Nighttime operation is a big problem, as is driving on relatively smooth, featureless terrain. The basic downside is that standard consumer cameras are passive devices. That's why Google uses LIDAR- it's an "active" technology that creates its own features. And driving is an application where the usual computer vision "it works most of the time" is just not good enough. Time of flight cameras are interesting sensors that combines active with passive technology. As this technology matures it might allow for self-driving cars without LIDAR.
anoncow 13y ago
How is Nokia Refocus similar or different to this? It allows refocusing a part of the image which blurs out the rest.(Not a pro) https://refocus.nokia.com/ https://refocus.nokia.com/
- pollen23 13y ago
  Nokia Refocus is quite different. Refocus is a set of photos with different focal lengths, and a look up table. For each "pixel" (The look up table isn't full resolution) the look up table tells which of the photos has the most variance (i.e., is the most focused) at that point, and the viewer simply switches the photo that's shown.
  - anoncow 13y ago
    Thanks!
coin 13y ago
Shallow depth of field is so overused these days. I much prefer having the entire frame in focus, and let me decides what to focus on. I understand the photographer is trying to emphasize certain parts of the photo, but in the end it feels too limiting. It's analogues to mobile "optimized" websites - just give me all the content and I'll choose what I want to look at.
dharma1 13y ago
the accurate depth map creation from 2 photos on a mobile device is impressive. The rest has been done many times before This is cool, but I am waiting more for RAW images exposed in Android camera API. Will be awesome to do some cutting edge tonemapping on 12bits of dynamic range that the sensor gives, which is currently lost.
benmorris 13y ago
The app is fast on my nexus 5. The lense blur feature is really neat. I've taken some pictures this evening and they have turned out great. Overall a nice improvement.
deleted 13y ago
[deleted]
alexnewman 13y ago
Got me beat
Splendor 13y ago
Isn't the real story here that Google is continuing to break off core pieces of AOSP and offer them directly via the Play Store?
matthiasb 13y ago
I don't see this mode. I have a Note 3 from Verizon. Do you?
Lutin 13y ago
This app is now on the Play Store and works with most phones and tablets running Android 4.4 KitKat. Unfortunately it seems to crash on my S3 running CM 11, but your experience may vary. https://play.google.com/store/apps/details?id=com.google.android.GoogleCamera https://play.google.com/store/apps/details?id=com.google.and...
anigbrowl 13y ago
It's interesting that the DoF is calculated in the app. I am wondering if this uses some known coefficients about smartphone cameras to save computation, but in any case I hope this depth mapping becomes available in plugin forms for Photoshop and other users. As an indie filmmaker, it would save a lot of hassle to be able to shoot at infinity focus all the time and apply bokeh afterwards; of course an algorithmic version would likely never get close to what you can achieve with quality optics, but many situations where image quality is 'good enough' for artistic purposes (eg shooting with a video-capable DSLR) then faster is better.
zmmmmm 13y ago
If nothing else, these improvements make HTC's gimmick of adding the extra lens while giving up OIS seem all the more silly.
jheriko 13y ago
This sounds clever but also massively complex for what it does. I don't have anything finished but I can think of a few approaches to this without needing to reconstruct 3d things with clever algorithms... still very neat visually if technically underwhelming
spot 13y ago
i just noticed i have the update and i tried it out. wow, first try. amazing: https://plus.google.com/+ScottDraves/posts/W4ozBLTBmKy https://plus.google.com/+ScottDraves/posts/W4ozBLTBmKy
jestinjoy1 13y ago
This is what i got with Moto G Google Camera App http://i.imgur.com/a6AxO4e.jpg http://i.imgur.com/a6AxO4e.jpg
- r00fus 13y ago
  Your pic looks better than the ones on Google's blog.
sivanmz 13y ago
It's a cool gimmick that would be useful for Instagram photos of food. But selfies will still be distorted when taken up close with a wide angle lens. It would be interesting to pair this with Nokia's high megapixel crop-zoom.
apunic 13y ago
Game changer
tdicola 13y ago
Neat effect--I'm definitely interested in trying this app. Would be cool to see them go further and try to turn highlights in the out of focus areas into nice octagons or other shapes caused by the the aperature blades in a real camera.
sytelus 13y ago
Wow.. this is missing the entire point on why lens blur occurs. Lens blur in normal photographs is the price you pay because you want to focus sharply on a subject. The reason photos with blur looks "cool" is not because the blur itself but its because the subject is so sharply focused that its details are order of magnitude better. If you take a random photo, calculate depth map somehow, blur our everything but the subject then you are taking away information from the photo without adding information to the subject. The photos would look "odd" to the trained eyes at best. For casual photograph, it may look slightly cool on small screens like phone because of relatively increased perceived focus on subject but it's fooling eyes of casual person. If they want to really do it (i.e. add more details to subject) then they should use multiple frames to increase resolution of the photograph. There is a lot of research being done on that. Subtracting details from background without adding details to subject is like doing an Instagram. It may be cool to teens but professional photographers know it's a bad taste.
- blt 13y ago
  That is not true. Stopping down the lens aperture increases the depth of field without reducing detail in the focal plane, until the diffraction limit (around f/16 for a 50mm lens on a full frame DSLR).
- alkonaut 13y ago
  > Subtracting details from background without adding details to subject ... Not sure what you mean by this. Blur is only due to focus distances and aperture sizes. Making the depth of field narrower (making the OOF regions more blurry) does not add detail to the areas that are in focus. Usually, it's even the other way around. Example: Say we shoot a portrait of a person at 5m with a forest 30m away in the background, at 3 different apertures: f/1.4, f./11 and f/20 At the largest aperture (f/1.4) the background will be completely out of focus and the face of the subject will have sharpness at "80%" of what my lens/sensor combo can do in terms of resolution. The less-than-excellent subject sharpness is because lenses aren't perfect and using the largest aperture will reveal this. Even if you use an expensive professional lens, it will have it's maximum sharpness at some aperture that is smaller than the largest. What does happen in the shallow DOF shot is that we have a form of perceived sharpness (usually referred to as "pop") which is an effect that is simply due to the fact that the subject is so distinct from the background. At f/11 the subject sharpness is better than at f/4. It is now probably near 100% of the maximum resolution the sensor/lens combination can deliver. The background is significantly more discernible/focused now. If it was a green blur in the f/1.4 shot is now a forest of very slightly blurred trees. At f/20 the subject's sharpness is again less (e.g. 90%), this time due to the physical limitation known as diffraction that occurs for very small apertures compared to the wavelength. This shot has completely focused trees in the background. To put it another way: when you take the f/11 portrait and go to f/1.4 you take away almost ALL of the background information, and SOME of the foreground information, while adding NO new information. The entire shot will be less focused when you do.
ohwp 13y ago
Nice! Since they got a depth map, 3D-scanning can be a next step.
angusb 13y ago
A couple of other really cool depth-map implementations: 1) The Seene app (iOS app store, free), which creates a depth map and a pseudo-3d model of an environment from a "sweep" of images similar to the image acquisition in the article 2) Google Maps Photo Tours feature (available in areas where lots of touristy photos are taken). This does basically the same as the above but using crowdsourced images from the public. IMO the latter is the most impressive depth-mapping feat I've seen: the source images are amateur photography from the general public, so they are randomly oriented (and without any gyroscope orientation data!), and uncalibrated for things like exposure, white balance, etc. Seems pretty amazing that Google have managed to make depth maps from that image set.
thomasfl 13y ago
I wish google camera gets ported to iOS. The best alternative for iOS seems to bee the "Big Lens" app, where you have to manually create a mask to specify the focused area.
marko1985 13y ago
Happy for this "invention" but I would wait for this kind of stuff when smartphones will have all their laser sensors for depth measurment, so this calculations doesn't require a sequnce of taken picture, as the main character could move quickly and deform the final picture or the blur effect. But for static photography or selfies looks amazing.
thenomad 13y ago
So, is there a way to get the depth map out of the image separately for more post-processing? Fake DOF is nice, but there are a lot more fun things you can use a depth map for. For example, it seems like ghetto photogrammetry (turning photographs into 3D objects) wouldn't be too far away.
panrafal 12y ago
I've created a parallax viewer for lens blur photos. It's an open source web app available at http://depthy.stamina.pl/ http://depthy.stamina.pl/ . It lets you extract the depthmap, works on chrome with webgl and looks pretty awesome on some photos. There is quite a few things you can do with this kind of images, so feel free to play around with the source code on github https://github.com/panrafal/depthy https://github.com/panrafal/depthy

644 points by cleverjake 13y ago

spyder 13y ago

But it can be used only on static subjects because it needs series of frames for depth.

bckrasnow 13y ago

Well, the Lytro guys are screwed now. They're selling a $400 camera with this feature as the main selling point.

sjtrny 13y ago
Except you don't have to move the Lytro

guardian5x 13y ago

I guess that is exactly the same as Nokias Refocus that is on the Lumia Phones for quite some time: https://refocus.nokia.com/ https://refocus.nokia.com/

refrigerator 13y ago
Refocus is awesome and I owe many but not as powerful as this at all - Refocus takes like 10 photos at different levels of focus, so you're limited to the amount of bokeh that the small phone sensor can produce, i.e you only get a nice blurred background when you're taking a really close up picture of something. The new Google Camera feature lets you take a portrait at a normal distance, and simulate f1.8, which is a lot better than what Refocus can do.

goatslacker 13y ago

On iOS you can customize your DoF with an app called Big Lens. Normally apps like Instagram and Fotor let you pick one point in the picture or a vertical/horizontal segment and apply focus there while blurring the background. Big Lens is more advanced since it lets you draw with your finger what you'd like to be in focus. They also include various apertures you can set (as low as f/1.8) as well as some filters -- although I personally find the filters to be overdone but others might find them tasteful.

scep12 13y ago

Impressive feat. Took a few snaps on my Nexus 4 and it seems to work really well given a decent scene.

ctingom 13y ago
Agreed. This has been one of my biggest gripes with Instagram's blur feature as well.
jevinskie 13y ago
I tried it out as well. Good results, especially from their first public iteration.

deleted 13y ago

[deleted]

jawns 13y ago

Regarding the technology (achieving shallow depth of field through an algorithm), not Google's specific implementation ... Up until now, a decently shallow depth of field was pretty much only achievable in DSLR cameras (and compacts with sufficiently large sensor sizes, which typically cost as much as a DSLR). You can simulate it in Photoshop, but generally it takes a lot of work and the results aren't great. The "shallow depth of field" effect was one of the primary reasons why I bought a DSLR. (Yeah, yeah, yeah, quality of the lens and sensor are important too.) Being able to achieve a passable blur effect, even if it's imperfect, on a cellphone camera is really pretty awesome, considering the convenience factor. And if you wanted to be able to change the focus after you take the picture, you had to get a Lytro light field camera -- again, as expensive as a DSLR, but with a more limited feature set. Regarding Google's specific implementation ... I've got a Samsung Galaxy S4 Zoom, which hasn't yet gotten the Android 4.4 update, so I can't use the app itself to evaluate the Lens Blur feature, but based on the examples in the blog post, it's pretty good. It's clearly not indistinguishable from optical shallow depth of field, but it's not so bad that it's glaring. That you can adjust the focus after you shoot is icing on the cake, but tremendously delicious icing. The S4 Zoom is a really terrific point-and-shoot that happens to have a phone, so I'm excited to try it out. Even if I can use it in just 50% of the cases where I now lean on my DLSR, it'll save me from having to lug a bulky camera around AND be easier to share over wifi/data.

theoh 13y ago
I don't know for sure but the code may be derived from or related to Marc Levoy's SynthCam: https://sites.google.com/site/marclevoy/ https://sites.google.com/site/marclevoy/ It does similar things, and in fact I could believe it if the Google app was just a dumbed down version of the functionality intended to be usable by a wider audience.
- aray 13y ago
  There have been multiple implementations of this class of algorithm (collectively "synthetic aperture") using different techniques. Edit: below mentions the HTC "double camera" phones, and in fact it's also possible to create synthetic aperture photography with multiple cameras instead of moving a single camera to multiple positions. Then you have the added advantage of not needing to assume a static scene! But a single-camera algorithm is great for rolling out to the common devices most of us have in our pockets. Marc Levoy's SynthCam used a circular wiping motion, and attempted to "paint" the 3d space occupied by your theoretical processed aperture. This Google Camera only requires a single linear move, and processes the rest!

gamesurgeon 13y ago

One of the greatest features is the ability to change your focus point AFTER you shoot. This is huge.

wmf 13y ago
As long as the original photo was sharp.
Panoramix 13y ago
Kind of like an all-software Lytro https://www.lytro.com/ https://www.lytro.com/

jnevelson 13y ago

So Google basically took what Lytro has been using hardware to achieve, and did it entirely in software. Pretty impressive.

ISL 13y ago
The difference is that Lytro actually does it, instead of simulating it. It'd be fun to play around with the software to see in which cases it breaks (perhaps taking a photo of a framed landscape photo with another landscape behind, for example)
- apu 13y ago
  Lytro is also "just" simulating it, but with slightly more/different data. They capture a light field but that doesn't magically give them depth values; they have to estimate them using an optimization, and then render the final image using a very similar algorithm.
  - ISL 13y ago
    Edit: Whoops: I mis-read the article; 'Lens Blur' uses multiple frames. That lets you get a lot more information.
    - ISL 13y ago
      Well, The Lytro will still win on moving subjects where a sequence convolves subject motion with the depth map.
- frozenport 13y ago
  In reality the Lytro has abysmal performance, which questions the value of lightfield technology as a whole. Instead of capturing the 'light field' you may as well capture most of the image. Secondly the loss of the 2nd derivative of the k-vector makes lightfield currently unsuitable for use in a microscope (although they are working on it...). Although there might be better techniques for refocusing, such as division in the spatial frequency domain. see http://www.cnet.com.au/nokia-vs-lytro-the-refocusing-challenge-339346170.htm http://www.cnet.com.au/nokia-vs-lytro-the-refocusing-challen...

frenchman_in_ny 13y ago

Does this pretty much blow Lytro out of the water, and mean that you no longer need dedicated hardware to do this?

gtremper 13y ago
This doesn't capture the whole light field, which is the advantage of a light field camera. Changing the focus is just a fun application if you have access to the whole light field. http://en.wikipedia.org/wiki/Light_field http://en.wikipedia.org/wiki/Light_field
sp332 13y ago
The Lytro images have a lot more data in them (although probably fewer pixels). You can move around slightly in a Lytro photo because it samples light from all directions. Also, Google's version doesn't seem to use the multiple exposures in the final photo. It only uses them to determine which pixels to blur in the first photo, and uses normal gaussian blur instead of simulating a lens.
- avaku 13y ago
  It does simulate the lens, it says so in the article
  - sp332 13y ago
    Oh... well it doesn't do a very good job. The last example has her hair blurred where it should be sharp. That's not very lens-like.
    - avaku 13y ago
      Yeah, in the first example too (hair at the bottom). But it's still much better than Gaussian blur...

bitJericho 13y ago

If you couple this with instagram does it break the cosmological fabric?

themgt 13y ago

Is looking at the examples giving anyone else a headache? It's like the software blur falls into some kind of uncanny valley for reality.

zippergz 13y ago
It doesn't give me a headache, but to my eyes it isn't remotely convincing. There's no way I'd look at any of those example pictures and not realize the blurring was done with postprocessing.
- JTon 13y ago
  I'm a photography layman and I really can't tell it was done with postprocessing. What gives it away for you?
  - bronson 13y ago
    The blur is far too uniform and boring. Actual bokeh has more intrigue to it: lens aberrations, crazy highlights, wonderful patterns, even when the depth of field isn't very extreme. Check out some real examples on google image search: https://www.google.com/search?q=bokeh+examples&espv=210&es_sm=91&tbm=isch&imgil=4yWA5_ormTgB0M%253A%253Bhttps%253A%252F%252Fencrypted-tbn0.gstatic.com%252Fimages%253Fq%253Dtbn%253AANd9GcRLNLx7njG3LMIaGPxX-biipqDXL1X8ayqBMerER-ru4kInUwuR%253B500%253B498%253B5PGfCFJ_2qPyJM%253Bhttp%25253A%25252F%25252Fwww.slrphotographyguide.com%25252Ftips%25252Fbokeh-DOF.shtml&source=iu&usg=__4dcsDgtrRNOwCo3ZfbA8svWIPJM%3D&sa=X&ei=cNtOU8_UGIrn8gHHyYCYDA&ved=0CDEQ9QEwAw#facrc=_&imgdii=_&imgrc=4yWA5_ormTgB0M%253A%3B5PGfCFJ_2qPyJM%3Bhttp%253A%252F%252Fwww.slrphotographyguide.com%252Ftips%252Fimages%252Fcirclular-bokeh.jpg%3Bhttp%253A%252F%252Fwww.slrphotographyguide.com%252Ftips%252Fbokeh-DOF.shtml%3B500%3B498 https://www.google.com/search?q=bokeh+examples&espv=210&es_s... I'm impressed! Looks like Google solved the hard problem. Implementing a more pleasing blur than plain gaussian doesn't sound too hard.
    - justincormack 13y ago
      Especially in large format petzval and similar lenses eg https://www.flickr.com/photos/richard314159/galleries/72157624453174055/ https://www.flickr.com/photos/richard314159/galleries/721576...

Spittie 13y ago

I find it funny that this was one of the "exclusive features" of the HTC One M8 thanks to the double camera, and days after it's release Google is giving the same ability to every Android phones. I'm sure the HTC implementation works better, but this is still impressive.

dannyr 13y ago
Well, only Android 4.4.x phones. But coming to other Android version.
- pjmlp 13y ago
  Where do they state < 4.4 users would ever get it? I only find mentions of 4.4.
kbrower 13y ago
From what I have seen the HTC implementation does not work better. It looks really synthetic. Google is doing a much better job.

grecy 13y ago

We had an interesting discussion about this a few nights ago at a Photojournalism talk. In that field, digital edits are seriously banned, to the point multiple very well known photo journalists have been fired for one little use of the clone tool [1] and other minor edits. It's interesting to think I can throw an f/1.8 lens on my DSLR and take a very shallow depth of field photo, which is OK, even though it's not very representative of what my eyes saw. If I take the photo at f/18 then use an app like the one linked, producing extremely similar results, that's banned. Fascinating what's allowed and what's not. I find even more interesting is the allowance of changing color photos to B/W, or that almost anything that "came straight off the camera" no matter how far it strays from what your eyes saw. [1] http://www.toledoblade.com/frontpage/2007/04/15/A-basic-rule-Newspaper-photos-must-tell-the-truth.html http://www.toledoblade.com/frontpage/2007/04/15/A-basic-rule...

VikingCoder 13y ago
Radiologists are the same... You take a Computed Tomography series of images, doing all kinds of back-projection (duh) and volumetric anisotropic edge-preserving smoothing, and then you apply segmentation and false-color, semi-transparent transfer functions and lighting... And then you apply one more post-processing effect to try to highlight something, and they freak out that you're no longer showing them the raw data! That does not mean what you think it does.
headShrinker 13y ago
Photojournalism is a weird cross over of history/evidence collection and drifting in to art. They fit together like oil and water, for all the reasons you pointed out.
- bowerbird 13y ago
  on the other hand, i know photographers in art/advertising -- working for some of the biggest companies, like apple -- who brag that "not one pixel goes out without being touched". -bowerbird
aray 13y ago
Could you speak to how familiar photojournalists are with the math and transformations that take place inside a digital camera? For example, almost certainly your cameras have dead pixels, which are processed away during the demosaicing stage, but showing them would be a "straight off the camera image" that I doubt any photojournalist would desire. Additionally many scene-wide process steps (like lens shading map estimation) can be changed in post-processing if the cameras automatic algorithms (3a, etc) "decided" wrong.

the_cat_kittles 13y ago

Isn't it interesting how, by diminishing the overall information content of the image by blurring it, it actually communicates more (in some ways, particularly depth) to the viewer?

selmnoo 13y ago
Well, usually it's for a specific purpose (drawing attention to something) or aesthetic that you use a shallow depth of field. There have been artists in the past that tried to go extreme in the other direction, trying to capture as wide DOF as they could. This is a good example: http://gallery.realitydesign.com/dof.jpg http://gallery.realitydesign.com/dof.jpg Both of these are good individually. But what is best depends totally on what exactly you're going for. Stanley Kubrick did very interesting experimentation with this.

sib 13y ago

If only they had not confused shallow depth of field with Bokeh (which is not the shallowness of the depth of field, but, rather, how out-of-focus areas are rendered), this writeup would have been much better. http://en.wikipedia.org/wiki/Bokeh http://en.wikipedia.org/wiki/Bokeh Cool technology, though.

jcampbell1 13y ago
Bokeh = shallow depth of field effects The author isn't the one confused.
- sib 13y ago
  Bokeh is the quality of the rendering of out of focus areas. It's not just making something out of focus (or creating shallow depth of field). 'Bokeh has been defined as "the way the lens renders out-of-focus points of light".'
- chernand 13y ago
  Hi, I am the author of the Lens Blur blog post and the sentence was indeed wrong as a result of multiple edits. Bokeh and shallow depth of field are indeed two different things. By Bokeh we mean that the blur is synthesized using a disk kernel, e.g. as opposed to a Gaussian Blur. The blog is now fixed.
  - josephagoss 13y ago
    Hey, is there any chance we can download the RAW data alongside the JPG file?
    - chernand 13y ago
      You can extract the computed depthmap, all-in-focus image, and focus settings from the XMP data of the jpeg. See https://developers.google.com/depthmap-metadata/ https://developers.google.com/depthmap-metadata/ for the depthmap format.

mcescalante 13y ago

I may be wrong because I don't know much about image based algorithms, but this seems to be a pretty successful new approach to achieving this effect. Are there any other existing "lens blur" or depth of field tricks that phone makers or apps are using? I'd love to see their code open sourced.

matthiasv 13y ago
The novelty I see here, is the combination and integration into an app. Creating blurred images from a photograph and its depth map can be achieved with a GIMP plugin since at least 2008 [1]. Combining a series of photos to create a 3D model is also widely implemented, e.g. in the Photosynth application developed at U Washington/Microsoft which according to Wikipedia exists as a tech preview since 2006. [1] http://registry.gimp.org/node/1444 http://registry.gimp.org/node/1444
- mcescalante 13y ago
  Super, this is exactly what I was wondering - whether they really had done something brand new or just put it into an app that lets you choose the blur point yourself after.
theoh 13y ago
SynthCam is one, free but not open source. You might be interested in the various bits of lightfield rendering code released by Stanford: http://lightfield.stanford.edu/ http://lightfield.stanford.edu/
- aray 13y ago
  Synthetic aperture-like algorithms are also common at SIGGRAPH if you go through the past two decades. A quick search didn't unearth any, but there is open source software to do parallax depth inferencing, and you could just apply proportional gaussian blur kernels to each depth segment to get a very similar effect.
  - 13y ago

dperfect 13y ago

I believe the algorithm could be improved by applying the blur to certain areas/depths of the image without including pixels from very distant depths, and instead blurring/feathering edges with an alpha channel over those distant (large depth separation) pixels. For example, if you look at the left example photo by Rachel Been[1], the hair is blurred together with the distant tree details. If instead the algorithm detected the large depth separation there and applied the foreground blur edge against an alpha mask, I believe the results would look a lot more natural. [1] http://4.bp.blogspot.com/-bZJNDZGLS_U/U03bQE2VzKI/AAAAAAAAARI/yQwRovcDWRQ/s1600/image3.png http://4.bp.blogspot.com/-bZJNDZGLS_U/U03bQE2VzKI/AAAAAAAAAR...

gomox 13y ago
Yes! It shouldn't be too hard to implement either, compared to depth detection.
- Theodores 13y ago
  Actually I would say it is a hard problem. Look at anything filmed in a studio against a blue/green screen. This is with perfect lighting, a perfectly racked camera, a proper camera tripod/dolly, a perfectly even background screen, the presenter wearing safe colours and anything else I forgot to mention controlled. Now look at the fine hair detail. Does it resolve too well in front of the keyed in <weathermap>? Probably not. You need a very highly skilled operator to get the fine hair detail right. Hollywood can afford the pros, a regional news station in Papua New Guinea maybe not so. I would not say fine hair detail is out of the question, however, we might have to live without it when using this faux depth affect for now.
  - gomox 13y ago
    It doesn't seem to be the same league of problem. Because you are going to blur the stuff in the background anyway, you can be agressive with the masking and the blurring will make it imperceptible. With blue/green screens you are making changes much more severe than just blurring (in terms of the colors, hues and shades involved). Source: Hardly an expert, but I am quite a photography geek, and I've done some work on this type of graphical algorithms in the past (not this type of fancy 3D stuff though).

fidotron 13y ago

Doesn't look totally convincing, but it's good for a first version. The real problem with things like this is the effect became cool by virtue of the fact it needed dedicated equipment. Take that away and the desire people will have to apply the effect will be greatly diminished.

defdac 13y ago

Is this related to the point cloud generation feature modern compositing programs use, like Nuke? Example/tutorial video: http://vimeo.com/61463556 http://vimeo.com/61463556 (skip to 10:27 for magic)

kingnight 13y ago

I'd like to see an example of a evening/night shot using this. I can't imagine the results are anything like the examples here, but would love to be surprised. Are there more samples somewhere?

avaku 13y ago
Next they should introduce an infrared laser for measuring depth like in Xbox, then is will totally work in the dark :)
- spyder 13y ago
  Google Project Tango: https://www.google.com/atap/projecttango/ https://www.google.com/atap/projecttango/

nostromo 13y ago

I sure wish you could buy a DSLR that just plugs into your iPhone. I don't want any of that terrible DSLR software -- just the hardware. I think many devices should become BYOD (bring your own device) soon, including big things like cars. edit: I don't just want my pictures to be saved on my phone. I'd like the phone to have full control of the camera's features -- so I can use apps (like timelapse, hdr, etc.) directly within the camera.

DocG 13y ago
There is something similar like this available already! Although, I use it on android. DSLR Controller(BETA) with OTG supported Android phone and almost all of the digital DSLR cameras. I use it with my 600D to take timelapse (it does not have it built in). You can use it with tablet to have bigger screen for example. And it supports almost every setting. EDIT: Free "does it work with your device" version: https://play.google.com/store/apps/details?id=us.zig.dslr https://play.google.com/store/apps/details?id=us.zig.dslr Full version: https://play.google.com/store/apps/details?id=eu.chainfire.dslrcontroller https://play.google.com/store/apps/details?id=eu.chainfire.d...
- EricaJoy 13y ago
  You might want to have a look at DSLRDashboard (https://play.google.com/store/apps/details?id=com.dslr.dashboard https://play.google.com/store/apps/details?id=com.dslr.dashb...) which is free, frequently updated, has high reviews, works with any PTP compatible camera, and is also open source.
aray 13y ago
They have large-aperture digital cameras that are similar already: http://store.sony.com/smartphone-attachable-lens-style-camera-zid27-DSCQX100/B/cat-27-catid-All-Cyber-shot-Cameras http://store.sony.com/smartphone-attachable-lens-style-camer... I'm interested why you'd want DSLR, though, because if it attaches to my phone i'd probably be happy to use the phone screen as the viewfinder and save the depth and weight that would otherwise go to a moving mirror assembly.

techaddict009 13y ago

Just installed it. Frankly speaking I loved the new app!

avaku 13y ago

So glad I did the Coursera course on Probabilistic Graphical Models, so I totally have an understanding of how this is done when they mention Markov Random Field...

salimmadjd 13y ago

Is the app taking more than one photo? It wasn't clear in the blog post. AFAIU to have any depth perception you need to take more than one photo. Calculate the pupil distance (the distance the phone moved) then match image features between the two or more images. Calculate the amount of movement between the matching features to then calculate the depth. As described you then map the depth into an alpha transparency and then apply the blurred image with various blur strength over the original image. Since you're able to apply the blur after the image, it would mean the google camera always takes more than one photo. Also a Cool feature would be to animate the transition from no blur to DOF blur as a short clip or use the depth perception to apply different effect than just blur, like selective coloring, or other filters.

mikeleung 13y ago
Yes, the app requires you to move the camera after your initial shot which takes another picture to calculate depth
- kordless 13y ago
  Using the app it feels like it knows how much you are moving it, which would indicate it's not taking more 'pictures' but actually building a depth map for the first image it takes.
slaven 13y ago
Yes - you have to move the camera upwards as it takes a series of photos. It is not working from a single photo.
- JimmaDaRustla 13y ago
  I just used it, it seems to use the movement to calculate the depth, but the initial image is not blurred or mutated in any way other than how it has calculated the depth.
rasz_pl 13y ago
>AFAIU to have any depth perception you need to take more than one photo no you dont, its very tricky, but doable with one: http://www.cs.cornell.edu/~asaxena/learningdepth/ http://www.cs.cornell.edu/~asaxena/learningdepth/ http://www.cs.cornell.edu/~asaxena/reconstruction3d/ http://www.cs.cornell.edu/~asaxena/reconstruction3d/

DangerousPie 13y ago

Isn't this just a copy of Nokia's Refocus? https://refocus.nokia.com/ https://refocus.nokia.com/ edit - better link: http://www.engadget.com/2014/03/14/nokia-refocus-camera-app-lumia-update/ http://www.engadget.com/2014/03/14/nokia-refocus-camera-app-...

deleted 13y ago
[deleted]
piyush_soni 13y ago
Wow. I didn't know about this.
timbre 13y ago
The method at least is very different. The Google app is doing structure from motion, which essentially uses parallax to get the 3D shape of the scene. From there, you can blur/deblur according to depth. The Nokia app uses focal sweep, i.e. it just takes lots of pictures of the same scene focussed at different depths. I'm not sure what the pros and cons of each approach are.
- Orangeair 13y ago
  The Nokia app requires a longer exposure time, encompassing multiple shots, which basically means that anything moving is a no-go. The quality of its pictures is theoretically superior, though, because the depth effect comes from the photos themselves, and not a simulation. Google's version, on the other hand, works with only a single photo, so it's more versatile. It may suffer issues with quality, though (arising from the difficulties in accurately extracting depth values from a 2D image). That has yet to be seen.
  - zmmmmm 13y ago
    > Google's version, on the other hand, works with only a single photo, so it's more versatile. This doesn't seem to be what everyone else is saying. Most people here are saying it takes multiple shots as you move the camera.

seba_dos1 13y ago

Looks exactly like "shallow" mode of BlessN900 app for Nokia N900 from few years ago. It's funny to see how most of the "innovations" in mobile world presented today either by Apple or Google was already implemented on open or semi-open platforms like Openmoko or Maemo few years before. Most of them only as experiments, granted, but still shows what the community is capable of on its own when not putting unnecessary restrictions on it.

kbrower 13y ago

I did a quick comparison of a full frame slr vs moto x with this lens blur effect. I tried to match the blur amount, but made no other adjustments. Work really well compared to everything else I have seen! http://onionpants.s3.amazonaws.com/IMG_0455.jpg http://onionpants.s3.amazonaws.com/IMG_0455.jpg

piyush_soni 13y ago
Humm ... Looks only 'ok-ish' to me ... Though I've done some experiments (no DSLR here) just with people, and it looks good there.

servowire 13y ago

I'm no photographer, but I was tought this was called bokeh not blur. Blur is more because of motion during open shutter.

yellow 13y ago
"Bokeh" is specific to photo characteristics. "blur" is more general but still applies: "a thing that cannot be seen clearly".
randyrand 13y ago
I think it's more accurate to call bokeh a type of blur. But I still agree with you in that the name Google Lens Bokeh is more accurate, but it sounds worse so...
sib 13y ago
This is actually not bokeh at all, it is just simulating being out of focus.

insickness 13y ago

> First, we pick out visual features in the scene and track them over time, across the series of images. Does this mean it needs to take multiple shots for this to work?

sjtrny 13y ago
Yes it says so in the previous paragraph

DanielBMarkham 13y ago

Lately I've been watching various TV shows that are using green screen/composite effects. At times, I felt there was some kind of weird DOF thing going on that just didn't look right. Now I know what that is. Computational DOF. Interesting. Along these lines, wasn't there a camera technology that came out last year that allowed total focus/DOF changes post-image-capture? It looked awesome, but IIRC, the tech was going to be several years until released. ADD: Here it is. Would love to see this in stereo 4K: http://en.wikipedia.org/wiki/Lytro http://en.wikipedia.org/wiki/Lytro The nice thing about this tech is that in stereo, you should be able to eliminate the eyeball-focus strain that drives users crazy.

CSDude 13y ago

I wonder what is the exact reason that my country is not included. It is just a fricking camera app.

sneilan 13y ago

Nobody mentioned that today is google's earning date.

mauricesvay 13y ago

The interesting part is not that it can blur a part of the image. The interesting part is that it can generate a depth map automatically from a series of images taken from different points of view, using techniques used in photogrammetry.

nileshtrivedi 13y ago

With these algorithms, will it become feasible to make a driverless car that doesn't need a LIDAR and can run with just a few cameras? Currently, the cost of LIDARs are prohibitive to make (or even experiment with) a DIY self-driving car.

sjtrny 13y ago
In the future maybe. Computing the depth maps in this simple case is not very costly because it only requires a relatively sparse amount of key points. To recreate the full scene geometry around a car while moving at 100kmph is a lot harder. Not only do you need to use a massive amount of key points but you need to produce a depth map in millisecond time frames.
Houshalter 13y ago
Yes. Machine vision has advanced a lot in recent years and it might just be possible. There is at least one startup trying to make self-driving cars with just machine vision. Computing the differences between several cameras can be a judge of distance, but you can also see how much the object moves as the car moves, and get an estimate based on normal machine vision (how big objects like that normally are, objects nearby it, where it's shadow is, etc.)
- nileshtrivedi 12y ago
  Do you happen to remember the name of this startup? I have been thinking of experimenting with automated driving. Have started reading up on computer vision and found some encouragement from things like these: https://www.youtube.com/watch?v=dcm9NpMNi68 https://www.youtube.com/watch?v=dcm9NpMNi68
rjdagost 13y ago
These algorithms can allow you to have a self-driving car with only cameras. But, there would be a lot of problems if you tried to make a camera-only system for consumer vehicle navigation. Vision systems need distinct "features" in images to find and track across frames to allow you to compute distance, speed, etc. If you don't have many features, pure vision approaches won't work. Nighttime operation is a big problem, as is driving on relatively smooth, featureless terrain. The basic downside is that standard consumer cameras are passive devices. That's why Google uses LIDAR- it's an "active" technology that creates its own features. And driving is an application where the usual computer vision "it works most of the time" is just not good enough. Time of flight cameras are interesting sensors that combines active with passive technology. As this technology matures it might allow for self-driving cars without LIDAR.

anoncow 13y ago

How is Nokia Refocus similar or different to this? It allows refocusing a part of the image which blurs out the rest.(Not a pro) https://refocus.nokia.com/ https://refocus.nokia.com/

pollen23 13y ago
Nokia Refocus is quite different. Refocus is a set of photos with different focal lengths, and a look up table. For each "pixel" (The look up table isn't full resolution) the look up table tells which of the photos has the most variance (i.e., is the most focused) at that point, and the viewer simply switches the photo that's shown.
- anoncow 13y ago
  Thanks!

coin 13y ago

Shallow depth of field is so overused these days. I much prefer having the entire frame in focus, and let me decides what to focus on. I understand the photographer is trying to emphasize certain parts of the photo, but in the end it feels too limiting. It's analogues to mobile "optimized" websites - just give me all the content and I'll choose what I want to look at.

dharma1 13y ago

the accurate depth map creation from 2 photos on a mobile device is impressive. The rest has been done many times before This is cool, but I am waiting more for RAW images exposed in Android camera API. Will be awesome to do some cutting edge tonemapping on 12bits of dynamic range that the sensor gives, which is currently lost.

benmorris 13y ago

The app is fast on my nexus 5. The lense blur feature is really neat. I've taken some pictures this evening and they have turned out great. Overall a nice improvement.

deleted 13y ago

[deleted]

alexnewman 13y ago

Got me beat

Splendor 13y ago

Isn't the real story here that Google is continuing to break off core pieces of AOSP and offer them directly via the Play Store?

matthiasb 13y ago

I don't see this mode. I have a Note 3 from Verizon. Do you?

Lutin 13y ago

This app is now on the Play Store and works with most phones and tablets running Android 4.4 KitKat. Unfortunately it seems to crash on my S3 running CM 11, but your experience may vary. https://play.google.com/store/apps/details?id=com.google.android.GoogleCamera https://play.google.com/store/apps/details?id=com.google.and...

anigbrowl 13y ago

It's interesting that the DoF is calculated in the app. I am wondering if this uses some known coefficients about smartphone cameras to save computation, but in any case I hope this depth mapping becomes available in plugin forms for Photoshop and other users. As an indie filmmaker, it would save a lot of hassle to be able to shoot at infinity focus all the time and apply bokeh afterwards; of course an algorithmic version would likely never get close to what you can achieve with quality optics, but many situations where image quality is 'good enough' for artistic purposes (eg shooting with a video-capable DSLR) then faster is better.

zmmmmm 13y ago

If nothing else, these improvements make HTC's gimmick of adding the extra lens while giving up OIS seem all the more silly.

jheriko 13y ago

This sounds clever but also massively complex for what it does. I don't have anything finished but I can think of a few approaches to this without needing to reconstruct 3d things with clever algorithms... still very neat visually if technically underwhelming

spot 13y ago

i just noticed i have the update and i tried it out. wow, first try. amazing: https://plus.google.com/+ScottDraves/posts/W4ozBLTBmKy https://plus.google.com/+ScottDraves/posts/W4ozBLTBmKy

jestinjoy1 13y ago

This is what i got with Moto G Google Camera App http://i.imgur.com/a6AxO4e.jpg http://i.imgur.com/a6AxO4e.jpg

r00fus 13y ago
Your pic looks better than the ones on Google's blog.

sivanmz 13y ago

It's a cool gimmick that would be useful for Instagram photos of food. But selfies will still be distorted when taken up close with a wide angle lens. It would be interesting to pair this with Nokia's high megapixel crop-zoom.

apunic 13y ago

Game changer

tdicola 13y ago

Neat effect--I'm definitely interested in trying this app. Would be cool to see them go further and try to turn highlights in the out of focus areas into nice octagons or other shapes caused by the the aperature blades in a real camera.

sytelus 13y ago

Wow.. this is missing the entire point on why lens blur occurs. Lens blur in normal photographs is the price you pay because you want to focus sharply on a subject. The reason photos with blur looks "cool" is not because the blur itself but its because the subject is so sharply focused that its details are order of magnitude better. If you take a random photo, calculate depth map somehow, blur our everything but the subject then you are taking away information from the photo without adding information to the subject. The photos would look "odd" to the trained eyes at best. For casual photograph, it may look slightly cool on small screens like phone because of relatively increased perceived focus on subject but it's fooling eyes of casual person. If they want to really do it (i.e. add more details to subject) then they should use multiple frames to increase resolution of the photograph. There is a lot of research being done on that. Subtracting details from background without adding details to subject is like doing an Instagram. It may be cool to teens but professional photographers know it's a bad taste.

blt 13y ago
That is not true. Stopping down the lens aperture increases the depth of field without reducing detail in the focal plane, until the diffraction limit (around f/16 for a 50mm lens on a full frame DSLR).
alkonaut 13y ago
> Subtracting details from background without adding details to subject ... Not sure what you mean by this. Blur is only due to focus distances and aperture sizes. Making the depth of field narrower (making the OOF regions more blurry) does not add detail to the areas that are in focus. Usually, it's even the other way around. Example: Say we shoot a portrait of a person at 5m with a forest 30m away in the background, at 3 different apertures: f/1.4, f./11 and f/20 At the largest aperture (f/1.4) the background will be completely out of focus and the face of the subject will have sharpness at "80%" of what my lens/sensor combo can do in terms of resolution. The less-than-excellent subject sharpness is because lenses aren't perfect and using the largest aperture will reveal this. Even if you use an expensive professional lens, it will have it's maximum sharpness at some aperture that is smaller than the largest. What does happen in the shallow DOF shot is that we have a form of perceived sharpness (usually referred to as "pop") which is an effect that is simply due to the fact that the subject is so distinct from the background. At f/11 the subject sharpness is better than at f/4. It is now probably near 100% of the maximum resolution the sensor/lens combination can deliver. The background is significantly more discernible/focused now. If it was a green blur in the f/1.4 shot is now a forest of very slightly blurred trees. At f/20 the subject's sharpness is again less (e.g. 90%), this time due to the physical limitation known as diffraction that occurs for very small apertures compared to the wavelength. This shot has completely focused trees in the background. To put it another way: when you take the f/11 portrait and go to f/1.4 you take away almost ALL of the background information, and SOME of the foreground information, while adding NO new information. The entire shot will be less focused when you do.

ohwp 13y ago

Nice! Since they got a depth map, 3D-scanning can be a next step.

angusb 13y ago

A couple of other really cool depth-map implementations: 1) The Seene app (iOS app store, free), which creates a depth map and a pseudo-3d model of an environment from a "sweep" of images similar to the image acquisition in the article 2) Google Maps Photo Tours feature (available in areas where lots of touristy photos are taken). This does basically the same as the above but using crowdsourced images from the public. IMO the latter is the most impressive depth-mapping feat I've seen: the source images are amateur photography from the general public, so they are randomly oriented (and without any gyroscope orientation data!), and uncalibrated for things like exposure, white balance, etc. Seems pretty amazing that Google have managed to make depth maps from that image set.

thomasfl 13y ago

I wish google camera gets ported to iOS. The best alternative for iOS seems to bee the "Big Lens" app, where you have to manually create a mask to specify the focused area.

marko1985 13y ago

Happy for this "invention" but I would wait for this kind of stuff when smartphones will have all their laser sensors for depth measurment, so this calculations doesn't require a sequnce of taken picture, as the main character could move quickly and deform the final picture or the blur effect. But for static photography or selfies looks amazing.

thenomad 13y ago

So, is there a way to get the depth map out of the image separately for more post-processing? Fake DOF is nice, but there are a lot more fun things you can use a depth map for. For example, it seems like ghetto photogrammetry (turning photographs into 3D objects) wouldn't be too far away.

panrafal 12y ago

I've created a parallax viewer for lens blur photos. It's an open source web app available at http://depthy.stamina.pl/ http://depthy.stamina.pl/ . It lets you extract the depthmap, works on chrome with webgl and looks pretty awesome on some photos. There is quite a few things you can do with this kind of images, so feel free to play around with the source code on github https://github.com/panrafal/depthy https://github.com/panrafal/depthy