10 ms·
Parallax effect from Google Lens Blur photos
- bergie 13y agoSource code: https://github.com/panrafal/depthy https://github.com/panrafal/depthy
- splitbrain 13y agoAll I see is a black page? (Chromium 33.0.1750.152) 404s for the script and CSS files
- bergie 13y agoYep, seems he deployed a bad build. Was working a bit earlier, and hopefully soon again Edit: works now http://depthy.stamina.pl/ http://depthy.stamina.pl/
- roeme 13y agoSurprisingly well faked in face of the fact that there's no real 3D information in the pictures. There simply cannot be new 'pixels' appearing that were (i.e. before applying the effect) hidden from objects in the cameras line-of-sight. This is evident upon closer inspection (look at edges around the approximate middle of the DOF). The trompe-l'œil then falls a bit apart. Interestingly, I did not notice above with the IOS7 background parallax; I'm wondering why? Special images, or a stricter constraint on movement?
- bergie 13y agoYeah, it appears to be a regular JPG with a depth map added to the metadata. Originally Google uses this for refocusing, but the parallax is a neat trick
- crazygringo 13y agoIndeed -- there are a bunch of spots where, if you look closely, you can see the foreground texture being "repeated" in the background texture, when you've moved your cursor to the edge. I wonder if a better kind of pixel-filling-in algorithm could make this more believable, or if the problem is with the inherent inaccuracies of the depth map. But still, as a whole it's remarkable effective. I'd like to be able to appreciate it without moving my mouse -- I wonder what kind of camera movement function, at what speed, would make for the most unobtrusive effect that would still give the perception of depth?
- ctdonath 13y agoI'd like to be able to appreciate it without moving my mouse Oculus Rift? Not just different left/right angles for each eye, but also can rotate the angles by tilting head. That would be a spectacular way to view still photos in VR.
- kordless 13y agoGiven it only takes panning the image slightly and adjusting the parallax a bit, I think that's probably doable. It would be the modern day version of this: http://en.wikipedia.org/wiki/Stereoscopy http://en.wikipedia.org/wiki/Stereoscopy
- hnha 13y agoFrom my understanding there actually is 3D information reconstructed via sfm from the hand jittering. Only few points and of questionable quality but enough for the foreground extraction (for blurring).
- panrafal 13y agoIt's exactly that - single image, and a depth map calculated from a series of shots made upwards. Both these are bundled in a fake-bokeh image. There is obviously no more pixels. However... If you choose your subject wisely, the effect can be pretty believeable, using a simple displacement map. On iOS though it's something different. There's no depth map in there, just a flat image moving counterwise to your hand movements.
- roeme 13y agoNow that you explained the IOS7 trick, it's obvious. (Given the environment: background filling image on a physical device movement as opposed to a image on a website).
- panrafal 13y agoOn iOS you have the icons as a first plane - that makes the trick. If you have an Android device around, try depthy on it - it's way better this way imo.
- spyder 13y agoThere is more pixels captured compared to a single shot image as you said: "series of shots made upwards". So it captures some pixels that is hidden when the camera moves upwards, but if the simulated parallax is bigger than the original camera movement then there will be still missing pixels. Probably this could be improved by doing bigger movements with the camera, like with other 3D reconstruction software.
- baddox 13y agoIt sounds like there are two issues here. One is how the depth map is generated, and the other is how the resulting image file is formatted. For the former, several still images are collected while the camera is moving, which provides parallax which can be used to generate the depth map. For the latter, I don't know, but it would certainly be possible to bundle both the depth map and multiple "layers" of photograph that could be used to actually expose new pixels in the background when the foreground moves.
- berkut 13y agoYeah. Also, as the depth map only has a single depth value per pixel, you can get aliasing around the edges of the objects in focus as they have a halo around them.
- nerfhammer 13y agoPeople in the machine learning community have been working on this sort of problem for awhile, the results are often very impressive: http://make3d.cs.cornell.edu/ http://make3d.cs.cornell.edu/
- kcbanner 13y agoThere actually is 3D information in the form of a depth map including in the images, put their by the Google Camera app.
- arscan 13y agoIt might be helpful to separate the code that creates the parallax effect from the angular app that displays the website & examples. This is a really cool trick and it would be nice to be able to easily reuse it. For those looking, it seems like the "magic" happens in a couple of spots (I think): https://github.com/panrafal/depthy/blob/master/app/scripts/pixi/DepthFilter.js https://github.com/panrafal/depthy/blob/master/app/scripts/p... and in https://github.com/panrafal/depthy/blob/master/app/scripts/controllers/viewer.js https://github.com/panrafal/depthy/blob/master/app/scripts/c...
- panrafal 13y agoThere already is a library for extracting the depth map here: https://github.com/spite/android-lens-blur-depth-extractor https://github.com/spite/android-lens-blur-depth-extractor (internet rocks, isn't it ;) ) As for the Shader - it's just a few lines of actual code. Depthy is a quick weekend hack. There's a loong way before it.
- aantix 13y agoHas anyone managed to replicate the lens blur effect that's being utilized in the new Android camera app? Or at least know what research paper it's based on?
- panrafal 13y agoYes, here: http://jabtunes.com/labs/lensblur/index3.html http://jabtunes.com/labs/lensblur/index3.html and here: http://quasimondo.com/QuasimondoLibsJS/demos/DepthMapLensBlur.html http://quasimondo.com/QuasimondoLibsJS/demos/DepthMapLensBlu...
- comex 13y agoThat seems to just be using the depth map information stored by the app, not replicating the effect from scratch.
- andybak 13y agoNice work! Are you doing any 'content-aware fill' style magic on the invented occluded pixels? I wonder what else can be done with the depth data... Fog is the obvious one. You could potentially insert elements into the scene that knew when they were behind foreground items. I'm sure some kind of point-cloud or mesh could also be derived but not sure how good it would be. Funnily enough I nearly posted to /r/android/ earlier "I wish you could save the generated z-buffer" - it didn't occur to me to actually look!
- panrafal 13y agoFog, refocus, background separation, chroma shifting. As for the fill, if the depthmap is of good quality and without sharp edges - there is no need for that, unless you go berzerk with the scale of displacement.
- BHSPitMonkey 13y agoIf you're interested in Google's metadata spec for what the camera actually captures: https://developers.google.com/depthmap-metadata/ https://developers.google.com/depthmap-metadata/
- aleyan 13y agoWell done. I have been working with animating 2d images with a hand constructed depth map in the past to a decent 3d effect[1]. The illusion when guided by your mouse as in the OP seems to be much stronger then in left-right strafing that I did. One point that can be worked on, as pointed out by other people here is the occluded pixels (this is an issue when an image has sharp changes in distance from the camera). You simply can't show what wasn't seen by the camera, so you have to fake it. In the pre-baked animations I did, the occluded pixels were duplicated and then manually fixed by me in Photoshop. Perhaps the solution here would be to capture and store images from several perspectives. These could then be used to generate the depthmap using what algorithms Lens Blur used, and also to interpolate the occluded pixels when viewing the photo from different perspectives. [1] http://fooladder.tumblr.com/ http://fooladder.tumblr.com/
- kordless 13y agoI can't seem to get the top view when I upload an image. The image and map display correctly below. I'll try downloading it and running it myself.
- rellik 13y agoSame here (in firefox). Tried in Chrome and it works fine.
- ctdonath 13y agoInteresting contrast, the next-gen Lytro camera featuring such refocusing & perspective shift: https://preorder.lytro.com/lytro-illum-pre-order https://preorder.lytro.com/lytro-illum-pre-order
- aaronetz 13y agoRelated (works from a single image): http://make3d.cs.cornell.edu/ http://make3d.cs.cornell.edu/
- redthrowaway 13y agoI'm putting the over/under on how long till this is used for porn at 4.5 days. Any takers?
- FlailFast 13y agoWell, it's been a while since my Internet video days, but hell, I love 3d parallax effects. When are we shooting?
- josefresco 13y agoI had the same thought... how this effect could enhance porn picture sites. Given the dominance of "tube" porn sites, I doubt it will be as big as it could have been when picture sites dominated. However I could still see some niche sites using it to "get a better look" or on-page to attract clicks.
- MrScruff 13y agoAs I understand it, the depth map is created by slowly moving the camera while taking images of the same objects. It probably wouldn't work as well with a moving subject (even small movements).
- shawabawa3 13y ago"slowly" is < 0.5seconds. Would work fine on models
- judk 13y agoAndroid Chrome gets "your browser does not support WebGL. You'll not see the parallax effect... Try using Chrome on your desktop or Android device"
- Tomdarkness 13y agoWorks fine in Chrome on my Nexus 5.
- hahainternet 13y agoTry the Chrome Beta. Works on mine.
- pornel 13y agoWorks in Firefox for Android.
- dragonwriter 13y agoWorks fine on Android Chrome on my Note 3.
- ChrisClark 13y agoWorks fine in my Chrome, not beta, version 34.0.1847.114.
- bostonpete 13y agoAny way to save the hypnotize view to an animated gif? That'd be really cool...
- than 13y agoDamn this is cool. Even though there's some smearing to make up for the lack of information, especially behind the railings on the deck, the effect is striking, and subtle enough to be believable at a casual glance.
- tst 13y agoI would be interested to see how this effect affects the conversion rate. I could imagine that it could work very well on physical products.
- juanpdelat 13y agoHow is the parallax effect working without using my mouse and shaking my laptop? Am I going crazy or do Mac Books Pro have accelerometers? Edit: Had to google it, they have something called Sudden Motion Sensor http://support.apple.com/kb/HT1935 http://support.apple.com/kb/HT1935
- deleted 13y ago[deleted]
- jameshart 13y agoInteresting that on the shelf image, it's the slightly diffuse reflections of the items on the shelf in the wood grain that breaks the illusion for me.
- hadem 13y ago"Try using Chrome on your desktop or Android device"... I am...
- panrafal 13y agoI will add <span ng-if="Modernizr.Android">another </span>Android ;)
- general_failure 13y agoVery nice. Is there a writeup on how this works? It appears the shadow on the pole in the second picture moves! (maybe it's just some gimmick).
- ahassan 13y agoIt would be cool if this was added to the iOS parallax effect.
- Aardwolf 13y agoThis requires a depth map in the image. Would it also work on any image by calculating a depthmap from the blurriness?
- wtmcc 13y agoNo, because “blurriness” (low local contrast) may indicate something besides a particular depth. Consider, for instance, a head-on photograph of a print of a shallow-focused photo. The region that print embodies will have plenty of variation in contrast, but exist at a single depth. Also, consider that blurring increases in both in front of and behind the center of focus; how could we tell which depth the blurring indicated? Something similar to what you suggest is, however, done in software autofocus, which can take repeated samples at different focal distances to clear things up. Maybe that’s something to think about, e.g. for a static subject. http://en.wikipedia.org/wiki/Autofocus#Contrast_detection http://en.wikipedia.org/wiki/Autofocus#Contrast_detection
- darkmighty 13y agoYup, there's no simple way to recover depth from blur: there will be featureless regions where you can't tell if there was any blur -- in other words, there is no universal way to tell if a region has gone through a low-pass filter or if it is natively low frequency. Would heuristics work well? I can think of a handful, but none really good.
- prawn 13y agoThis could be an effective differentiator for a real estate web site featuring property photos. Or even an online store where product photos are done in this way.