5 ms·
Other commenters here are correct that the LIDAR is too low-resolution to be used as the primary source for the depth maps. In fact, iPhones use four-ish method
by Uncorrelated 1y ago
Other commenters here are correct that the LIDAR is too low-resolution to be used as the primary source for the depth maps. In fact, iPhones use four-ish methods, that I know of, to capture depth data, depending on the model and camera used. Traditionally these depth maps were only captured for Portrait photos, but apparently recent iPhones capture them for standard photos as well.
1. The original method uses two cameras on the back, taking a picture from both simultaneously and using parallax to construct a depth map, similar to human vision. This was introduced on the iPhone 7 Plus, the first iPhone with two rear cameras (a 1x main camera and 2x telephoto camera.) Since the depth map depends on comparing the two images, it will naturally be limited to the field of view of the narrower lens.
2. A second method was later used on iPhone XR, which has only a single rear camera, using focus pixels on the sensor to roughly gauge depth. The raw result is low-res and imprecise, so it's refined using machine learning. See: https://www.lux.camera/iphone-xr-a-deep-dive-into-depth/ https://www.lux.camera/iphone-xr-a-deep-dive-into-depth/
3. An extension of this method was used on an iPhone SE that didn't even have focus pixels, producing depth maps purely based on machine learning. As you would expect, such depth maps have the least correlation to reality, and the system could be fooled by taking a picture of a picture. See: https://www.lux.camera/iphone-se-the-one-eyed-king/ https://www.lux.camera/iphone-se-the-one-eyed-king/
4. The fourth method is used for selfies on iPhones with FaceID; it uses the TrueDepth camera's 3D scanning to produce a depth map. You can see this with the selfie in the article; it has a noticeably fuzzier and low-res look.
You can also see some other auxiliary images in the article, which use white to indicate the human subject, glasses, hair, and skin. Apple calls these portrait effects mattes and they are produced using machine learning.
I made an app that used the depth maps and portrait effects mattes from Portraits for some creative filters. It was pretty fun, but it's no longer available. There are a lot of novel artistic possibilities for depth maps.
- snowdrop 1y agoFor method 3 that article is 5 years old, see: https://github.com/apple/ml-depth-pro?tab=readme-ov-file https://github.com/apple/ml-depth-pro?tab=readme-ov-file
- heliographe 1y ago> but apparently recent iPhones capture them for standard photos as well. Yes, they will capture them from the main photo mode if there’s a subject (human or pet) in the scene. > I made an app that used the depth maps and portrait effects mattes from Portraits for some creative filters. It was pretty fun, but it's no longer available What was your app called? Is there any video of it available anywhere? Would be curious to see it! I also made a little tool, Matte Viewer, as part of my photo tool series - but it’s just for viewing/exporting them, no effects bundled: https://apps.apple.com/us/app/matte-viewer/id6476831058 https://apps.apple.com/us/app/matte-viewer/id6476831058
- lxgr 1y ago> Yes, they will capture them from the main photo mode if there’s a subject (human or pet) in the scene. One of the example pictures on TFA is a plant. Given that, are you sure iOS is still only taking depth maps for photos that get the "portrait" icon in the gallery? (Or have they maybe expanded the types of possible portrait subjects?)
- heliographe 1y agoIt will capture the depth map and generate the semantic mattes (except in some edge cases) no matter the subject if you explicitly set the camera in Portrait mode, which is how I would guess the plant photo from the article was captured. My previous comment was about the default Photo mode. If you have a recent iPhone (iPhone 15 or above iirc) try it yourself - taking a photo of a regular object in the standard Photo mode won’t yield a depth map, but one of a person or pet will. Any photo taken from the Portrait mode will yield a depth map. You can find out more about this feature by googling “iPhone auto portrait mode”. Apple’s documentation is less helpful with the terminology; they call it “Apply the portrait effect to photos taken in Photo mode” https://support.apple.com/guide/iphone/edit-portrait-mode-photos-iph310a9a220/18.0/ios/18.0#ipha7919eb64 https://support.apple.com/guide/iphone/edit-portrait-mode-ph...
- lxgr 1y ago> [...] if you explicitly set the camera in Portrait mode, which is how I would guess the plant photo from the article was captured. Ah, that makes sense, thank you!
- tgma 1y agoCan method 4 be used by a security application to do liveness detection?
- jjcob 1y agoThat's what FaceID does.
- tgma 1y agoObviously -- that is not my question though. I was curious if that data is exposed via API or within the image for front camera as well, so a third party app can do it.
- Someone 1y agoI think so. https://developer.apple.com/documentation/avfoundation/capturing-depth-using-the-lidar-camera https://developer.apple.com/documentation/avfoundation/captu..., https://developer.apple.com/documentation/avfoundation/capturing-photos-with-depth https://developer.apple.com/documentation/avfoundation/captu... show how to get a depth map.
- oxym0ron 1y agoOne thing worth noting: LIDAR is primarily optimized for fast AF and low-light focusing rather than generating full-res depth maps.
- int_19h 1y agohttps://lookingglassfactory.com https://lookingglassfactory.com is a holographic image frame that can show those iPhone photos with depth maps in them in actual 3D.