4 ms·

To be honest, I wouldn't be surprised if Apple isn't already doing this and just doesn't say so explicitly. There is a crazy amount of image processing going on

by jrflo 2mo ago

To be honest, I wouldn't be surprised if Apple isn't already doing this and just doesn't say so explicitly. There is a crazy amount of image processing going on behind the scenes in each smartphone photo. Would be interesting to compare a photo taken with the native camera app and photosynthesis to compare. I'm assuming your demo images on your site are the raw and processed images from your app that don't go through Apple's "secret sauce" processing pipeline.

selectodude 2mo ago
I think this is just Deep Fusion.
- deleted 2mo ago
  [deleted]
walrus01 2mo ago
Phones have been doing this for a LONG time, if you had a oneplus 7 pro (released may 2019) and took it into an extremely dark area, like where your eyes can't see anything at all, and take a 'night mode' photo, it asks you to hold it steady for about 4-5 seconds and is clearly taking a series of photos with long exposure time. Then it stacks them and uses data from the sensor of hand movement jitter to deblur. The result when done outdoors (like, standing in a dark forest at midnight) will be a photo where you can actually see what's around you, and details show up as if it was a brightly full moon lit night.
- somehnguy 2mo ago
  I think it's interesting that the same functionality appeared on iPhone in 2019 as well. Google tells me it was the iPhone 11 and iOS 13.
  - marmarama 2mo ago
    Night Sight appeared on the Google Pixel camera app in 2018. I think Nokia was doing it first on phones before Microsoft destroyed the remains of their phone business. The reborn Nokia/HMD was also doing multi-sensor image stacking back in 2019 on the Nokia 9 Pureview.
    - throwaway219450 2mo ago
      The related paper (2019): https://google.github.io/night-sight/ https://google.github.io/night-sight/ Google has some great blogs about their camera technology https://research.google/blog/night-sight-seeing-in-the-dark-on-pixel-phones/ https://research.google/blog/night-sight-seeing-in-the-dark-...
- hbn 2mo ago
  That's a bunch of photos from the same camera, not 2 lenses at the same time
  - walrus01 2mo ago
    I know, I just don't think it's a huge jump from "take multiple photos over a 2s period from one camera and stack them" to "take several photos from discrete physical cameras and stack them". As a subscription of paid app I am not the target market for the person who made this, I agree with another person commenting in here that said they should be working for a phone manufacturer.
    - omneity 2mo ago
      I’d argue the opposite. The geometry involved in reconciling photos from two separate lenses with different focal lengths is quite challenging. If you pay close attention you will see some differences in the before/after samples in the site (for example the ear in that kitchen selfie). The math is surely more complex than aligning same-lens photography even when you consider the change in angle and perspective between two shots from the same lens.
      - account42 2mo ago
        Not really. Whether you have two different lenses or the same lenses with some offset from hand movement you need to accurately model the lens(es). Once you can do that for one lens you can just as easily do it for two - it's just more variables to optimize, not a fundamentally harder problem. Panorama software has supported images from different lenses since forever.
- LoganDark 2mo ago
  There is an iOS app that does this called Spectre: https://spectre.cam https://spectre.cam From the authors of Halide.
  - ShinyLeftPad 2mo ago
    It's different from just stacking exposures, it's a paid app that does progressive animation of exposures for creative effect and says it's "AI-powered" whatever that means.
    - LoganDark 2mo ago
      It's a paid app that I literally have, and all it does is take multiple photos and composite them together. It can either naïvely average them together in-place like a traditional long exposure, or align them using machine learning. I'm actually appalled they use the term "AI" considering who they are but that doesn't matter, it's ML. I don't use any of the animation crap, but the aligned long exposures can be useful for ultrawide takes since that camera is so noisy. Edit: actually I can't seem to find the setting that aligns them. I seem to remember it being able to do that but maybe it doesn't... which would be disappointing...
      - walrus01 2mo ago
        People are slapping "AI" on anything these days to try to sell more units of whatever it is.
      - sandofsky 2mo ago
        > I'm actually appalled they use the term "AI" considering who they are but that doesn't matter, it's ML. I agree, and I built it! The app launched in 2019. Ideally I would have said ML, but it felt like AI had more recognition among non-technical users. If I launched it in 2026, it would 100% say "ML" not only because it's more technically accurate, but to make it abundantly clear it doesn't generate slop.
      - ShinyLeftPad 2mo ago
        I have it too but after like 3 times I never used it again, felt like a cheap gimmick. I regret falling for hype. If anything they do good marketing
- xyzzy_plugh 2mo ago
  The Amazon Fire Phone did this in 2014. I'm not aware of any prior art in mainstream devices aside from gimmicks like 3D (stereoscopic) photos. I can't remember if the Fire Phone had any relevant gimmicks but it certainly used multiple lenses to produce a single image.
  - walrus01 2mo ago
    UAV sensor people made a thing that uses 300+ phone size sensors in a single pod, and software workflow to fuse the output together. It's been around quite a few years now. https://www.google.com/search?client=firefox-b-d&q=ARGUS+photo+pod+uav https://www.google.com/search?client=firefox-b-d&q=ARGUS+pho...
    - xyzzy_plugh 2mo ago
      Ah yes, how could I forget the ARGUS-IS, a household staple of the early 21st century. Those were the days...
      - walrus01 2mo ago
        I keep an extra one next to my panini press, the image quality from the whiteboard notes area on my fridge has never been sharper!
stavros 2mo ago
This should be easy to check. In these photos, the center of the image has higher detail than the edges.
- kndidneid 2mo ago
  Maybe on this guy’s app, but not the way Apple does it.
  - account42 2mo ago
    If that's the case then it would disprove the original assertion that Apple already does the same thing. You can't have the same high detail at the edges without making shit up because those parts of the image are out of frame for the higher zoom lens.
sajomes 2mo ago
Yes exactly, the site shows photos all taken from within the app. I have done A/B tests against photos taken in Camera.app and the Photosynthesis composite does significantly beat the detail central area covered by the second lens in Photosynthesis. Because of the limitations of capturing multiple lenses at once through isVirtualDeviceConstituentPhotoDeliveryEnabled, there is a second capture mode in the app for 'Quality' which takes a simultaneous capture then takes sequential shots from each lens which do benefit from Deep Fusion, then uses those for the stitch (as long as the Deep Fusion images outperform the simultaneous capture results). That's also what I do for our Night mode.
101008 2mo ago
Hijacking this comment because of this line that reminded me of something: "There is a crazy amount of image processing going on behind the scenes in each smartphone" I bought the Pixel 10 because it said it had great zooming capabilities. After 5X (that means 10X), it's completely stupid. Basically it takes a very blurred photo and then tries to reconstruct it with AI. The final results are completely awful and really distant from reality. Any advice in how to improve that?
- RamblingCTO 2mo ago
  Wasn't that the same manufacturer that got caught with the moon AI stuff? Lol
  - dgacmu 2mo ago
    The was Samsung, iirc: https://www.reddit.com/r/Android/comments/11nzrb0/samsung_space_zoom_moon_shots_are_fake_and_here/ https://www.reddit.com/r/Android/comments/11nzrb0/samsung_sp...
    - RamblingCTO 2mo ago
      Gotcha, thanks! Funny that they didn't learn from that one
  - viciousvoxel 2mo ago
    That was samsung
- dgacmu 2mo ago
  In photos after you can toggle between the AI modified and original. The original isn't as blurry as the preview shown - there's still a lot of image stacking happening and it's often very decent in my experience. Edited to add: I have a pixel 10 pro which has a better zoom lens, so I could be having a different experience than you...
- nemomarx 2mo ago
  You can use third party camera apps - https://f-droid.org/en/packages/net.sourceforge.opencamera/ https://f-droid.org/en/packages/net.sourceforge.opencamera/ is a decent one. I know there's a few professional ish options too with dslr ish focus control.
00kevn 2mo ago
One time I took a photo of a snake in the road with my iphone in low light. When I looked at the photo afterwards the snake was removed by the AI enhancement
- callc 2mo ago
  Similar story but my low-light snake photo turned out much less “scary” than in real life. I’m guessing it has to do with how intensely focused / scared my brain was with the little guy
kndidneid 2mo ago
It is part of Apple’s “secret sauce”, yes.
- jrflo 2mo ago
  Thank you for confirming my suspicions, lol
Modified3019 2mo ago
I use iPhones (paid for by work) as a tool for my agronomy job. I take a large amount of photos throughout the day to help document what I find in the field. Since at least the iPhone 12, holding the phone still and letting scene sit for a few seconds will result in a sharper photo. I’m not sure if it’s a subtle version focus stacking or a sharpening algorithm, but it’s definitely there. On my 16e, immediately looking at a photo just taken can result in the image noticeably sharpening after a second or 2, so this may be a post processing step as well.
- Gigachad 2mo ago
  From what I can tell, it's taking multiple photos and stacking them in post. So giving the phone more time to capture sharp exposures to stack would do a lot to help.
- KeplerBoy 2mo ago
  It's probably a version of this: https://dl.acm.org/doi/abs/10.1145/3306346.3323024 https://dl.acm.org/doi/abs/10.1145/3306346.3323024 Subtle movement between shots can be used to create an image sharper than any of the inputs.
- Cthulhu_ 2mo ago
  I found this as I was taking a picture of a hare after dark without flash, the image that came up was both relatively sharp and much more illuminated / clear than my own eyes; it can't just have been long exposure time (the mechanical stabilization in phone cameras can't be that good), so I'm confident it takes many photos and layers them smartly, not unlike astrophotography.
- HumblyTossed 2mo ago
  For some time how, when you have the camera open and lining up a shot, iOS is taking image after image. When you press the shutter, algorithms determine the best or sharpest shot and processes it further. Phil introduced it with iPhone 11 as "deep fusion"
Melatonic 2mo ago
Don't they do this with the "macro" setting already ?
dbbk 2mo ago
But they do do this? They talk about it? It's not a secret
numpad0 2mo ago
Reading title and this ^ comment before loading the page made my brain assume that this is about combining front and rear image sensors, and additionally using eye/facial tracking to infer user intention + satisfaction to help with stabilization/framing/post-processing. I'm glad(?) we aren't there yet.

by jrflo 2mo ago

selectodude 2mo ago

I think this is just Deep Fusion.

deleted 2mo ago
[deleted]

walrus01 2mo ago

Phones have been doing this for a LONG time, if you had a oneplus 7 pro (released may 2019) and took it into an extremely dark area, like where your eyes can't see anything at all, and take a 'night mode' photo, it asks you to hold it steady for about 4-5 seconds and is clearly taking a series of photos with long exposure time. Then it stacks them and uses data from the sensor of hand movement jitter to deblur. The result when done outdoors (like, standing in a dark forest at midnight) will be a photo where you can actually see what's around you, and details show up as if it was a brightly full moon lit night.

somehnguy 2mo ago
I think it's interesting that the same functionality appeared on iPhone in 2019 as well. Google tells me it was the iPhone 11 and iOS 13.
- marmarama 2mo ago
  Night Sight appeared on the Google Pixel camera app in 2018. I think Nokia was doing it first on phones before Microsoft destroyed the remains of their phone business. The reborn Nokia/HMD was also doing multi-sensor image stacking back in 2019 on the Nokia 9 Pureview.
  - throwaway219450 2mo ago
    The related paper (2019): https://google.github.io/night-sight/ https://google.github.io/night-sight/ Google has some great blogs about their camera technology https://research.google/blog/night-sight-seeing-in-the-dark-on-pixel-phones/ https://research.google/blog/night-sight-seeing-in-the-dark-...
hbn 2mo ago
That's a bunch of photos from the same camera, not 2 lenses at the same time
- walrus01 2mo ago
  I know, I just don't think it's a huge jump from "take multiple photos over a 2s period from one camera and stack them" to "take several photos from discrete physical cameras and stack them". As a subscription of paid app I am not the target market for the person who made this, I agree with another person commenting in here that said they should be working for a phone manufacturer.
  - omneity 2mo ago
    I’d argue the opposite. The geometry involved in reconciling photos from two separate lenses with different focal lengths is quite challenging. If you pay close attention you will see some differences in the before/after samples in the site (for example the ear in that kitchen selfie). The math is surely more complex than aligning same-lens photography even when you consider the change in angle and perspective between two shots from the same lens.
    - account42 2mo ago
      Not really. Whether you have two different lenses or the same lenses with some offset from hand movement you need to accurately model the lens(es). Once you can do that for one lens you can just as easily do it for two - it's just more variables to optimize, not a fundamentally harder problem. Panorama software has supported images from different lenses since forever.
LoganDark 2mo ago
There is an iOS app that does this called Spectre: https://spectre.cam https://spectre.cam From the authors of Halide.
- ShinyLeftPad 2mo ago
  It's different from just stacking exposures, it's a paid app that does progressive animation of exposures for creative effect and says it's "AI-powered" whatever that means.
  - LoganDark 2mo ago
    It's a paid app that I literally have, and all it does is take multiple photos and composite them together. It can either naïvely average them together in-place like a traditional long exposure, or align them using machine learning. I'm actually appalled they use the term "AI" considering who they are but that doesn't matter, it's ML. I don't use any of the animation crap, but the aligned long exposures can be useful for ultrawide takes since that camera is so noisy. Edit: actually I can't seem to find the setting that aligns them. I seem to remember it being able to do that but maybe it doesn't... which would be disappointing...
    - walrus01 2mo ago
      People are slapping "AI" on anything these days to try to sell more units of whatever it is.
    - sandofsky 2mo ago
      > I'm actually appalled they use the term "AI" considering who they are but that doesn't matter, it's ML. I agree, and I built it! The app launched in 2019. Ideally I would have said ML, but it felt like AI had more recognition among non-technical users. If I launched it in 2026, it would 100% say "ML" not only because it's more technically accurate, but to make it abundantly clear it doesn't generate slop.
    - ShinyLeftPad 2mo ago
      I have it too but after like 3 times I never used it again, felt like a cheap gimmick. I regret falling for hype. If anything they do good marketing
xyzzy_plugh 2mo ago
The Amazon Fire Phone did this in 2014. I'm not aware of any prior art in mainstream devices aside from gimmicks like 3D (stereoscopic) photos. I can't remember if the Fire Phone had any relevant gimmicks but it certainly used multiple lenses to produce a single image.
- walrus01 2mo ago
  UAV sensor people made a thing that uses 300+ phone size sensors in a single pod, and software workflow to fuse the output together. It's been around quite a few years now. https://www.google.com/search?client=firefox-b-d&q=ARGUS+photo+pod+uav https://www.google.com/search?client=firefox-b-d&q=ARGUS+pho...
  - xyzzy_plugh 2mo ago
    Ah yes, how could I forget the ARGUS-IS, a household staple of the early 21st century. Those were the days...
    - walrus01 2mo ago
      I keep an extra one next to my panini press, the image quality from the whiteboard notes area on my fridge has never been sharper!

stavros 2mo ago

This should be easy to check. In these photos, the center of the image has higher detail than the edges.

kndidneid 2mo ago
Maybe on this guy’s app, but not the way Apple does it.
- account42 2mo ago
  If that's the case then it would disprove the original assertion that Apple already does the same thing. You can't have the same high detail at the edges without making shit up because those parts of the image are out of frame for the higher zoom lens.

sajomes 2mo ago

Yes exactly, the site shows photos all taken from within the app. I have done A/B tests against photos taken in Camera.app and the Photosynthesis composite does significantly beat the detail central area covered by the second lens in Photosynthesis. Because of the limitations of capturing multiple lenses at once through isVirtualDeviceConstituentPhotoDeliveryEnabled, there is a second capture mode in the app for 'Quality' which takes a simultaneous capture then takes sequential shots from each lens which do benefit from Deep Fusion, then uses those for the stitch (as long as the Deep Fusion images outperform the simultaneous capture results). That's also what I do for our Night mode.

101008 2mo ago

Hijacking this comment because of this line that reminded me of something: "There is a crazy amount of image processing going on behind the scenes in each smartphone" I bought the Pixel 10 because it said it had great zooming capabilities. After 5X (that means 10X), it's completely stupid. Basically it takes a very blurred photo and then tries to reconstruct it with AI. The final results are completely awful and really distant from reality. Any advice in how to improve that?

RamblingCTO 2mo ago
Wasn't that the same manufacturer that got caught with the moon AI stuff? Lol
- dgacmu 2mo ago
  The was Samsung, iirc: https://www.reddit.com/r/Android/comments/11nzrb0/samsung_space_zoom_moon_shots_are_fake_and_here/ https://www.reddit.com/r/Android/comments/11nzrb0/samsung_sp...
  - RamblingCTO 2mo ago
    Gotcha, thanks! Funny that they didn't learn from that one
- viciousvoxel 2mo ago
  That was samsung
dgacmu 2mo ago
In photos after you can toggle between the AI modified and original. The original isn't as blurry as the preview shown - there's still a lot of image stacking happening and it's often very decent in my experience. Edited to add: I have a pixel 10 pro which has a better zoom lens, so I could be having a different experience than you...
nemomarx 2mo ago
You can use third party camera apps - https://f-droid.org/en/packages/net.sourceforge.opencamera/ https://f-droid.org/en/packages/net.sourceforge.opencamera/ is a decent one. I know there's a few professional ish options too with dslr ish focus control.

00kevn 2mo ago

One time I took a photo of a snake in the road with my iphone in low light. When I looked at the photo afterwards the snake was removed by the AI enhancement

callc 2mo ago
Similar story but my low-light snake photo turned out much less “scary” than in real life. I’m guessing it has to do with how intensely focused / scared my brain was with the little guy

kndidneid 2mo ago

It is part of Apple’s “secret sauce”, yes.

jrflo 2mo ago
Thank you for confirming my suspicions, lol

Modified3019 2mo ago

I use iPhones (paid for by work) as a tool for my agronomy job. I take a large amount of photos throughout the day to help document what I find in the field. Since at least the iPhone 12, holding the phone still and letting scene sit for a few seconds will result in a sharper photo. I’m not sure if it’s a subtle version focus stacking or a sharpening algorithm, but it’s definitely there. On my 16e, immediately looking at a photo just taken can result in the image noticeably sharpening after a second or 2, so this may be a post processing step as well.

Gigachad 2mo ago
From what I can tell, it's taking multiple photos and stacking them in post. So giving the phone more time to capture sharp exposures to stack would do a lot to help.
KeplerBoy 2mo ago
It's probably a version of this: https://dl.acm.org/doi/abs/10.1145/3306346.3323024 https://dl.acm.org/doi/abs/10.1145/3306346.3323024 Subtle movement between shots can be used to create an image sharper than any of the inputs.
Cthulhu_ 2mo ago
I found this as I was taking a picture of a hare after dark without flash, the image that came up was both relatively sharp and much more illuminated / clear than my own eyes; it can't just have been long exposure time (the mechanical stabilization in phone cameras can't be that good), so I'm confident it takes many photos and layers them smartly, not unlike astrophotography.
HumblyTossed 2mo ago
For some time how, when you have the camera open and lining up a shot, iOS is taking image after image. When you press the shutter, algorithms determine the best or sharpest shot and processes it further. Phil introduced it with iPhone 11 as "deep fusion"

Melatonic 2mo ago

Don't they do this with the "macro" setting already ?

dbbk 2mo ago

But they do do this? They talk about it? It's not a secret

numpad0 2mo ago

Reading title and this ^ comment before loading the page made my brain assume that this is about combining front and rear image sensors, and additionally using eye/facial tracking to infer user intention + satisfaction to help with stabilization/framing/post-processing. I'm glad(?) we aren't there yet.