18 ms·

A reality bending mistake in Apple's computational photography

504 points by indrora 3y ago

po__studio 3y ago
That’s fascinating
leephillips 3y ago
The article says that “The final composite image should be the best, most realistic interpretation of that moment.” But that doesn’t make any sense. If the there were three real people, rather than one person and two reflections, the algorithm would have created an image of a moment that never existed, stitching together different images taken at different times. The only difference is that we might not notice and mistake the fake image for what we expect from a old-fashioned photograph. I find what Apple is doing repulsive.
- tmalsburg2 3y ago
  Imagine such an image being used as evidence is a court case. E.g. showing someone pointing a gun at another person when that actually never happened.
  - Aurornis 3y ago
    Seems extremely unlikely. The phone isn’t going to make drastic alterations to reality. It’s just combining multiple exposures. It can’t make someone point a gun if they’re not pointing a gun. You could try to imagine a scenario where someone was moving extremely fast through the frame while someone was whipping a gun around at high speed in the same direction and the phone just happened to stitch two moments a fraction of a second apart such that they look slightly closer together, but to get to that point you’d still need someone pointing a gun in the same direction that someone is going. It’s really hard to imagine a scenario where two exposures a fraction of a second apart could be stitched together to tell a completely false story, but maybe it exists. In that case, I suspect the lawyers would be all over this explanation to try to dismiss the evidence.
    - tmalsburg2 3y ago
      It’s unlikely, true. But precisely that makes it so dangerous. If such a picture is presented as evidence in a murder case, the possibility that it is telling the wrong story will be discounted and someone may go to prison for the rest of their lives.
    - adastra22 3y ago
      What the comment you're replying to is saying could absolutely have happened. Imagine a "Han shot first" sort of situation: two people with guns, one shoots the other. The shooter claims it was self-defense, as the other guy went to fire first but was just slower. An iPhone picture captures the moment, but has the shooter firing, and the other guy's gun still at his side. This is perfectly analogous to TFA--notice that the woman has enough time to move her arms into very different positions in the same composited moment.
    - 0cVlTeIATBs 3y ago
      Well, the Rittenhouse case had a very important moment when the victim admitted that, while looking at a freeze-frame view of a video taken when he was shot, he had raised his pistol which was pointed at Rittenhouse only a fraction of a second before. [0] That photo was critical for the defense getting him to say that. There were also concerns over wether or not zooming in on an iPad should be allowed in that case--like if a red pixel next to a blue could create a purple one, etc. [0] https://www.denverpost.com/2021/11/08/shooting-victim-says-he-was-pointing-his-gun-at-rittenhouse/ https://www.denverpost.com/2021/11/08/shooting-victim-says-h...
  - PlunderBunny 3y ago
    As far as the phone knows, there were three people in the photo, and it captured the 'best' picture of each one and composited them.
    - thfuran 3y ago
      Yes, and when there actually are three people, the so-called photograph could, by the same mechanism, show them in an arrangement that never actually happened.
  - dylan604 3y ago
    images like this would easily be torn apart by experts brought in to testify against it. then again, for something this obvious, it probably wouldn't need an expert.
    - tmalsburg2 3y ago
      If this phenomenon is so obvious, why is this story the top post on HN? And why did it take a picture of the mirror scenario to make people aware of this issue? Hell, the article even implies that this an issue only with images of mirrors when that is of course completely false.
      - dylan604 3y ago
        You seem to think that all lawyers are dumb, and unable to defend against photographic evidence. If you think a lawyer would not be able to find a witness that fully understands how the modern mobile device camera systems alter images, you're just not being honest with yourself. The Apple propaganda videos tout the fact their cameras are doing something with "AI" even if they don't tell you exactly what. To assume that people are so unawares that it took this picture is just not being honest with the conversation
        tmalsburg2 3y ago
        No, I do not think that all lawyers are dumb. What a bizarre thing to say. Why derail an interesting conversation in such an aggressive way?
        dylan604 3y ago
        I guess we have different definitions of aggressive, but okay, wasn't meant to be aggressive. Your comment about "so obvious" seems a bit obtuse to me and is where the conversation went off the rails. Why are you even questioning how obvious this is one a source image as the one from the TFA? Just based on that, I rejected your premise of this conversation as not being very conducive to anything approaching realistic.
  - _kb 3y ago
    That threat model has existed since the birth of photoshop et al.
    - autoexec 3y ago
      There's a huge difference between someone intentionally altering an image according to their wishes and someone not even aware of changes that have been done. Before, forensic experts could decide if an image had been altered in photoshop, but I guess the only sane conclusion now is that anything taken with an iphone is fake and untrustworthy.
      - KennyBlanken 3y ago
        As opposed to Samsungs which will take any vaguely circular bright white object and turn it into the moon? https://www.reddit.com/r/Android/comments/11nzrb0/samsung_space_zoom_moon_shots_are_fake_and_here/ https://www.reddit.com/r/Android/comments/11nzrb0/samsung_sp...
        autoexec 3y ago
        Yeah, that's trash too. At this point I think it's fair to say you just can't trust a picture taken from a cell phone.
        Dah00n 3y ago
        Yes, except it is called "Moon Mode" isn't it? The iPhone default mode isn't called "unReality Mode", so they aren't saying they are faking it. I don't know if Samsung state it is faked, but Apple definitely does not.
    - flashback2199 3y ago
      Scenario: It's a decade from today and phones are not just stitching together multiple photos but also using generative AI. Apple and all of the other phone makers insist the implementation is safe. The suspect appears in the photo doing something they never actually did. In the chaos of the crime, the phone was left at the crime scene where it was found by law enforcement, no chance the photo could have been photoshopped. The photo is shown as evidence in court. Without the photo, there is not enough evidence to convict. The entire case now hinges on the content of the photo. The suspect insists they never did what is shown in the photo, but it doesn't matter. The suspect is convicted, put on death row, and executed by the state. Thankfully, there is a silver lining: everyone's photos are a couple percent prettier, which helped bring more value to shareholders.
- dylan604 3y ago
  How is what it is attempting to do any different from when someone takes multiple pictures of a group shot, and then uses the different shots to ensure no blinks and "best" expression from each subject? There's a reason professional photogs use the mulitiple snaps mode for non-sports. It used to be a lot of work in post, but a lot of apps have made that easier to now it's a built in feature of our phones.
  - leephillips 3y ago
    The difference is in what some photographers call “editorial integrity”. There’s nothing wrong with any kind of image manipulation, as long as it’s done knowingly and deliberately, and as long as the nature of the manipulation is known to the audience. But the typical iphone consumer is just taking snaps and sharing them around, and almost no one knows what’s really happening. It’s creepy and unethical.
    - Toutouxc 3y ago
      > and almost no one knows what’s really happening And almost no one cares, btw. And what do you even propose? A mandatory “picture might not represent reality” watermark? Because the way I see it, you either take the computational features away from people, and prevent them from taking an entire class of pictures, or you add a big fat warning somewhere that no one will read anyway, or you keep things the way they are. Which one of these is the ethical choice?
      - leephillips 3y ago
        I’m not proposing anything mandatory. I would like it if the biggest corporation in the world would consider the effects of their technology on society, rather than solely on their profits. That they would at least let ethical considerations potentially influence their design decisions. But that’s not how they got to be the biggest company in the world, so I don’t expect this to happen.
  - deleted 3y ago
    [deleted]
- drtgh 3y ago
  "interpretation of that moment" from a camera it is not a photograph, it's a fake image kidnapping the word "photograph".
  - autoexec 3y ago
    It's impossible to enforce, but it'd be a more honest if people called them something else. iPictures maybe? As in "Hey, check out this iPicture™ of my kid taking his first steps!"
  - Toutouxc 3y ago
    A photograph is always an interpretation. A photograph from, say, a modern $2000 big boy camera: Does not capture the color of every pixel, and merely infers it from the surrounding ones. Is usually viewed on a sRGB screen with shitty contrast and a dynamic range significantly smaller than the one of the camera, which is significantly smaller than the one of human eyes, which is still significantly smaller than what we encounter in the real world. Does not capture a particular moment in time, but a variable length period that’s also shifted between the top part of the image and the bottom part (a couple of ms for mechanical shutter, tens or hundreds of ms for electronic). Has no idea about the white balance of the scene. Has no idea about the absolute brightness of the scene. Usually has significant perspective distortion. Usually has most of the scene out-of focus and thus misrepresents reality (buildings aren’t built and people aren’t born out of focus).
    - drtgh 3y ago
      Yep, of course, the colors, the sensor or the film chemicals that react to the frequencies of those photons, the angles of those photons through the lenses, all adapted to our range of view and color perception could be considered even are in the limit of the physics within natural perception, what also could be being crossed when the parameters and mechanics takes certain values, and from MHO its more or less acceptable, our naked eyes can discern how are being evaluated those parameters and mechanics. Nevertheless, what we are seeing with the post-process of the cameras described in the OP cross the threshold of what until this moment in the history is considered natural "photograph" that can be obtained analogically. Those are manipulated images, undetectable for the naked eye, that are incorporating elements, those are a "composition" that can not be obtained analogically. those images are fakes, lies. and by the read in the comments it seems the user can not even disable such composed interpretation.
dgacmu 3y ago
I had a related experience with my Pixel 7 a few months ago. In this picture of a deer: https://photos.app.goo.gl/qChwaw9C29WVdAmr6 https://photos.app.goo.gl/qChwaw9C29WVdAmr6 If you zoom in, you'll note that .. everything at a detail level looks like an oil painting, especially the deer's face and the wall behind it. Very weird effect, and that's certainly not what the wall or deer actually look like. No filters applied. Computational photography is really awesome and modestly worrisome.
- rafram 3y ago
  That’s due to basic denoising, not anything more advanced. I have photos from my iPhone 6s circa 2016 that look similar when you zoom in.
  - dgacmu 3y ago
    interesting - thank you!
  - KennyBlanken 3y ago
    That's not "basic denoising" and photos from my 8 look nothing like that; the "watercolor" or "oil painting" look started around the iPhone 12. Basic denoising is stuff like chroma or hue smoothing. This is very aggressive patterning. That's a daylight photo that the phone turned into an oil painting. https://www.google.com/search?q=pixel+photo+oil+painting+look https://www.google.com/search?q=pixel+photo+oil+painting+loo...
    - dagmx 3y ago
      It’s just what happens when you really aggressively do noise reduction on small sensor data. The noise pattern of any sensor that small will blotch in patterns that when combined with aggressive noise reduction, it will look this way. It’s less apparent on bigger sensors but you’ll see the same if you really crank the noise reduction
      - panarky 3y ago
shrizza 3y ago
This is Apple so we like to call it a reality-distortion effect.
- deleted 3y ago
  [deleted]
ClassyJacket 3y ago
So this wasn't just rolling shutter?
- deleted 3y ago
  [deleted]
- rjeli 3y ago
  No, the difference is too large for it to be rolling shutter. (not sure if the rolling shutter is even vertical or horizontal). It seems like it takes a short video and stitches together the sections with the least motion blur in each frame.
- tmalsburg2 3y ago
  Rolling shutter could in principle explain this, but it’s probably too fast to capture these three poses which must be temporally rather far apart.
  - stephen_g 3y ago
    Not probably too fast - definitely too fast. Rolling shutter occurs over a single read-out of the sensor, so she'd have to have held all the three poses during the 1/100th of a second or so exposure for it to be possibly related to rolling shutter.
    - tmalsburg2 3y ago
      Completely valid nitpick.
- vlovich123 3y ago
  No. Rolling shutter should typically be too fast for a shot like that. Much more likely it's a result of 3 photos taken at slightly different times and automatically composited. Appears to be a well known effect.
- kag0 3y ago
  No. A rolling shutter deals with different parts of the sensor being exposed at different times within one frame. It generally requires a much faster object (think airplane propeller or light with PWM). This is effectively multiple frames captured normally and a mistake in the algorithm stitching them into one frame.
Aurornis 3y ago
Purists will always hate the idea of computational photography, but I love how well my iPhone captures handheld nighttime, low-light, and motion images. I’ve captured some photos at night that I never would have thought possible from a physically small sensor due to the laws of physics. Is it 100% pixel perfect to what was happening at the time? No, but I also don’t care. I’ve used HDR exposure stacking in the past. I’ve used focus stacking in the past for shallow depth of field. I’ve even played with taking multiple photos of a crowded space and stitching them together to make an image of the space without any people or cars. None of them are pixel perfect representations of what I saw, but I don’t care. I was after an image that captured the subject and combining multiple exposures gets the job done.
- tonmoy 3y ago
  I just want to be able to turn it off when I want to
  - HumblyTossed 3y ago
    Agreed. But since the 13 (12 was the last), there's no way to remove even the HDR processing.
- afavour 3y ago
  I have no objection with what we have today either but it does feel slippery slope-y, especially in the AI era. Before we know it our cameras won’t just be picking the best frame for each person, it’ll be picking the best eye, tweaking their symmetry, making the skin tone as aesthetically pleasing as possible, etc etc etc. It was always this way to an extent but now that we’ve given up the pretence of photos being “real” it feels inevitable to me. I’m reminded of that scene in WALL-E where it shows the captain portraits as people get fatter and fatter. It’s clearly inaccurate: over time the photos should show ever more attractive, chisled captains. They’d still be obese in real life though.
  - simias 3y ago
    While I don't use TikTok I often see videos from there and it's really spooky to me how aggressive and omnipresent filtering seems to have become in that community. Even mundane non-fashion non-influencer "vlog" style content is often heavily filtered and, even more scary IMO, I often don't notice immediately and only catch it if there's a small glitch in the algorithm for instance when the person moves something in front of their face. And sometimes you can tell that the difference from their real appearance is very significant. I really wonder what's that doing to insecure teenagers with body issues.
gnabgib 3y ago
Is maybe the original article[0] (referenced by this short recap piece) a better source? [0]: https://petapixel.com/2023/11/16/one-in-a-million-iphone-photo-shows-two-versions-of-the-same-woman/ https://petapixel.com/2023/11/16/one-in-a-million-iphone-pho...
- atomlib 3y ago
  Why not link directly to Instagram instead then? https://www.instagram.com/p/CzPGNmJIebC/ https://www.instagram.com/p/CzPGNmJIebC/
  - jahnu 3y ago
    Off topic, but does anyone know why when I click on the above link in Firefox the back history is gone as if it was opened in a fresh tab? IG doing sketchy stuff to discourage navigating back to where I came from? Or perhaps the Firefox Facebook container protecting me?
    - Kye 3y ago
      It opens Facebook and Instagram in a Facebook container by default. The container has no history.
      - justinclift 3y ago
        Ugh, that seems like a bug which should be fixed. It's pretty inconvenient for users. :(
        Timshel 3y ago
        Probably not trivial since the back button might be linked to history and the container is doing it's job in isolation. I would expect there is a setting to force opening the container in a new tab if you want to be able to go back.
        3y ago
xyztimm 3y ago
So the mirror with the hands clasped?… is just completely made up? Because she’s clearly not doing that irl.
- w-ll 3y ago
  2 mirrors, 3 differnt arm poses. "Photos" are not really any more, the seem to be few second clips, and yea the processing merged 3 differnt poses.
  - threeseed 3y ago
    You are conflating two different aspects. Live Photo is a feature where it captures a couple of seconds of video before/after the photo is taken. From the article that feature was not enabled. The computational pipeline is where you press the shutter and it blends a few frames together in order to do focus stacking, HDR etc. Based on what I have tried with my iPhone it is doing this in < 100ms which is not enough time to produce these sort of artefacts.
    - hnburnsy 3y ago
      Couldn't it be using frames from before the shutter was pressed? Could the processing take longer when people\animals are in the frame?
      - threeseed 3y ago
        Absolutely. But only about less than like 100ms worth at best. The total latency of shutter + computational photography + save to disk is not greater than the time between her moving from one pose to another. So the artefacts you would expect to see if this whole pipeline was out of sync was both poses stitched on top of each other in the mirror. Not multiple ones in different mirrors. And definitely not with the whole image as clean as this.
    - w-ll 3y ago
      maybe? you can also turn your iphone off, but it still broadcast btle and cell signals. who knows what any feature toggle does anymore, let alone on/off. lol do-not-track is tracked.
khazhoux 3y ago
I’m gonna call bullshit here. The difference in arm pose couldn’t be covered in any reasonable burst period.
- charcircuit 3y ago
  I'm skeptical too. Her arms look so stiff so I would exepct her to have been moving slowly.
- mdonahoe 3y ago
  Do you work at Apple?
- Filligree 3y ago
  The “burst” is actually a two second video. Plenty of time for that.
  - deleted 3y ago
    [deleted]
  - threeseed 3y ago
    She said she never used live photo. And the latency for taking a standard photo is not 2 seconds.
    - Filligree 3y ago
      Indeed, since it looks at data from before you press the trigger as well. The camera is always active while you have the camera app; storing a few seconds of pictures is a trivial amount of memory. Live Photo means storing the video in the image file. Turning it off doesn’t mean the video is no longer used, just that it isn’t stored.
- threeseed 3y ago
  Surprised you were downvoted because this doesn't sound right at all. The poses would be at least a few seconds apart which rules out anything in Apple's computational photography pipeline e.g. focus and dynamic range stacking. At least based on what they have communicated to date. We know Google demonstrated an AI model that was capable of selecting human parts from multiple photos but that was a showcase feature not something quietly added.
PlunderBunny 3y ago
I heard mention on a podcast recently [0] that if you hold down the button in the iPhone camera app it will capture a set of photos and then mark the one that it thinks is the 'best' (based on, for example, the photo where everyone has their eyes open). Not the same as what happened here of course. (I keep forgetting to try this, not least because I always try to get the people out of my photos!) [0] This one, I think: https://podcasts.apple.com/nz/podcast/the-talk-show-with-john-gruber/id528458508?i=1000632890966 https://podcasts.apple.com/nz/podcast/the-talk-show-with-joh...
- carterschonwald 3y ago
  I think there’s also how the Live Photo feature works. Where taking a photo is literally a short video and it picks the best frame
- macintux 3y ago
  Unfortunately they changed the behavior a few major releases ago: now holding the button down switches to video recording. Annoying for me, since I don't do video. If you use the software button you can slide to the left for burst, but afaik there's no way to trigger burst photos from the volume buttons. Maybe the new programmable button on the iPhone 15 series.
  - kderbe 3y ago
    https://support.apple.com/guide/iphone/take-burst-mode-shots-ipha42c55cd0/ios https://support.apple.com/guide/iphone/take-burst-mode-shots... "Tip: You can also press and hold the volume up button to take Burst shots. Go to Settings > Camera, then turn on Use Volume Up for Burst."
    - macintux 3y ago
      Thanks, I glanced at that page but didn't notice the tip at the bottom.
  - dmix 3y ago
    That’s still a cool shortcut I didn’t know about. It not only switches to video but it “gates” the button press so it only records a video for as long as you press the button
emptybits 3y ago
If not already, I expect examples like this will be accumulated and trotted out as part of legal defenses. Valid or not, gross examples like this will probably nudge some judges and juries over their threshold for reasonable doubt. AI manipulations are happening and their bounds can be hard to predict.
- ealexhudson 3y ago
  Imagine a politician photoed reacting to a member of the public in disgust, except the face was stitched in from moments after. Or, worse, someone captured at the scene of a bomb reacting before the bomb went off? We have an inbuilt set of assumptions about causality that this AI now violates. That's potentially huge in some very specific scenarios...
  - fennecbutt 3y ago
    Hey I have a free energy machine to sell you
weird-eye-issue 3y ago
With how much manipulation happens to photos by default it seems like it would be easier to get them thrown out of court as evidence
- j16sdiz 3y ago
  Without better evidence, the court will keep accepting them. Just look at how bad fingerprints are accepted
  - callalex 3y ago
    See also: polygraphs.
    - zimpenfish 3y ago
      See also: bite mark evidence[0], blood splatter[1], etc.[2] [0] https://www.theguardian.com/us-news/2022/apr/28/forensics-bite-mark-junk-science-charles-mccrory-chris-fabricant https://www.theguardian.com/us-news/2022/apr/28/forensics-bi... [1] https://features.propublica.org/blood-spatter/blood-spatter-criminal-justice-courts-forensic-evidence/ https://features.propublica.org/blood-spatter/blood-spatter-... [2] https://www.currentaffairs.org/2023/01/we-need-to-get-junk-science-out-of-courtrooms https://www.currentaffairs.org/2023/01/we-need-to-get-junk-s...
  - actionfromafar 3y ago
    I'd say, the courts will keep accepting bad evidence no matter what you do.
- joeframbach 3y ago
  It's not so farfetched. https://www.theverge.com/2021/11/10/22775580/kyle-rittenhouse-trial-judge-apple-ai-pinch-to-zoom-footage-manipulation-claim https://www.theverge.com/2021/11/10/22775580/kyle-rittenhous...
hnburnsy 3y ago
How long is this burst (feels longer than 'short' in this case)? Does it start before you press the shutter button? Does this post processing apply to RAW? Does Apple document how this post processing works? More importantly, how does one turn it off or is there another camera app that allows one to turn it off?
doctoboggan 3y ago
I don't think this is true, Apple hasn't openly said they do this level of manipulation (although that doesn't mean they don't necessarily), and I don't think the range of motions she would have to go through would be possible within a single capture. Even with "Live Photos" this wouldn't happen.
- tiltowait 3y ago
  I’m also skeptical. I want it to be true, because it’s pretty wild, but it seems a little too perfect. I suppose people must already be trying to prove or disprove it.
- nickelpro 3y ago
  Ya the uncritical acceptance of these very non-technical articles on HN is a little disappointing
  - dfxm12 3y ago
    I think you have unrealistic expectations for this community.
- _maelstrom 3y ago
  The article says it was a panorama photo, which makes it much less interesting than live/hdr creating the same effect.
comfysocks 3y ago
My best guess is that there was a brightness gradient across the scene, and this is the result of tone mapping from an EV bracketed burst. This might result in “time delay” instead of the more typical “ghosting” artifacts.
luuurker 3y ago
A modern Google Pixel starts saving frames as soon you open the camera app. When we finally take the picture, it uses some of the older frames to have what's essentially HDR stacking without the delay. I wouldn't be surprised if a similar thing happened here. Different frames, processing picks the best exposure for each part of the picture and you get this effect.
- londons_explore 3y ago
  It's also why having the camera open and previewing destroys the battery. Some people I know keep the camera open for hours - especially when exploring a new place and wanting to take a photo of a cool thing at a moments notice, and then moan their battery dies at lunchtime.
deleted 3y ago
[deleted]
ISL 3y ago
I hope, but am not certain, that by configuring my Pixel to save both a jpeg and a raw for each image, at least the raw would avoid these shenanigans.
- Someone1234 3y ago
  Unless the raw is actually a stack of images, it may not. Both Android and iOS are taking multiple exposures and combining them into a single HDR image. This is before it hits the camera app. This has nothing to do with "Live Photos" on iOS to be clear.
seshagiric 3y ago
I am always amazed how people even catch (come across) issues like this...the chances are 1 in a million?
kazinator 3y ago
Changing the positions of limbs is computational, but it's not photography. Photography refers to the capturing of the reflected light to create an image, which then corresponds to how the scene actually was. Bringing out detail in shadows being called "computational photography": that I could swallow.
- hgomersall 3y ago
  Are colour filters allowed?
  - cubefox 3y ago
    The human eye does color temperature correction automatically, and so do digital cameras, basically since they are available. So yes that is okay obviously.
    - hgomersall 3y ago
      Why obviously? There's a collection of discussed filters here: https://www.northlight-images.co.uk/using-effects-lens-filters/ https://www.northlight-images.co.uk/using-effects-lens-filte... Some colour filters are more than just colour correction. At least one of those results in intentional spatial transformation. Is that allowed?
- thfuran 3y ago
  It's not changing limb positions computationally, it's computationally jamming limb positions recorded at different instants into one 'photograph'.
gnicholas 3y ago
In my family we like to take 'pano-rankenstein' photos, where you do a pano across a person's face as they are rapidly changing expressions in dramatic ways. The results are pretty hilarious, as the phone tries to stitch your face together into one cohesive image.
STELLANOVA 3y ago
Slight off-topic but I still can't believe Apple engineers are not able to fix stupid green dot flare when shooting with direct sunlight in the frame... Super easy to fix computationally but for some reason still there for years...
- panarky 3y ago
  Pretty cool how the green dot became a green crescent during the last solar eclipse.
rompledorph 3y ago
If it has not already happened, evidence will get thrown out of court because the image no longer represents the reality. Too many filters and other AI improvements.
- hnburnsy 3y ago
  Some one is writing an episode of law and order with AI photograohy getting defendant off.
causi 3y ago
One is reminded of how Samsung cameras are trained to put the details of the moon over anything that looks vaguely like the moon, but just one, so you can photograph two blurry images of the moon and one of them will look like it was taken by Hubble.
TeMPOraL 3y ago
Tangent, but: > A U.K. comedian and actor named Tessa Coates Why is it always some kind of celebrity that "discovers" stuff like this? Was it luck (yet again), or are those extreme failure modes of computational photography already somewhat known, just too nerdy to report on until they can be attached to a public person?
- edent 3y ago
  Because the typical Instagram account is only followed by a handful of people. So anything unusual is unlikely to get any traction.
  - TeMPOraL 3y ago
    Might be my age showing, but I swear it wasn't like this just a few years ago - this kind of story would genuinely go viral from the first rando posting it on social media ("wtf is my camera broken?") or the second rando that adapted it into a goofy scene/video ("look I'm a vampire now").
    - londons_explore 3y ago
      Social media algorithms have changed. Viral things used to be semi-random (eg. the 6:01pm on sunday bug, which heavily promoted things initially published at 6:01pm on a sunday because thats when some weekly stats aggregation function ran, and the age of the video in seconds was used as a divisor, so you got huge popularity boosts for the first week if you could post in the right second).
      - actionfromafar 3y ago
        Yep. The algorithms have locked onto value stream extraction mode.
Frummy 3y ago
I was in a butterfly house, and closeup photos removed the legs of the butterflies to keep only the wings lol. e: image link https://ibb.co/nwbw5xY https://ibb.co/nwbw5xY
- londons_explore 3y ago
  That will be the denoising. Legs are thin and can easily be interpreted as noise - especially when using digital zoom and dim light so there are very few pixels and each pixel is very noisy.
  - gpas 3y ago
    More like the poor masking algorithm interpreted the legs as background and blurred them.
  - cubefox 3y ago
    Denoising is often great, but sometimes it really destroys details. It is basically impossible to capture a snowy day (or heavy rain) with a digital camera. The denoiser will remove 90% of the snowflakes. Analog cameras, by contrast, remove nothing.
    - londons_explore 3y ago
      Interestingly, some phones now use a model of the camera sensor to detect how much noise there 'should' be (based on temperature, age of sensor, test results of the specific sensor from the factory, exposure time and light levels). They then only remove that much noise, hopefully leaving snow. Every other phone just has an algorithm to estimate how much noise is in an image by sampling a few patches - and that will remove snow.
      - cubefox 3y ago
        Interesting. Do you know which phones have this feature?
    - 3y ago
Moldoteck 3y ago
Computational photography is just at it's first steps. Google photos on p8 allow to replace faces. I expect this feature will be implemented to be done automatically to show you the ideal photo. And I expect some parents will be happy with it - having an ideal photo with ideal face of their children
karmakaze 3y ago
'photo' is now synonymous with photoshopped.
karmakaze 3y ago
It wasn't long before we hit this Xerox digits[0] moment in 'photos'. What features do you want in your photocopier or photos? [0] https://www.theverge.com/2013/8/6/4594482/xerox-copiers-randomly-replacing-numbers-in-documents https://www.theverge.com/2013/8/6/4594482/xerox-copiers-rand...
- coremoff 3y ago
  very different causes though; the xerox one was an image compression bug, IIRC; this is a "chop up multiple photots and stich together the best composite". Still got room for library bugs! (wasn't the Samsung moon pictures sort of more like the xerox one?)
sinuhe69 3y ago
Of course, this is one in a million chance but it highlights very nicely IMO a much bigger issue: what is reality? Some would say only the classical optical camera would capture faithfully our reality. But does it? The reality of the sunlight is a broad spectrum of radio emissions: UV, infrared and more. Does the optical camera capture these? No. Thus, which reality does it capture? Our perceived reality? Other would argue: at least the optical system would capture events in time faithfully. But does it? What would we see in a femto second? Certainly not the pictures we normally see. So the results of an optical system are also super imposed realities, not very much different than the results of a computational photography. There is simply no single one reality, only our perceived realities. But if so, can we still call it reality or it’s merely a product of our sense, our perception and hallucination?
- Gigachad 3y ago
  Probably something closest to what your eye sees is ideal for most photos. But dumb optical cameras have all kinds of artefacts that eyes don’t. When I slightly bump the camera, the whole image comes out blurry, my eyes don’t do that. Things like lens flares don’t exist either.
  - andreicap 3y ago
    > all kinds of artefacts that eyes don’t. Eyes do have lots of artefacts, your brain fills in the gaps, like the blind spot [1] It's not much more different than computational photography, really. [1] https://en.wikipedia.org/wiki/Blind_spot_(vision) https://en.wikipedia.org/wiki/Blind_spot_(vision)
- TeMPOraL 3y ago
  > There is simply no single one reality, only our perceived realities. This does not follow at all from your earlier paragraph. The reality we're talking about here, which regular photography reflects while computational one doesn't, is the correlation of recorded data with the state of the world. The pixels of a regular photo are highly correlated with reality - they may have been subject to some analog and digital transformation, and of course quantization, but there's a straightforward and reversible (with some loss of fidelity) function mapping pixels to the photographed event. Computational photography, in contrast, decorrelates pixels from reality, and discards source measurements, leaving you with a recording of something that never happened, but is sort of similar to the thing that did. I elaborated on this elsewhere in the thread, so let me instead point at another way of noticing the difference. Photogrammetry is the science and technology of recovering the information about reality from photos, and it works because pixels of regular photos are highly correlated with reality. Apply the same techniques to images made via computational photography, and the degree of uncertainty and fidelity loss will reflect the degree to which the computational photos are AI interpretations/bullshit.
NikkiA 3y ago
If the 'camera' is guessing as to intent, it's no longer a camera, sorry.
- Gigachad 3y ago
  It isn’t guessing. It’s just stacked multiple shots. Everything in the photo is real, just at slightly different moments.
- devnullbrain 3y ago
  Every digital photo interpolates or otherwise guesses the unknown data between sensors of each color during debayering.
saiya-jin 3y ago
Next time somebody here will be bashing other manufacturers like say Samsung for clarifying 50x zoom moon shots (like most of HN did when this was a topic for a day or two few months ago), remember kids that this is what all manufacturers do. Or half-reversing photo of kitten in the grass as somebody mentioned.
dusted 3y ago
This happens more and more, I guess it's an unholy mix of "better compression as long as you don't actually look at the image" and "ai improvement".. A few weeks ago, I took a really lovely picture of my son, composition, facial expression, focus, light, it was _PERFECT_ except.. The algorithms in my phone had decided that it'd be better to scrape off his fucking skin and replace it with the texture of the wall behind him! Of course it must be my fault for buying such a cheap phone, it only a Galaxy 22 Ultra, I'm sure the 23 Ultra is better... But it was not out when I changed phones.. Wtf^wtf.. So I go turn on RAW so I can at least salvage picture in the future, except RAW only works in the "pro" camera mode which is inconvenient to use and sometimes it silently falls back to non-pro.. In the end I gave up and installed a third party camera application, I guess I just have to trust Mark, at least he hasn't actively messed up my photos. https://play.google.com/store/apps/details?id=net.sourceforge.opencamera https://play.google.com/store/apps/details?id=net.sourceforg...
- fennecbutt 3y ago
  I have a Samsung S23U and previously a S22U (and before that one plus) and I've never had that happen to any of my photos (and I take a lot). Hell, I don't think it's using computational photography besides basic denoising/lightening magic for low light (and their stupid moon replacement thing...they already beat Apple just having the 10x lens, no need for the moon bs). I think the default camera app is fine.
  - dusted 3y ago
    Here's a photo taken by my S22U, http://dusted.dk/pages/computers/J-11/DEC_J-11_Gallery/full/2.jpg http://dusted.dk/pages/computers/J-11/DEC_J-11_Gallery/full/... that's a cardboard box on top of a matte PC case, but somehow the phone sees this strange blurred pattern in it, as if the phone was moving (which it was not) while at the same time, the label is artificially sharp (almost as if edge detection is taking place), at the same time, you can't see a single fiber of the cardboard box (except for the very edge where it looks like it's grown hair, which it has not).. Have you maybe looked close enough at your photos ?
ClumsyPilot 3y ago
So many comments, not a single one pointing out that someone will go to prison over this 'photo', when it shows something that never actually happened.
- PUSH_AX 3y ago
  We've been able to create images and video of things that didn't happen for decades.
  - Dah00n 3y ago
    This is not the same thing and you know it.
verisimi 3y ago
And, of course, there is the story where Samsung replaces the moon with its high res version. I don't think photography from phone can actually be trusted to be a faithful representation of reality as it is not a purely mechanical series of actions. This will be even worse if ai is involved in the process. I certainly think that there is an argument for photographic evidence to be inadmissible in court.
verytrivial 3y ago
Defense lawyers take note.
tomovo 3y ago
Good luck capturing a sprint finish with that kind of camera.
- actionfromafar 3y ago
  The photo shows they all ran over the line at the same time!
deleted 3y ago
[deleted]
bmacho 3y ago
This reminds me of printers, scanners and photocopies that have a "compressing" step, and replace numbers with other numbers just for fun. Like you have one job, make the numbers correct, and you fail to do that.
- cubefox 3y ago
  Yeah. The difference is that people get really angry when they hear scanners are doing this, while in the current case (look at the other comments) most people treat it like a perfectly acceptable curiosity.
pluc 3y ago
It's funny that the iPhone has half a dozen cameras, uses the cameras as the main selling point (look at posters!) yet the image generated is coming essentially from a piece of code.
- octacat 3y ago
  Code is important too. They sell a good combo.
  - pluc 3y ago
    Yeah but I don't see that put forward nearly as much as all the camera tech they keep updating and promoting. I was just in Italy and they have building-sized promotional posters zoomed in on all the cameras at the back of the phone. Agreed it doesn't lend itself to advertising so much because it's less.. visual, but still I thought that was interesting.
    - octacat 3y ago
      marketing rarely correlates with reality. nowadays. uber: there is a car in 8 minutes from uuu! oh, thanks for ordering. now it is 20 minutes (30 in reality). and even if correlates, it is cherrypicked (hi, sony camera specs.)
vachina 3y ago
So can we say iPhone pictures are not admissible in the court of law? Since the photos are more like hand drawings (I.e. subject to interpretation by the algo) than a snapshot of time.
k310 3y ago
May I suggest getting a nice small pocket camera? I have used a lumix with 8X optical zoom (for a very long time) that is smaller than most smartphones (albeit thicker, thanks to the fetish for ultra-thin phones), has a removable SD card, and doesn't do processing beyond what it normally takes to make jpegs. Newer ones might have raw, I haven't checked. It has outlived many phones, has a removable battery, doesn't have a single f/1.8 aperture, and has a macro mode that doesn't require you to be 1/4 inch from the subject. If it's lost or stolen, I haven't handed over all my financial and personal data. What will they think of next?
_maelstrom 3y ago
It's buried in the article, but: "Coates was moving when the photo was taken, so when the shutter was pressed, many differing images were captured in that instant as the camera swept over the scene, since it was a panoramic photo capture." So it's a pano doing normal pano things, but it's just surprising in context. If you have not yet tried it, you can have friends jump in and out of the frame as you slowly move across a scene and have silly photos where they appear more than once! Edit: typo
realclayman 3y ago
This is very possible in panoramic mode. See the update in story: Update December 2, 11:05 Updated with details about the photo being shot in Panoramic mode, which is why the bride-to-be had time to change position between the shots found in the mirror.
fennecbutt 3y ago
And people laughed at Samsung's moon shot thing. I still use my phone camera, but man do I appreciate that all my Sony a7 may do is interpret the sensor using a colour profile and that's it.
classified 3y ago
Well, duh.

504 points by indrora 3y ago

That’s fascinating

The article says that “The final composite image should be the best, most realistic interpretation of that moment.” But that doesn’t make any sense. If the there were three real people, rather than one person and two reflections, the algorithm would have created an image of a moment that never existed, stitching together different images taken at different times. The only difference is that we might not notice and mistake the fake image for what we expect from a old-fashioned photograph. I find what Apple is doing repulsive.

tmalsburg2 3y ago
Imagine such an image being used as evidence is a court case. E.g. showing someone pointing a gun at another person when that actually never happened.
- Aurornis 3y ago
  Seems extremely unlikely. The phone isn’t going to make drastic alterations to reality. It’s just combining multiple exposures. It can’t make someone point a gun if they’re not pointing a gun. You could try to imagine a scenario where someone was moving extremely fast through the frame while someone was whipping a gun around at high speed in the same direction and the phone just happened to stitch two moments a fraction of a second apart such that they look slightly closer together, but to get to that point you’d still need someone pointing a gun in the same direction that someone is going. It’s really hard to imagine a scenario where two exposures a fraction of a second apart could be stitched together to tell a completely false story, but maybe it exists. In that case, I suspect the lawyers would be all over this explanation to try to dismiss the evidence.
  - tmalsburg2 3y ago
    It’s unlikely, true. But precisely that makes it so dangerous. If such a picture is presented as evidence in a murder case, the possibility that it is telling the wrong story will be discounted and someone may go to prison for the rest of their lives.
  - adastra22 3y ago
    What the comment you're replying to is saying could absolutely have happened. Imagine a "Han shot first" sort of situation: two people with guns, one shoots the other. The shooter claims it was self-defense, as the other guy went to fire first but was just slower. An iPhone picture captures the moment, but has the shooter firing, and the other guy's gun still at his side. This is perfectly analogous to TFA--notice that the woman has enough time to move her arms into very different positions in the same composited moment.
  - 0cVlTeIATBs 3y ago
    Well, the Rittenhouse case had a very important moment when the victim admitted that, while looking at a freeze-frame view of a video taken when he was shot, he had raised his pistol which was pointed at Rittenhouse only a fraction of a second before. [0] That photo was critical for the defense getting him to say that. There were also concerns over wether or not zooming in on an iPad should be allowed in that case--like if a red pixel next to a blue could create a purple one, etc. [0] https://www.denverpost.com/2021/11/08/shooting-victim-says-he-was-pointing-his-gun-at-rittenhouse/ https://www.denverpost.com/2021/11/08/shooting-victim-says-h...
- PlunderBunny 3y ago
  As far as the phone knows, there were three people in the photo, and it captured the 'best' picture of each one and composited them.
  - thfuran 3y ago
    Yes, and when there actually are three people, the so-called photograph could, by the same mechanism, show them in an arrangement that never actually happened.
- dylan604 3y ago
  images like this would easily be torn apart by experts brought in to testify against it. then again, for something this obvious, it probably wouldn't need an expert.
  - tmalsburg2 3y ago
    If this phenomenon is so obvious, why is this story the top post on HN? And why did it take a picture of the mirror scenario to make people aware of this issue? Hell, the article even implies that this an issue only with images of mirrors when that is of course completely false.
    - dylan604 3y ago
      You seem to think that all lawyers are dumb, and unable to defend against photographic evidence. If you think a lawyer would not be able to find a witness that fully understands how the modern mobile device camera systems alter images, you're just not being honest with yourself. The Apple propaganda videos tout the fact their cameras are doing something with "AI" even if they don't tell you exactly what. To assume that people are so unawares that it took this picture is just not being honest with the conversation
      - tmalsburg2 3y ago
        No, I do not think that all lawyers are dumb. What a bizarre thing to say. Why derail an interesting conversation in such an aggressive way?
        dylan604 3y ago
        I guess we have different definitions of aggressive, but okay, wasn't meant to be aggressive. Your comment about "so obvious" seems a bit obtuse to me and is where the conversation went off the rails. Why are you even questioning how obvious this is one a source image as the one from the TFA? Just based on that, I rejected your premise of this conversation as not being very conducive to anything approaching realistic.
- _kb 3y ago
  That threat model has existed since the birth of photoshop et al.
  - autoexec 3y ago
    There's a huge difference between someone intentionally altering an image according to their wishes and someone not even aware of changes that have been done. Before, forensic experts could decide if an image had been altered in photoshop, but I guess the only sane conclusion now is that anything taken with an iphone is fake and untrustworthy.
    - KennyBlanken 3y ago
      As opposed to Samsungs which will take any vaguely circular bright white object and turn it into the moon? https://www.reddit.com/r/Android/comments/11nzrb0/samsung_space_zoom_moon_shots_are_fake_and_here/ https://www.reddit.com/r/Android/comments/11nzrb0/samsung_sp...
      - autoexec 3y ago
        Yeah, that's trash too. At this point I think it's fair to say you just can't trust a picture taken from a cell phone.
      - Dah00n 3y ago
        Yes, except it is called "Moon Mode" isn't it? The iPhone default mode isn't called "unReality Mode", so they aren't saying they are faking it. I don't know if Samsung state it is faked, but Apple definitely does not.
  - flashback2199 3y ago
    Scenario: It's a decade from today and phones are not just stitching together multiple photos but also using generative AI. Apple and all of the other phone makers insist the implementation is safe. The suspect appears in the photo doing something they never actually did. In the chaos of the crime, the phone was left at the crime scene where it was found by law enforcement, no chance the photo could have been photoshopped. The photo is shown as evidence in court. Without the photo, there is not enough evidence to convict. The entire case now hinges on the content of the photo. The suspect insists they never did what is shown in the photo, but it doesn't matter. The suspect is convicted, put on death row, and executed by the state. Thankfully, there is a silver lining: everyone's photos are a couple percent prettier, which helped bring more value to shareholders.
dylan604 3y ago
How is what it is attempting to do any different from when someone takes multiple pictures of a group shot, and then uses the different shots to ensure no blinks and "best" expression from each subject? There's a reason professional photogs use the mulitiple snaps mode for non-sports. It used to be a lot of work in post, but a lot of apps have made that easier to now it's a built in feature of our phones.
- leephillips 3y ago
  The difference is in what some photographers call “editorial integrity”. There’s nothing wrong with any kind of image manipulation, as long as it’s done knowingly and deliberately, and as long as the nature of the manipulation is known to the audience. But the typical iphone consumer is just taking snaps and sharing them around, and almost no one knows what’s really happening. It’s creepy and unethical.
  - Toutouxc 3y ago
    > and almost no one knows what’s really happening And almost no one cares, btw. And what do you even propose? A mandatory “picture might not represent reality” watermark? Because the way I see it, you either take the computational features away from people, and prevent them from taking an entire class of pictures, or you add a big fat warning somewhere that no one will read anyway, or you keep things the way they are. Which one of these is the ethical choice?
    - leephillips 3y ago
      I’m not proposing anything mandatory. I would like it if the biggest corporation in the world would consider the effects of their technology on society, rather than solely on their profits. That they would at least let ethical considerations potentially influence their design decisions. But that’s not how they got to be the biggest company in the world, so I don’t expect this to happen.
- deleted 3y ago
  [deleted]
drtgh 3y ago
"interpretation of that moment" from a camera it is not a photograph, it's a fake image kidnapping the word "photograph".
- autoexec 3y ago
  It's impossible to enforce, but it'd be a more honest if people called them something else. iPictures maybe? As in "Hey, check out this iPicture™ of my kid taking his first steps!"
- Toutouxc 3y ago
  A photograph is always an interpretation. A photograph from, say, a modern $2000 big boy camera: Does not capture the color of every pixel, and merely infers it from the surrounding ones. Is usually viewed on a sRGB screen with shitty contrast and a dynamic range significantly smaller than the one of the camera, which is significantly smaller than the one of human eyes, which is still significantly smaller than what we encounter in the real world. Does not capture a particular moment in time, but a variable length period that’s also shifted between the top part of the image and the bottom part (a couple of ms for mechanical shutter, tens or hundreds of ms for electronic). Has no idea about the white balance of the scene. Has no idea about the absolute brightness of the scene. Usually has significant perspective distortion. Usually has most of the scene out-of focus and thus misrepresents reality (buildings aren’t built and people aren’t born out of focus).
  - drtgh 3y ago
    Yep, of course, the colors, the sensor or the film chemicals that react to the frequencies of those photons, the angles of those photons through the lenses, all adapted to our range of view and color perception could be considered even are in the limit of the physics within natural perception, what also could be being crossed when the parameters and mechanics takes certain values, and from MHO its more or less acceptable, our naked eyes can discern how are being evaluated those parameters and mechanics. Nevertheless, what we are seeing with the post-process of the cameras described in the OP cross the threshold of what until this moment in the history is considered natural "photograph" that can be obtained analogically. Those are manipulated images, undetectable for the naked eye, that are incorporating elements, those are a "composition" that can not be obtained analogically. those images are fakes, lies. and by the read in the comments it seems the user can not even disable such composed interpretation.

dgacmu 3y ago

I had a related experience with my Pixel 7 a few months ago. In this picture of a deer: https://photos.app.goo.gl/qChwaw9C29WVdAmr6 https://photos.app.goo.gl/qChwaw9C29WVdAmr6 If you zoom in, you'll note that .. everything at a detail level looks like an oil painting, especially the deer's face and the wall behind it. Very weird effect, and that's certainly not what the wall or deer actually look like. No filters applied. Computational photography is really awesome and modestly worrisome.

rafram 3y ago
That’s due to basic denoising, not anything more advanced. I have photos from my iPhone 6s circa 2016 that look similar when you zoom in.
- dgacmu 3y ago
  interesting - thank you!
- KennyBlanken 3y ago
  That's not "basic denoising" and photos from my 8 look nothing like that; the "watercolor" or "oil painting" look started around the iPhone 12. Basic denoising is stuff like chroma or hue smoothing. This is very aggressive patterning. That's a daylight photo that the phone turned into an oil painting. https://www.google.com/search?q=pixel+photo+oil+painting+look https://www.google.com/search?q=pixel+photo+oil+painting+loo...
  - dagmx 3y ago
    It’s just what happens when you really aggressively do noise reduction on small sensor data. The noise pattern of any sensor that small will blotch in patterns that when combined with aggressive noise reduction, it will look this way. It’s less apparent on bigger sensors but you’ll see the same if you really crank the noise reduction
    - panarky 3y ago

shrizza 3y ago

This is Apple so we like to call it a reality-distortion effect.

deleted 3y ago
[deleted]

ClassyJacket 3y ago

So this wasn't just rolling shutter?

deleted 3y ago
[deleted]
rjeli 3y ago
No, the difference is too large for it to be rolling shutter. (not sure if the rolling shutter is even vertical or horizontal). It seems like it takes a short video and stitches together the sections with the least motion blur in each frame.
tmalsburg2 3y ago
Rolling shutter could in principle explain this, but it’s probably too fast to capture these three poses which must be temporally rather far apart.
- stephen_g 3y ago
  Not probably too fast - definitely too fast. Rolling shutter occurs over a single read-out of the sensor, so she'd have to have held all the three poses during the 1/100th of a second or so exposure for it to be possibly related to rolling shutter.
  - tmalsburg2 3y ago
    Completely valid nitpick.
vlovich123 3y ago
No. Rolling shutter should typically be too fast for a shot like that. Much more likely it's a result of 3 photos taken at slightly different times and automatically composited. Appears to be a well known effect.
kag0 3y ago
No. A rolling shutter deals with different parts of the sensor being exposed at different times within one frame. It generally requires a much faster object (think airplane propeller or light with PWM). This is effectively multiple frames captured normally and a mistake in the algorithm stitching them into one frame.

Aurornis 3y ago

Purists will always hate the idea of computational photography, but I love how well my iPhone captures handheld nighttime, low-light, and motion images. I’ve captured some photos at night that I never would have thought possible from a physically small sensor due to the laws of physics. Is it 100% pixel perfect to what was happening at the time? No, but I also don’t care. I’ve used HDR exposure stacking in the past. I’ve used focus stacking in the past for shallow depth of field. I’ve even played with taking multiple photos of a crowded space and stitching them together to make an image of the space without any people or cars. None of them are pixel perfect representations of what I saw, but I don’t care. I was after an image that captured the subject and combining multiple exposures gets the job done.

tonmoy 3y ago
I just want to be able to turn it off when I want to
- HumblyTossed 3y ago
  Agreed. But since the 13 (12 was the last), there's no way to remove even the HDR processing.
afavour 3y ago
I have no objection with what we have today either but it does feel slippery slope-y, especially in the AI era. Before we know it our cameras won’t just be picking the best frame for each person, it’ll be picking the best eye, tweaking their symmetry, making the skin tone as aesthetically pleasing as possible, etc etc etc. It was always this way to an extent but now that we’ve given up the pretence of photos being “real” it feels inevitable to me. I’m reminded of that scene in WALL-E where it shows the captain portraits as people get fatter and fatter. It’s clearly inaccurate: over time the photos should show ever more attractive, chisled captains. They’d still be obese in real life though.
- simias 3y ago
  While I don't use TikTok I often see videos from there and it's really spooky to me how aggressive and omnipresent filtering seems to have become in that community. Even mundane non-fashion non-influencer "vlog" style content is often heavily filtered and, even more scary IMO, I often don't notice immediately and only catch it if there's a small glitch in the algorithm for instance when the person moves something in front of their face. And sometimes you can tell that the difference from their real appearance is very significant. I really wonder what's that doing to insecure teenagers with body issues.

gnabgib 3y ago

Is maybe the original article[0] (referenced by this short recap piece) a better source? [0]: https://petapixel.com/2023/11/16/one-in-a-million-iphone-photo-shows-two-versions-of-the-same-woman/ https://petapixel.com/2023/11/16/one-in-a-million-iphone-pho...

atomlib 3y ago
Why not link directly to Instagram instead then? https://www.instagram.com/p/CzPGNmJIebC/ https://www.instagram.com/p/CzPGNmJIebC/
- jahnu 3y ago
  Off topic, but does anyone know why when I click on the above link in Firefox the back history is gone as if it was opened in a fresh tab? IG doing sketchy stuff to discourage navigating back to where I came from? Or perhaps the Firefox Facebook container protecting me?
  - Kye 3y ago
    It opens Facebook and Instagram in a Facebook container by default. The container has no history.
    - justinclift 3y ago
      Ugh, that seems like a bug which should be fixed. It's pretty inconvenient for users. :(
      - Timshel 3y ago
        Probably not trivial since the back button might be linked to history and the container is doing it's job in isolation. I would expect there is a setting to force opening the container in a new tab if you want to be able to go back.
        3y ago

xyztimm 3y ago

So the mirror with the hands clasped?… is just completely made up? Because she’s clearly not doing that irl.

w-ll 3y ago
2 mirrors, 3 differnt arm poses. "Photos" are not really any more, the seem to be few second clips, and yea the processing merged 3 differnt poses.
- threeseed 3y ago
  You are conflating two different aspects. Live Photo is a feature where it captures a couple of seconds of video before/after the photo is taken. From the article that feature was not enabled. The computational pipeline is where you press the shutter and it blends a few frames together in order to do focus stacking, HDR etc. Based on what I have tried with my iPhone it is doing this in < 100ms which is not enough time to produce these sort of artefacts.
  - hnburnsy 3y ago
    Couldn't it be using frames from before the shutter was pressed? Could the processing take longer when people\animals are in the frame?
    - threeseed 3y ago
      Absolutely. But only about less than like 100ms worth at best. The total latency of shutter + computational photography + save to disk is not greater than the time between her moving from one pose to another. So the artefacts you would expect to see if this whole pipeline was out of sync was both poses stitched on top of each other in the mirror. Not multiple ones in different mirrors. And definitely not with the whole image as clean as this.
  - w-ll 3y ago
    maybe? you can also turn your iphone off, but it still broadcast btle and cell signals. who knows what any feature toggle does anymore, let alone on/off. lol do-not-track is tracked.

khazhoux 3y ago

I’m gonna call bullshit here. The difference in arm pose couldn’t be covered in any reasonable burst period.

charcircuit 3y ago
I'm skeptical too. Her arms look so stiff so I would exepct her to have been moving slowly.
mdonahoe 3y ago
Do you work at Apple?
Filligree 3y ago
The “burst” is actually a two second video. Plenty of time for that.
- deleted 3y ago
  [deleted]
- threeseed 3y ago
  She said she never used live photo. And the latency for taking a standard photo is not 2 seconds.
  - Filligree 3y ago
    Indeed, since it looks at data from before you press the trigger as well. The camera is always active while you have the camera app; storing a few seconds of pictures is a trivial amount of memory. Live Photo means storing the video in the image file. Turning it off doesn’t mean the video is no longer used, just that it isn’t stored.
threeseed 3y ago
Surprised you were downvoted because this doesn't sound right at all. The poses would be at least a few seconds apart which rules out anything in Apple's computational photography pipeline e.g. focus and dynamic range stacking. At least based on what they have communicated to date. We know Google demonstrated an AI model that was capable of selecting human parts from multiple photos but that was a showcase feature not something quietly added.

PlunderBunny 3y ago

I heard mention on a podcast recently [0] that if you hold down the button in the iPhone camera app it will capture a set of photos and then mark the one that it thinks is the 'best' (based on, for example, the photo where everyone has their eyes open). Not the same as what happened here of course. (I keep forgetting to try this, not least because I always try to get the people out of my photos!) [0] This one, I think: https://podcasts.apple.com/nz/podcast/the-talk-show-with-john-gruber/id528458508?i=1000632890966 https://podcasts.apple.com/nz/podcast/the-talk-show-with-joh...

carterschonwald 3y ago
I think there’s also how the Live Photo feature works. Where taking a photo is literally a short video and it picks the best frame
macintux 3y ago
Unfortunately they changed the behavior a few major releases ago: now holding the button down switches to video recording. Annoying for me, since I don't do video. If you use the software button you can slide to the left for burst, but afaik there's no way to trigger burst photos from the volume buttons. Maybe the new programmable button on the iPhone 15 series.
- kderbe 3y ago
  https://support.apple.com/guide/iphone/take-burst-mode-shots-ipha42c55cd0/ios https://support.apple.com/guide/iphone/take-burst-mode-shots... "Tip: You can also press and hold the volume up button to take Burst shots. Go to Settings > Camera, then turn on Use Volume Up for Burst."
  - macintux 3y ago
    Thanks, I glanced at that page but didn't notice the tip at the bottom.
- dmix 3y ago
  That’s still a cool shortcut I didn’t know about. It not only switches to video but it “gates” the button press so it only records a video for as long as you press the button

emptybits 3y ago

If not already, I expect examples like this will be accumulated and trotted out as part of legal defenses. Valid or not, gross examples like this will probably nudge some judges and juries over their threshold for reasonable doubt. AI manipulations are happening and their bounds can be hard to predict.

ealexhudson 3y ago
Imagine a politician photoed reacting to a member of the public in disgust, except the face was stitched in from moments after. Or, worse, someone captured at the scene of a bomb reacting before the bomb went off? We have an inbuilt set of assumptions about causality that this AI now violates. That's potentially huge in some very specific scenarios...
- fennecbutt 3y ago
  Hey I have a free energy machine to sell you

weird-eye-issue 3y ago

With how much manipulation happens to photos by default it seems like it would be easier to get them thrown out of court as evidence

j16sdiz 3y ago
Without better evidence, the court will keep accepting them. Just look at how bad fingerprints are accepted
- callalex 3y ago
  See also: polygraphs.
  - zimpenfish 3y ago
    See also: bite mark evidence[0], blood splatter[1], etc.[2] [0] https://www.theguardian.com/us-news/2022/apr/28/forensics-bite-mark-junk-science-charles-mccrory-chris-fabricant https://www.theguardian.com/us-news/2022/apr/28/forensics-bi... [1] https://features.propublica.org/blood-spatter/blood-spatter-criminal-justice-courts-forensic-evidence/ https://features.propublica.org/blood-spatter/blood-spatter-... [2] https://www.currentaffairs.org/2023/01/we-need-to-get-junk-science-out-of-courtrooms https://www.currentaffairs.org/2023/01/we-need-to-get-junk-s...
- actionfromafar 3y ago
  I'd say, the courts will keep accepting bad evidence no matter what you do.
joeframbach 3y ago
It's not so farfetched. https://www.theverge.com/2021/11/10/22775580/kyle-rittenhouse-trial-judge-apple-ai-pinch-to-zoom-footage-manipulation-claim https://www.theverge.com/2021/11/10/22775580/kyle-rittenhous...

hnburnsy 3y ago

How long is this burst (feels longer than 'short' in this case)? Does it start before you press the shutter button? Does this post processing apply to RAW? Does Apple document how this post processing works? More importantly, how does one turn it off or is there another camera app that allows one to turn it off?

doctoboggan 3y ago

I don't think this is true, Apple hasn't openly said they do this level of manipulation (although that doesn't mean they don't necessarily), and I don't think the range of motions she would have to go through would be possible within a single capture. Even with "Live Photos" this wouldn't happen.

tiltowait 3y ago
I’m also skeptical. I want it to be true, because it’s pretty wild, but it seems a little too perfect. I suppose people must already be trying to prove or disprove it.
nickelpro 3y ago
Ya the uncritical acceptance of these very non-technical articles on HN is a little disappointing
- dfxm12 3y ago
  I think you have unrealistic expectations for this community.
_maelstrom 3y ago
The article says it was a panorama photo, which makes it much less interesting than live/hdr creating the same effect.

comfysocks 3y ago

My best guess is that there was a brightness gradient across the scene, and this is the result of tone mapping from an EV bracketed burst. This might result in “time delay” instead of the more typical “ghosting” artifacts.

luuurker 3y ago

A modern Google Pixel starts saving frames as soon you open the camera app. When we finally take the picture, it uses some of the older frames to have what's essentially HDR stacking without the delay. I wouldn't be surprised if a similar thing happened here. Different frames, processing picks the best exposure for each part of the picture and you get this effect.

londons_explore 3y ago
It's also why having the camera open and previewing destroys the battery. Some people I know keep the camera open for hours - especially when exploring a new place and wanting to take a photo of a cool thing at a moments notice, and then moan their battery dies at lunchtime.

deleted 3y ago

[deleted]

ISL 3y ago

I hope, but am not certain, that by configuring my Pixel to save both a jpeg and a raw for each image, at least the raw would avoid these shenanigans.

Someone1234 3y ago
Unless the raw is actually a stack of images, it may not. Both Android and iOS are taking multiple exposures and combining them into a single HDR image. This is before it hits the camera app. This has nothing to do with "Live Photos" on iOS to be clear.

seshagiric 3y ago

I am always amazed how people even catch (come across) issues like this...the chances are 1 in a million?

kazinator 3y ago

Changing the positions of limbs is computational, but it's not photography. Photography refers to the capturing of the reflected light to create an image, which then corresponds to how the scene actually was. Bringing out detail in shadows being called "computational photography": that I could swallow.

hgomersall 3y ago
Are colour filters allowed?
- cubefox 3y ago
  The human eye does color temperature correction automatically, and so do digital cameras, basically since they are available. So yes that is okay obviously.
  - hgomersall 3y ago
    Why obviously? There's a collection of discussed filters here: https://www.northlight-images.co.uk/using-effects-lens-filters/ https://www.northlight-images.co.uk/using-effects-lens-filte... Some colour filters are more than just colour correction. At least one of those results in intentional spatial transformation. Is that allowed?
thfuran 3y ago
It's not changing limb positions computationally, it's computationally jamming limb positions recorded at different instants into one 'photograph'.

gnicholas 3y ago

In my family we like to take 'pano-rankenstein' photos, where you do a pano across a person's face as they are rapidly changing expressions in dramatic ways. The results are pretty hilarious, as the phone tries to stitch your face together into one cohesive image.

STELLANOVA 3y ago

Slight off-topic but I still can't believe Apple engineers are not able to fix stupid green dot flare when shooting with direct sunlight in the frame... Super easy to fix computationally but for some reason still there for years...

panarky 3y ago
Pretty cool how the green dot became a green crescent during the last solar eclipse.

rompledorph 3y ago

If it has not already happened, evidence will get thrown out of court because the image no longer represents the reality. Too many filters and other AI improvements.

hnburnsy 3y ago
Some one is writing an episode of law and order with AI photograohy getting defendant off.

causi 3y ago

One is reminded of how Samsung cameras are trained to put the details of the moon over anything that looks vaguely like the moon, but just one, so you can photograph two blurry images of the moon and one of them will look like it was taken by Hubble.

TeMPOraL 3y ago

Tangent, but: > A U.K. comedian and actor named Tessa Coates Why is it always some kind of celebrity that "discovers" stuff like this? Was it luck (yet again), or are those extreme failure modes of computational photography already somewhat known, just too nerdy to report on until they can be attached to a public person?

edent 3y ago
Because the typical Instagram account is only followed by a handful of people. So anything unusual is unlikely to get any traction.
- TeMPOraL 3y ago
  Might be my age showing, but I swear it wasn't like this just a few years ago - this kind of story would genuinely go viral from the first rando posting it on social media ("wtf is my camera broken?") or the second rando that adapted it into a goofy scene/video ("look I'm a vampire now").
  - londons_explore 3y ago
    Social media algorithms have changed. Viral things used to be semi-random (eg. the 6:01pm on sunday bug, which heavily promoted things initially published at 6:01pm on a sunday because thats when some weekly stats aggregation function ran, and the age of the video in seconds was used as a divisor, so you got huge popularity boosts for the first week if you could post in the right second).
    - actionfromafar 3y ago
      Yep. The algorithms have locked onto value stream extraction mode.

Frummy 3y ago

I was in a butterfly house, and closeup photos removed the legs of the butterflies to keep only the wings lol. e: image link https://ibb.co/nwbw5xY https://ibb.co/nwbw5xY

londons_explore 3y ago
That will be the denoising. Legs are thin and can easily be interpreted as noise - especially when using digital zoom and dim light so there are very few pixels and each pixel is very noisy.
- gpas 3y ago
  More like the poor masking algorithm interpreted the legs as background and blurred them.
- cubefox 3y ago
  Denoising is often great, but sometimes it really destroys details. It is basically impossible to capture a snowy day (or heavy rain) with a digital camera. The denoiser will remove 90% of the snowflakes. Analog cameras, by contrast, remove nothing.
  - londons_explore 3y ago
    Interestingly, some phones now use a model of the camera sensor to detect how much noise there 'should' be (based on temperature, age of sensor, test results of the specific sensor from the factory, exposure time and light levels). They then only remove that much noise, hopefully leaving snow. Every other phone just has an algorithm to estimate how much noise is in an image by sampling a few patches - and that will remove snow.
    - cubefox 3y ago
      Interesting. Do you know which phones have this feature?
  - 3y ago

Moldoteck 3y ago

Computational photography is just at it's first steps. Google photos on p8 allow to replace faces. I expect this feature will be implemented to be done automatically to show you the ideal photo. And I expect some parents will be happy with it - having an ideal photo with ideal face of their children

karmakaze 3y ago

'photo' is now synonymous with photoshopped.

karmakaze 3y ago

It wasn't long before we hit this Xerox digits[0] moment in 'photos'. What features do you want in your photocopier or photos? [0] https://www.theverge.com/2013/8/6/4594482/xerox-copiers-randomly-replacing-numbers-in-documents https://www.theverge.com/2013/8/6/4594482/xerox-copiers-rand...

coremoff 3y ago
very different causes though; the xerox one was an image compression bug, IIRC; this is a "chop up multiple photots and stich together the best composite". Still got room for library bugs! (wasn't the Samsung moon pictures sort of more like the xerox one?)

sinuhe69 3y ago

Of course, this is one in a million chance but it highlights very nicely IMO a much bigger issue: what is reality? Some would say only the classical optical camera would capture faithfully our reality. But does it? The reality of the sunlight is a broad spectrum of radio emissions: UV, infrared and more. Does the optical camera capture these? No. Thus, which reality does it capture? Our perceived reality? Other would argue: at least the optical system would capture events in time faithfully. But does it? What would we see in a femto second? Certainly not the pictures we normally see. So the results of an optical system are also super imposed realities, not very much different than the results of a computational photography. There is simply no single one reality, only our perceived realities. But if so, can we still call it reality or it’s merely a product of our sense, our perception and hallucination?

Gigachad 3y ago
Probably something closest to what your eye sees is ideal for most photos. But dumb optical cameras have all kinds of artefacts that eyes don’t. When I slightly bump the camera, the whole image comes out blurry, my eyes don’t do that. Things like lens flares don’t exist either.
- andreicap 3y ago
  > all kinds of artefacts that eyes don’t. Eyes do have lots of artefacts, your brain fills in the gaps, like the blind spot [1] It's not much more different than computational photography, really. [1] https://en.wikipedia.org/wiki/Blind_spot_(vision) https://en.wikipedia.org/wiki/Blind_spot_(vision)
TeMPOraL 3y ago
> There is simply no single one reality, only our perceived realities. This does not follow at all from your earlier paragraph. The reality we're talking about here, which regular photography reflects while computational one doesn't, is the correlation of recorded data with the state of the world. The pixels of a regular photo are highly correlated with reality - they may have been subject to some analog and digital transformation, and of course quantization, but there's a straightforward and reversible (with some loss of fidelity) function mapping pixels to the photographed event. Computational photography, in contrast, decorrelates pixels from reality, and discards source measurements, leaving you with a recording of something that never happened, but is sort of similar to the thing that did. I elaborated on this elsewhere in the thread, so let me instead point at another way of noticing the difference. Photogrammetry is the science and technology of recovering the information about reality from photos, and it works because pixels of regular photos are highly correlated with reality. Apply the same techniques to images made via computational photography, and the degree of uncertainty and fidelity loss will reflect the degree to which the computational photos are AI interpretations/bullshit.

NikkiA 3y ago

If the 'camera' is guessing as to intent, it's no longer a camera, sorry.

Gigachad 3y ago
It isn’t guessing. It’s just stacked multiple shots. Everything in the photo is real, just at slightly different moments.
devnullbrain 3y ago
Every digital photo interpolates or otherwise guesses the unknown data between sensors of each color during debayering.

saiya-jin 3y ago

Next time somebody here will be bashing other manufacturers like say Samsung for clarifying 50x zoom moon shots (like most of HN did when this was a topic for a day or two few months ago), remember kids that this is what all manufacturers do. Or half-reversing photo of kitten in the grass as somebody mentioned.

dusted 3y ago

This happens more and more, I guess it's an unholy mix of "better compression as long as you don't actually look at the image" and "ai improvement".. A few weeks ago, I took a really lovely picture of my son, composition, facial expression, focus, light, it was _PERFECT_ except.. The algorithms in my phone had decided that it'd be better to scrape off his fucking skin and replace it with the texture of the wall behind him! Of course it must be my fault for buying such a cheap phone, it only a Galaxy 22 Ultra, I'm sure the 23 Ultra is better... But it was not out when I changed phones.. Wtf^wtf.. So I go turn on RAW so I can at least salvage picture in the future, except RAW only works in the "pro" camera mode which is inconvenient to use and sometimes it silently falls back to non-pro.. In the end I gave up and installed a third party camera application, I guess I just have to trust Mark, at least he hasn't actively messed up my photos. https://play.google.com/store/apps/details?id=net.sourceforge.opencamera https://play.google.com/store/apps/details?id=net.sourceforg...

fennecbutt 3y ago
I have a Samsung S23U and previously a S22U (and before that one plus) and I've never had that happen to any of my photos (and I take a lot). Hell, I don't think it's using computational photography besides basic denoising/lightening magic for low light (and their stupid moon replacement thing...they already beat Apple just having the 10x lens, no need for the moon bs). I think the default camera app is fine.
- dusted 3y ago
  Here's a photo taken by my S22U, http://dusted.dk/pages/computers/J-11/DEC_J-11_Gallery/full/2.jpg http://dusted.dk/pages/computers/J-11/DEC_J-11_Gallery/full/... that's a cardboard box on top of a matte PC case, but somehow the phone sees this strange blurred pattern in it, as if the phone was moving (which it was not) while at the same time, the label is artificially sharp (almost as if edge detection is taking place), at the same time, you can't see a single fiber of the cardboard box (except for the very edge where it looks like it's grown hair, which it has not).. Have you maybe looked close enough at your photos ?

ClumsyPilot 3y ago

So many comments, not a single one pointing out that someone will go to prison over this 'photo', when it shows something that never actually happened.

PUSH_AX 3y ago
We've been able to create images and video of things that didn't happen for decades.
- Dah00n 3y ago
  This is not the same thing and you know it.

verisimi 3y ago

And, of course, there is the story where Samsung replaces the moon with its high res version. I don't think photography from phone can actually be trusted to be a faithful representation of reality as it is not a purely mechanical series of actions. This will be even worse if ai is involved in the process. I certainly think that there is an argument for photographic evidence to be inadmissible in court.

verytrivial 3y ago

Defense lawyers take note.

tomovo 3y ago

Good luck capturing a sprint finish with that kind of camera.

actionfromafar 3y ago
The photo shows they all ran over the line at the same time!

deleted 3y ago

[deleted]

bmacho 3y ago

This reminds me of printers, scanners and photocopies that have a "compressing" step, and replace numbers with other numbers just for fun. Like you have one job, make the numbers correct, and you fail to do that.

cubefox 3y ago
Yeah. The difference is that people get really angry when they hear scanners are doing this, while in the current case (look at the other comments) most people treat it like a perfectly acceptable curiosity.

pluc 3y ago

It's funny that the iPhone has half a dozen cameras, uses the cameras as the main selling point (look at posters!) yet the image generated is coming essentially from a piece of code.

octacat 3y ago
Code is important too. They sell a good combo.
- pluc 3y ago
  Yeah but I don't see that put forward nearly as much as all the camera tech they keep updating and promoting. I was just in Italy and they have building-sized promotional posters zoomed in on all the cameras at the back of the phone. Agreed it doesn't lend itself to advertising so much because it's less.. visual, but still I thought that was interesting.
  - octacat 3y ago
    marketing rarely correlates with reality. nowadays. uber: there is a car in 8 minutes from uuu! oh, thanks for ordering. now it is 20 minutes (30 in reality). and even if correlates, it is cherrypicked (hi, sony camera specs.)

vachina 3y ago

So can we say iPhone pictures are not admissible in the court of law? Since the photos are more like hand drawings (I.e. subject to interpretation by the algo) than a snapshot of time.

k310 3y ago

May I suggest getting a nice small pocket camera? I have used a lumix with 8X optical zoom (for a very long time) that is smaller than most smartphones (albeit thicker, thanks to the fetish for ultra-thin phones), has a removable SD card, and doesn't do processing beyond what it normally takes to make jpegs. Newer ones might have raw, I haven't checked. It has outlived many phones, has a removable battery, doesn't have a single f/1.8 aperture, and has a macro mode that doesn't require you to be 1/4 inch from the subject. If it's lost or stolen, I haven't handed over all my financial and personal data. What will they think of next?

_maelstrom 3y ago

It's buried in the article, but: "Coates was moving when the photo was taken, so when the shutter was pressed, many differing images were captured in that instant as the camera swept over the scene, since it was a panoramic photo capture." So it's a pano doing normal pano things, but it's just surprising in context. If you have not yet tried it, you can have friends jump in and out of the frame as you slowly move across a scene and have silly photos where they appear more than once! Edit: typo

realclayman 3y ago

This is very possible in panoramic mode. See the update in story: Update December 2, 11:05 Updated with details about the photo being shot in Panoramic mode, which is why the bride-to-be had time to change position between the shots found in the mirror.

fennecbutt 3y ago

And people laughed at Samsung's moon shot thing. I still use my phone camera, but man do I appreciate that all my Sony a7 may do is interpret the sensor using a colour profile and that's it.

classified 3y ago

Well, duh.