28 ms·

iPhone camera app replaces person’s head with a leaf in photo

732 points by davidbarker 5y ago

01acheru 5y ago
Heads turn into leaves, windows turn into Ryan Gosling, the power of image post processing using ML.
fxtentacle 5y ago
That looks to me like they are using deep learning with CNN for denoising. NVIDIA OptiX can produce similar artifacts. However, it appears they forgot to add a loss term to penalize if the source and the denoised result image turn out too different. NVIDIA's denoiser has user-configurable parameters for this trade-off.
- ladberg 5y ago
  I think it would be impossible to train the model in the first place without that loss term.
nla 5y ago
Welcome to computational photography!
dcdc123 5y ago
MKBHD made a relevant video. https://youtube.com/watch?v=VTrCPywCG5A https://youtube.com/watch?v=VTrCPywCG5A
warning26 5y ago
Maybe the person really is leaves, and we're all just blind to the truth
- Lamad123 5y ago
  agree
- ineedasername 5y ago
  Just like the fnords.
dt2m 5y ago
I’ve noticed some sort of image post-processing on the newer iPhones that removes noise and graininess, and instead adds this fake smoothness to all pictures. Haven’t found a way to disable it, save for shooting in RAW, which is impractical due to file size. Really disappointed that this seems to be a forced setting.
- rubatuga 5y ago
  I sometimes use the NightCap app for photos, and it doesn't have that AI bullshit.
- jcun4128 5y ago
  I have a cheaper phone that has this, makes your face look weird, it's too smooth LG Stylo 6 has "AI cam"
- berkut 5y ago
  I've had this (very agressive de-noising I think it is - it's at least almost identical) since I got my iPhone 6S in 2015: basically if you look at 1:1 (i.e. on a computer, as opposed to the small screen of the phone), it almost looks like a watercolour painting, due to how agressive it is. You can pretty much see it in almost all iPhone camera review sample images (and that of phones from other manufacturers). Even in photos taken in direct bright sunlight! I imagine it has an added side 'benefit' (due to the lack of noise/grain) of decreasing the images' sizes after compression.
  - dannyw 5y ago
    That's exactly why I switched to a Pixel.
- deleted 5y ago
  [deleted]
hulitu 5y ago
micheljansen 5y ago
This gives me the same feeling as those ML-powered “enhanced zoom” features: where does the photograph end and the machine made-up fantasy start?
- gsliepen 5y ago
  Obligatory Red Dwarf reference: https://www.youtube.com/watch?v=2aINa6tg3fo https://www.youtube.com/watch?v=2aINa6tg3fo
  - 14 5y ago
    Take my upvote. This has been posted to HN before but I love seeing it again. Red Dwarf was truly an amazing show way ahead of its time imo. For those who have never heard of it, it is a space based comedy I would recommend to all.
    - chris_wot 5y ago
      Definitely the best spoof of Blade Runner I've seen so far.
  - sergiotapia 5y ago
    https://www.youtube.com/watch?v=T5tIDad3xxs https://www.youtube.com/watch?v=T5tIDad3xxs I love these guys so much
- dathinab 5y ago
  "ML-powered" enhanced zoom can't be trusted. The moment you need ML magic to magnify you are basically telling the ML "guess what is there based on what we can see and the training database you where trained on". Also due to the nature of such algorithm they do not have any form of common sense or "realistic general purpose" world view, and tend to be black box so it's hard to check if they learned something "stupid"/"ridiculous". So it's totally possible that somehow it learned to place knifes into the hyper-magnified hands if they have a specific skin color and a turquoise sleeve on a picture with a color scheme as if it's a cloudy day. Or similar arbitrary things. Sure it's supper unlikely but not impossible and there is not good way to find such "bugs". Worse multiple ML systems trained on the same data might share some mistakes and there are just that many huge comprehensive datasets.
Cerium 5y ago
"Cameras" making changes to the image like this make the discussion about the image processing pipeline during the Rittenhouse trial seem a little less bizarre.
- tandymodel100 5y ago
  No, not really
  - kahrl 5y ago
    Yes, yes really. When real resolution is being substituted with the best guess of a completely closed source image processor, the court should be made aware of it.
    - tandymodel100 5y ago
      This sounds like a weird rationalization for an absurd case of technical ignorance. Like when people defended nuking hurricanes or using UV lights as a Covid therapy.
      - CamperBob2 5y ago
        Fact is, you have no idea what kind of unsolicited postprocessing the camera in the Rittenhouse trial might or might not have performed, and neither did the court. It's a huge potential problem, and getting worse by the day.
        deleted 5y ago
        [deleted]
        iszomer 5y ago
        Agreed as we've been living in the era of deep fakes for a while. I shudder to think how computational photography can advance to such a degree to blur the context to any unsuspecting user, whether accidental or intentional.
matsemann 5y ago
I hate how much phones lately alters the images. Of course it most of the time makes the images look better, and cameras are a big selling point on a phone. But I don't like how my photos of people suddenly have a filter applied to the faces, how a picture of leaves during fall have vibrance exaggerated, how the sky looks clearer than it really did.
- akomtu 5y ago
  Big tech has this uni-modal aporoach to users: they find what maximizes a metric, and works for 75% of the users, but roll it out to 100%. Dealing with the remaining 25% would have low ROI.
  - hetspookjee 5y ago
    I never understood this one size fits all approach of a lot of companies. Controlled opposition would often result in higher market penetration and more net happiness
    - dylan604 5y ago
      Mr Smarty Pants has the solution for all cameras to be 100% accurate 100% of the time pleasing 100% of the users. How humble of you to chat with us here on HN. /s You sound like you have not dealt with humans at the scale of the number of smart phone users. I've never dealt with the numbers of something like an iPhone, but over the course of my career have had multiple SKUs totaling over 1 million units. No matter what, there are always "people on the internet" that are dissatisfied enough to go online to voice their opinions. That's the only 100% is that there will always be someone unhappy for whatever reason(s).
  - amelius 5y ago
    Not only tech. Try finding shoes or pants when your size is 1.5 times sigma away from the mean.
    - iamacyborg 5y ago
layer8 5y ago
I wouldn‘t completely exclude the possibility that a random bit flip caused the ML processing to go haywire.
- 1_player 5y ago
  The probability of a bit flip enabling the leaf-replacer logic instead of causing a weird heisenbug and just crashing the camera app is astronomically low.
  - layer8 5y ago
    Just crashing the camera app won’t end up on Twitter and HN though.
  - DSMan195276 5y ago
    Yeah but how many iPhone pictures are taken every day? I'm not saying I'm convinced, but even something with 1/1,000,000,000 odds isn't really out of the question for an action that must happen at least millions of times a day.
    - 1_player 5y ago
      Your probability estimate is off by a dozen orders of magnitude. As a back of the napkin estimate, an iPhone with 8GB has 64 billion bits that can be flipped. One bit flip per week per user would already be extremely bad, as it would mean on average one random crash or case of data corruption per week. If you suppose the bit flip happens in the CPU registers, the number of state changes on a modern CPU is so incredibly huge that a random bit flip doesn't just cause a memory load from a wrong address or a crash is, again, much less probable than 1/1e9 odds, by a lot. And as explained in my other comment, you'd have to show that this behaviour can be expressed in one bit. Is there a bit anywhere in the iOS hardware and software that by changing from 0 to 1 can run a code path that replaces your face with a leaf? I doubt that. I stand by Occam's razor. This happens because it's a bug introduced by its programmers rather than a random gamma ray that turned a bit to 1.
vorpalhex 5y ago
Is it possible to opt in to this feature? I would love it if any unauthorized photos of me had me replaced with background scenery.
- NavinF 5y ago
  Sounds like you need to watch the White Christmas episode of Black Mirror.
hughrr 5y ago
Well it’s in telephoto and 1/121 exposure so the photographer was probably wobbling around like mad when it was taken and the overlay and computational image stuff got confused. I’m fine with this. I use a mini tripod with my 13 pro on telephoto. Back in the old days this would just look like ass instead.
- Dah00n 5y ago
  I would rate any camera that does this as 1/100. It cannot possibly be a worse camera. How anyone that have an interest in photography can see swapping a head with a leaf as just fine is beyond me. It's no better than using Snapchat like filters and calling it raw. It's is utterly broken and should be removed ASAP from all iPhones.
  - hughrr 5y ago
    I’ve taken about 8000 photos from iPhone 6 to 13pro and sifted through them all manually within a day or so of taking them and haven’t seen any anomalous things yet. The error margin is tiny and the benefits to the output are huge. My brain has more trouble accurately portraying things. This is not much of a problem. Reality is really transient and inconsistent anyway.
  - dntrkv 5y ago
    Are you suggesting that if you have a product that performs better than the previous version 99.999999% of the time, but that 0.000001% of the time it performs terribly, in your opinion, the new version is worse?
- jlnthws 5y ago
  Yes that’s the correct explanation: 1/120 speed for 220mm focal length equivalent is far too slow, at least 1 EV. The stabilization is probably not on par with big pro lenses and SLR so you might offset 1 more EV. Now the way the AI is fixing this seems obviously wrong but it’s “best effort”. I’m guessing it has taken several pictures and merged them. This can’t be right 100% of the time. Maybe they should add a “no smart self-similarity merges” option, I would use it most of the time.
ineedasername 5y ago
Maybe iPhones are now making aesthetic decisions? "No, that person's face... Well, let's just cover it with a leaf." It puts the censorship of Renaissance paintings & statues with figleafs over the naughty bits in a new perspective.
- dathinab 5y ago
  Yes they have been doing aesthetic decisions since a while. What started out with "simple" image stabilizations, noise filtering etc. has long become a pipeline of "apply AI magic onto the image which makes it how people think it should look" (instead of how it actually looks). Like making the sky much more bluer then it is. Or edges much sharper then anything such a camera could see (or sometimes especially in combination with digital zoom anything a human with sharp healthy eyes could see). And in case of image stabilization one thing you tend to turn is to take multiple pictures in a row and interpolate. Like some pictures with leafs "besides" the head and some with them behind the head. And then "magic" the head becomes the leaf.
implying 5y ago
This reminds me of Huawei camera app detecting pictures of the moon and superimposing a clear stock photo into your picture: https://www.androidauthority.com/huawei-p30-pro-moon-mode-controversy-978486/ https://www.androidauthority.com/huawei-p30-pro-moon-mode-co...
- Traubenfuchs 5y ago
  That‘s hilarious. The way AI is developing we are turned into children full of magical thinking like „I made a great moonshot with my amazing device!“ when it‘s actually 99% AI handholding.
  - therealdrag0 5y ago
    And I’m going to take thousands of pictures and never look at them.
    - feupan 5y ago
      Don’t worry I can now sync them to music and have other humanoids look at them instead while bored out of their minds.
alkonaut 5y ago
Those of us who have been shooting large digital cameras for the past decade and are some times sad that our photos often come out unsharp in poor light compared to smartphones can at least take some joy in this “no free lunch” demonstration. If this is due to stabilization and not some background blur face detection then it’s probably not something you can (or would want to) disable. Taking a telephoto shot with a tiny sensor in something other than great light (even a heavy overcast is often not enough) will require a lot of software processing. I’m not sure exactly what happened here but I’m pretty sure everyone asking for “unmodified raw photos” to be produced don’t understand what they are asking for. Those “unmodified” photos would be unusable in most cases outside very bright conditions.
- Gigachad 5y ago
  No free lunch, but an incredibly good value lunch. The quality that comes out of phone cameras is remarkable. And the times it messes up are so rare that it becomes a talking point worthy of hundreds of comments.
  - GenerocUsername 5y ago
    You guys are conflating 'adjusting light levels and sharness' with 'replacing whole objects in one area of scene with other objects from the scene' I think this particular big demonstrates a massive gap in what people think photoprocessing is, and what it has very recently become
    - judge2020 5y ago
      Simply adjusting light levels and sharpness wouldn't produce this good/clear of a photo in the conditions presented - AI/ML image post-processing is a hard requirement for sensors like these.
      - creato 5y ago
        The "hard requirement" is just multi-frame noise reduction, which has been around for a lot longer than the AI/ML hype wave, and has always had a risk of producing similar artifacts to this one.
vardump 5y ago
I wonder if this has any implications over iPhone (or cellphones in general) photos in court. This might be brought up to overturn any photo evidence from phones.
- hughrr 5y ago
  All photos regardless of the source are a corruption of reality both semantically and technically speaking. I think they should always be used on a case by case basis with corroborating evidence.
  - iamacyborg 5y ago
    I recall reading a lot of fantastic essays on the “reality” of photos when I studied photography at A Level. I’ll have to see if I can dig stuff up, but I recall there being a lot of controversy surrounding posing of bodies in early war photography.
    - roywiggins 5y ago
      Probably the most famous one: https://en.wikipedia.org/wiki/The_Falling_Soldier https://en.wikipedia.org/wiki/The_Falling_Soldier
      - iamacyborg 5y ago
        That one definitely but I was thinking of Alexander Gardner’s Home of a rebel sharpshooter.
- fknorangesite 5y ago
  ...um: https://appleinsider.com/articles/21/11/11/defense-in-kyle-rittenhouse-trial-argues-apple-ai-manipulates-video-footage https://appleinsider.com/articles/21/11/11/defense-in-kyle-r... Regardless of the validity of the objection in this specific case, yes it's already happening.
- post_break 5y ago
londons_explore 5y ago
The cause of this is image-stacking. The phone takes ~20 frames, over 0.2 seconds. In that time, lots of people and things in the frame move. Optical flow is used to track all moving parts of the image, and then 'undo' any movement, aligning all parts of the image. Then the frames are combined, usually by, for each pixel, taking something like the median or throwing out outliers and using the average. When the optical flow fails to track an object in more than half the frames, the 'outliers' that are thrown out can in fact be the image content you wanted. It happens with leaves a lot because they can flutter fast from one frame to the next, so tracking each individual leaf is hard. A few bad tracking results on more than half the frames, and all you end up seeing is leaves where there should be a face..
- dylan604 5y ago
  Oflow "glitches" are some of the most fun things in image processing, and definitely my favorite render glitches. When it works, it's amazing. When it doesn't, it's also just as amazing, but in a different manner.
deleted 5y ago
[deleted]
- beervirus 5y ago
pryce 5y ago
how do we exclude the possibility that we are just seeing a leaf on its way falling (or blowing) between the subject that the photographer? Logically an event like that would be followed by the iphone not detecting a face, and therefore not applying its usual face-related black-box features? Supposing this is the case and is 'bad', what exactly do we expect 'better behaviour' would mean in this situation?
- function_seven 5y ago
  The leaf that takes the place of the face is still attached to its branch. It's not falling down, it's hanging from the tree. And the person's head is missing in the areas you'd expect to still see it if it was a falling leaf.
  - dathinab 5y ago
    The was taken under zoom on a smartphone known to apply all kinds of "make thinks look nice AI magic". Both of following scenarios is possible: - Due to a combination of various post-processing steps including image stabilization the leaf replaced the head. - The leaf is on a branch which is above the head and was pushed down by the wind hiding the face, due to the unusual angel of the branch that is non-obvious. And the artifacts around the "face" come form the image sharpening/stabilization magic not knowing what to do with that pixels. - The leaf might also be falling and it's connection to the branch can be an optical illusion, that kind of leafs sometimes have a "mini branch part" attached to them when they fall. I would say both is likely, and given that it was supposedly done under high zoom I wouldn't be sure if the human doing the photo can/does see it correctly, because our brain also does "magic" to make the things we see look "better" (like our brain actually fills in details from memory/eperience in some situations).
dqpb 5y ago
[flagged]
avrionov 5y ago
I've been thinking a lot about this recently. The mobile phones photography went in a direction which makes the mobile photography quite questionable. ML algorithms can't be tested completely, there are going to be many corner cases where results like the above will be produced. The algorithms are trying to compensate for the limitations of the small sensors, limited lenses and the hands movement, but we end up getting images that are completely artificial in many cases. Multiple images are combined with data which just made up by the ML models. Better hardware (sensors) could solve some of this I hope.
syntaxing 5y ago
Wouldn’t looking at the RAW give insight on what’s happening here
- chrisseaton 5y ago
  The iPhone Camera app doesn't support RAW except on some specific Pro models, and then you have to turn it on - you can't get RAW retroactively.
xchaotic 5y ago
This is how it begins - SKYNET
beervirus 5y ago
osivertsson 5y ago
I sometimes hang out on mountainbike forums. A user posts that despite a new chain and setting up their rear derailleur according to instructions gear-changes are clunky or the chain skips when applying power on the pedals. The usual reply from the forum members is then to send a photo of the cassette since excessive wear there leads to these problems. This used to clearly show that the user needed to replace their worn cassette too. Lately these photos of cassettes can look all mangled. Like ML tried to “enhance” mountains or the back of an alligator or some other structure from nature on to the cassette. This has to be a problem in all sorts of technical trouble-shooting.
- internet2000 5y ago
  Focusing on the wrong thing? That’s not a ML issue.
- wrboyce 5y ago
  I’d be curious to see this, could you link to some examples? Cheers.
- mc32 5y ago
  They need to offer an option for a "raw" image as they used to be called --which were minimally processed and were not compressed.
  - Razengan 5y ago
    iOS does
    - jakear 5y ago
      How do I get this? Sick of seeing oversaturated landscapes. Edit: why is this downvoted? honest question... I'm on iOS and didn't see anything in settings.
      - mc32 5y ago
        Some models of iPhones the "PROs"[1] support something they call "Pro Raw" which isn't actual RAW images: "Apple ProRAW combines the information of a standard RAW format along with iPhone image processing, which gives you more flexibility when editing the exposure, color, and white balance in your photo." So it's better than JPEG but not as good as RAW. Anyhow apparently Apple supports quasi raw at the system level but doesn't always get exposed to the user. but 3rd part apps can take advantage of this even for non "PRO" Apple telephones. [1]https://support.apple.com/en-us/HT211965 https://support.apple.com/en-us/HT211965
Zak 5y ago
This seems like it might be a case of the phone combining images taken with two or more of its cameras.
bendbro 5y ago
If anyone remembers the infamous "pickles" drama in the Rittenhouse case, this is another example of why it's so important to know exactly what technology is doing to a photograph. An aside, the judge's demonstrated understanding of pixels, despite being slimed by most media, was actually quite good. During the trial he demonstrated understanding of how an algorithm can take as input the original pixels and calculate estimated (or interpolated, whatever you want to call it generically) pixels from that input.
literallyaduck 5y ago
Remember the Rittenhouse trial and the big to do about the video being enhanced when zoomed? Digital evidence and photos are suspicious for the purpose of evidence. Not commenting on the results of the trial. Sidebar not specific to that trial, slow motion, and slowed video shouldn't be shown to jurors because it creates the illusion of having more time to think. Everything in real life happens at one speed, you can't back it up, slow it down, or armchair qb it then decide if an action was appropriate.
- mikewhy 5y ago
  > the Rittenhouse trial and the big to do about the video being enhanced when zoomed I keep thinking about this, and was already when the case was going on. If linear interpolated zoom isn't allowed, no iPhone photo after a certain time should be.
  - wmf 5y ago
    If linear interpolated zoom isn't allowed... Think about analog/optical/chemical photography. In the old days did juries look at camera negatives with a loupe? Of course not, they looked at enlargements. What "algorithm" does an optical enlarger use?
    - tagoregrtst 5y ago
      Continuously linear implemented with an analog device vs digital “linear” (ie, watch your floats! Take care of quanization error! Have you kept the colors separated according to the Beyer pattern?). No one has a problem with a mathematically perfect linear transformation, and film enlargers come very close to that ideal (yes they distort, but in a very obvious way and by degrading detail not adding detail that isn't there) The analog picture is much harder doctor. More gracious artifacts in blow up (grains are random, versus sharp grid). Much more detail is recorded (by virtue of the size of the sensor and therefore the diffraction limit. Sensor resolution is not too useful).
      - sudosysgen 5y ago
RicoElectrico 5y ago
Block matching is a somewhat popular noise removal technique predating neural networks. https://en.wikipedia.org/wiki/Block-matching_and_3D_filtering https://en.wikipedia.org/wiki/Block-matching_and_3D_filterin...
kgen 5y ago
This is so interesting, I how you debug something like this (assuming there is no copy of the original pre-processed image as well)?
iandanforth 5y ago
Three dots instead of four.
rurp 5y ago
I recently got a Pixel and was disappointed to see that the stock camera app has no way to control the focus manually. I guess Google is so impressed with their post-processing they think users won't need any control. Turns out, their software isn't that great at focusing in many of the photos I take, so I need a different camera app. My only requirements are: 1. Manual focus 2. Wide angle lens support I thought a replacement would be trivial to find but I've tried a bunch of apps at this point and haven't found a single one that checks both boxes. Can anyone recommend a basic android camera? Preferably one that doesn't distort the world too much or replace faces with leaves. I've been using Open Camera which is pretty great, but doesn't support wide angle lens shots as far as I can tell.
- TedShiller 5y ago
  > Can anyone recommend a basic android camera? I can't, unfortunately
- Zak 5y ago
  Does Open Camera not show the switch multi-camera button on your phone? From https://opencamera.org.uk/help.html https://opencamera.org.uk/help.html Switch multi-camera icon - This icon only shows on devices with more than one front and/or back cameras, and allows you to switch between those cameras. For example, a device might have two back cameras, one standard and one ultra-wide, this icon will switch between the standard and ultra-wide camera. If Settings/On screen GUI/"Multiple cameras icon" is disabled, then this icon will not show; instead the "Switch camera" icon can by used to cycle through all the cameras. Note that some devices do not allow third party applications to access their multiple cameras, in which case Open Camera isn't able to use them.
  - user_7832 5y ago
    Not the person you were replying to but I also have a Pixel 5 (still on Android 11). I didn't realize it until the previous comment but I also cannot access the wide angle camera. The button simply doesn't exist. Other 3rd party apps like Lightroom also couldn't find the 2nd camera. But a gcam mod finds it unsurprisingly. To answer op's grievance though the wide angle is fixed focus, so they could use gcam for the wide angle and open camera for the regular lens.
chrstphrknwtn 5y ago
At some point the images from phones are no longer photographs… but some kind of photo-illustration. Taking landscape shots with new iPhones creates some very intense over-saturation of skies and some general HDR-like balancing (even with HDR setting turned off). I understand that phone makers want to give people devices that “make the best photos” but it does sometimes feel like the image processing is going to far, and producing a representation of reality that is weirdly unrealistic.
anonnyj 5y ago
My wife's phone produced some odd results. We went out shooting Christmas decorations around town at night... The next day when we were looking at them they appeared very much like daytime photos. WTH
- mr_toad 5y ago
  The phone will take a lot of shots at different exposure levels and combine them. It lets you take shots at night without having lights overexposed and the background underexposed.
danShumway 5y ago
Before we criticize, before jumping to conclusions -- I just want to point out, it is a great leaf. Are we certain that the camera's decision was wrong? I mean, maybe we should consider the camera's judgement about what the best picture is. I'm not a photographer, if my camera tells me that my friend's head is best represented as a combination of sticks, leaves, and other natural objects, who am I to argue with that? I haven't been to photography college. This is a darn good picture of a leaf person.
- lkois 5y ago
  I also don't quite understand the leap to controversy, which seems to rest on the assumption that this person does not have a pile of leaves for a head.
  - kingcharles 5y ago
    Right. To me the photo was set up for the retweets and likes. They got their leaf-headed friend to stand in front of some leaves and took a photo. Probably didn't even use an iPhone. This might have been some Samsung psy-op to discredit the iPhone neural processor.
    - yoav 5y ago
      I think it’s absurd to assume this guy has friends or that there are people with leaves for heads. These are clearly a couple of jackets stuffed with leaves. You don’t see the hands or the other friend except a sleeve. This is the same ploy I did many years ago trying to convince the internet I had friends.
- Centmo 5y ago
  Clearly, the ML algorithm had been given the goal of optimizing the happiness of the user. It had ascertained that the user was quite active on social media and had tied a significant amount of their self-worth to the number of likes received from posts and photos. It had correctly calculated that the user would extract more enjoyment from the photo by way of likes if their friend's face were replaced with an arrangement of foliage. The user clearly did not like that friend very much anyway due to the lack of engagement with their posts. Truly impressive technology.
725686 5y ago
Reminded me of the infamous Xerox copier bug that changed numbers in copies! https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...
- arnaudsm 5y ago
  The fact that a bug this severe and overengineered took so look to be noticed is both hilarious and extremely scary about how we trust modern software.
pvaldes 5y ago
<code for turning more people into sleaves of the iphone starts here...>
anigbrowl 5y ago
Probably because she is Canadian
webkike 5y ago
Kinda looks like a water droplet or something
bargle0 5y ago
How long until your phone won’t let you take pictures of certain things?
deleted 5y ago
[deleted]
nutanc 5y ago
This is now popular enough that the owner should probably create an NFT out of this :)
peterkelly 5y ago
Autotune for photography.
lattice_epochs 5y ago
This just reminds me of the early iPhone panorama stitching.
mark-r 5y ago
The leaf over the face is much larger than the leaves in the background. That leads me to believe it was an actual leaf in the foreground hanging in an inconvenient place.
bitwize 5y ago
My guess is this is some sort of AI driven compression. The compressor presented the neural network with images of the woman's head and some background leaves and said: "Corporate needs you to tell the difference between these two pictures." The neural network happily replied, "They're the same picture". And so the compressor discarded the woman's head from the data and replaced it with the leaves, which have unerringly been found to be equivalent.
root_axis 5y ago
Why does everyone automatically assume this explanation is correct? Based on the photo, it looks to me like a real leaf in the foreground, probably having fallen at the perfect time to create this photo. I would be curious to hear Apple's explanation on this...
- vmception 5y ago
  > Why does everyone automatically assume this explanation is correct? because they want to. thats how the internet works right now, it mirrors your greatest gripe or nightmare, which is exactly when you should have the most skepticism.
  - krapp 5y ago
    Except the leaf isn't so close to the foreground that it completely obscures the person's head, and there is clearly other distortion/artifacting in the image. Something more than that is going on. Useful skepticism actually takes evidence into account, rather than dismissing claims at a glance, or because "thats how the internet works right now."
    - vmception 5y ago
      https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026
      - krapp 5y ago
        Yep, fair enough I have to eat those words. Although to be fair(er), people didn't believe otherwise because, as you insisted, they were buying mindlessly into some internet hype/rage generator "mirroring their latest gripe or nightmare." The original photo did look odd and it generated a lot of interesting discussion. Honestly, it still does to me. While that reasoning was (kind of) correct (the leaf didn't fall at just the right time, it was attached to a branch,) both you and root_axis above assumed people were being driven by stupidity rather than curiosity, which is still an attitude we could use a lot less of.
omk 5y ago
Few things seem to be at play here. 1. iPhone uses face tracking to adjust focus on subject's face 2. Same face detection can end up detecting faces in arbitrary objects. Very common occurring. Here the leaf. 3. iPhone uses multiple lens to compose a single photo. 1 lens focused on the persons face while another with better focus on the face estimated the face to be within the leaves. 4. The composite photo now has picture of leaf replacing the person's face.
yuvalkarmi 5y ago
Plot twist: this person actually has a leaf for a head
dmitshur 5y ago
I had a feeling of recalling something relevant to this topic recently. Turns out it was a video by Marques Brownlee, "Smartphone Cameras vs Reality!". This instance would've been a great example there. The section at t=364 [1] is quite relevant. [1] https://www.youtube.com/watch?v=MZ8giCWDcyE&t=364 https://www.youtube.com/watch?v=MZ8giCWDcyE&t=364
alboy 5y ago
I take it the processing model was trained on a dataset of Magritte’s paintings?
dehrmann 5y ago
Remember that bit in the Kyle Rittenhouse trial about image resizing?
sathishmanohar 5y ago
She is a ghost! Duh.
ronenlh 5y ago
Can this be an argument for plausible deniability in court?
lloydjones 5y ago
I wonder whether there’s a point where photographic evidence can’t reasonably be used in court etc because of this? I’m aware that Photoshop has been around for an age.
gfykvfyxgc 5y ago
Apple would tell you this is actually the photo you wanted to take.
ezconnect 5y ago
This is worse than the Huawei moon scandal.
aimor 5y ago
I had a problem where letting iPhones automatically touch up portraits taken on a dedicated camera would result in black blobs being inserted over the whites of the subject's eyes. That's just to say that these problems are common and to watch out for it on your own photos. When you know it can happen in subtle ways you'll pay more attention and you'll start to see big errors in images you thought were fine at first glance.
netcan 5y ago
When I was 12 I learned to make autocorrect replace my 8 yr old brother's name with "poophead." It was a genius prank. If I worked at Apple today, I'd do the same thing with his 30 year old head.
tomduncalf 5y ago
I had something like this happen the either week - someone took a photo of a group of us against a wooden wall in low light with the front camera of the iPhone 13 Pro Max, and while the photo looked great zoomed out, if you looked closely it had applied the vertical line and texture of the wooden slats to my face, so I had a sort of wooden face! My assumption is that this is over enthusiastic ML processing trying to enhance photos in challenging conditions… makes me think of e.g. https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-added-ryan-goslings-face-to-this-photo/ https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a... Similarly, I’ve noticed that the iPhone 13 Pro tends to use the wide angle lens for 3x zoom photos in low light then upscale it. This makes some sense as it can capture more light and the photos look good zoomed out, but if you zoom in, you’ll see things like text and small faces have been replaced with random blurry “ML-looking” smudges. If you want to avoid this, right now I think the only way to do so is to use a third party camera app. I use Halide and it’s excellent, it will only use the lens you have selected even if it is in challenging lighting conditions, and it applies much less processing (and you can further disable more processing if you like, I’ve not had the need). The default camera is generally great for snapshots that you aren’t going to zoom into etc. though. Would be nice if they could tweak it to be a bit less aggressive in its processing, perhaps adding a “pro” mode or similar.
- albert_e 5y ago
  I worry that smartphone photos will become inadmissible in court as evidence ... and will also give all shady characters like politicians a cover to change narratives
  - retube 5y ago
    I think we're already past that point....
  - rowanG077 5y ago
    I hope that smartphones photos become inadmissable. Because photos taken don't reflect what really happened. I also hope smartphones start storing unprocessed version of every photo that will be admissable in court.
busia 5y ago
A.I. is watching you
rbrbr 5y ago
space_rock 5y ago
So now we can't trust our photos to be fake? Let's get rid of the ML. Optional post processing only. This is why an iPhone photo won't be able to be used in court
aloer 5y ago
I upgraded from an iPhone X to a 13 pro when it was released and since then it’s been a very mixed experience. The range of quality is incredibly frustrating. I enjoy the speed with which I can take pictures that are not blurry. But in most cases I wish I could reduce the “AI aggressiveness” since they all look so artificially sharp. But that’s only part of it. I have pictures of my family this Christmas where we posed in the same location, same light and with just a few seconds apart and the pictures look completely different when it comes to the colors. “Different AI interpretation” I tell those around me that haven’t come in contact with computational photography yet. Then I apologize, saying that I can’t know how the picture will turn out before it’s taken. And that there is no way to reanalyze. No way to feed RAW back to “the AI”. But such is the future… There’s a picture of me and my partner against a perfectly blue sky. I am a few cm behind her. She looks normal while my face looks half transparent / whitened like a ghost in the sky. There’s pictures of text and logos on billboard and such that are so smooth as if they are photoshopped on top. I often tell those people that not only do we need to accept that photos now are not the same concept as photos in the past, we also have to accept that the next iOS update could change the algorithm and photos from this Christmas will be different to photos from next Christmas even when shot with the same camera. Frustrating
emerged 5y ago
Well to be fair, its incredible how far head to leaf technology has come.
jinto36 5y ago
Here's a link to the image in question directly: https://pbs.twimg.com/media/FH0N9HNWQAE0n9R?format=jpg&name=large https://pbs.twimg.com/media/FH0N9HNWQAE0n9R?format=jpg&name=...
Jenda_ 5y ago
So… when are we going to see made up numbers in photographed documents again? https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... Superimposing other's people face onto a photo from a crime scene? Or, as part of my work, I photograph electric switchboards for documentation and diagnostic purposes. Hopefully I won't get any hallucinated connections, right?
GoToRO 5y ago
Impossible to take photo of stars on iphone. Whatever sw they use, it will create random stars in the photo. Like instead of 1 star you get ten.
mitchcohentoo 5y ago
Mystery solved! https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026
cdaringe 5y ago
Man takes photo of leaf, surprised to see leaf in photo
rrauenza 5y ago
There was an update -- sounds like a leaf was really in the shot: > Big news! I sent @sdw the original image. He theorized a leaf from a foreground tree obscured the face. I didn’t think anything was in view, and the closest tree is a Japanese Maple (smaller leaves). But he’s right! Here’s a video I just shot, showing the parallax. Wow!
- tqi 5y ago
  This thread has been an amazing example of confirmation bias.
- darkwizard42 5y ago
  LOL amazing that the explanation was so simple!
- roryokane 5y ago
  Link to that tweet: https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026. Nitter proxy: https://nitter.net/mitchcohen/status/1476951534160257026 https://nitter.net/mitchcohen/status/1476951534160257026
- agar 5y ago
  I wish this could be pinned to the top. Perhaps a follow-up post to HN would be worthwhile. So many people will leave this thread incorrectly believing this was an iPhone problem. It could easily become the data point people use to cast doubt on all pictures they don't like.
- davidbarker 5y ago
  I'm the OP (of the HN submission — not the tweet). Really interesting discussions. I've upvoted this — hopefully it makes it to the top. Here's the link to the HN discussion on the update: https://news.ycombinator.com/item?id=29750660 https://news.ycombinator.com/item?id=29750660
- ffggvv 5y ago
  even if its a foreground tree it still looks blurry and strange doesn't it?
resters 5y ago
It was actually a leaf from a foreground tree, not an image processing artifact: https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026
bereasonable 5y ago
He posted an update. It was a leaf on a tree in front of the woman and it obscured her face. https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026
davidhariri 5y ago
This title should be updated. The author has since figured out that it was in fact a leaf obstructing the person’s face.

732 points by davidbarker 5y ago

Heads turn into leaves, windows turn into Ryan Gosling, the power of image post processing using ML.

That looks to me like they are using deep learning with CNN for denoising. NVIDIA OptiX can produce similar artifacts. However, it appears they forgot to add a loss term to penalize if the source and the denoised result image turn out too different. NVIDIA's denoiser has user-configurable parameters for this trade-off.

ladberg 5y ago
I think it would be impossible to train the model in the first place without that loss term.

nla 5y ago

Welcome to computational photography!

dcdc123 5y ago

MKBHD made a relevant video. https://youtube.com/watch?v=VTrCPywCG5A https://youtube.com/watch?v=VTrCPywCG5A

warning26 5y ago

Maybe the person really is leaves, and we're all just blind to the truth

Lamad123 5y ago
agree
ineedasername 5y ago
Just like the fnords.

dt2m 5y ago

I’ve noticed some sort of image post-processing on the newer iPhones that removes noise and graininess, and instead adds this fake smoothness to all pictures. Haven’t found a way to disable it, save for shooting in RAW, which is impractical due to file size. Really disappointed that this seems to be a forced setting.

rubatuga 5y ago
I sometimes use the NightCap app for photos, and it doesn't have that AI bullshit.
jcun4128 5y ago
I have a cheaper phone that has this, makes your face look weird, it's too smooth LG Stylo 6 has "AI cam"
berkut 5y ago
I've had this (very agressive de-noising I think it is - it's at least almost identical) since I got my iPhone 6S in 2015: basically if you look at 1:1 (i.e. on a computer, as opposed to the small screen of the phone), it almost looks like a watercolour painting, due to how agressive it is. You can pretty much see it in almost all iPhone camera review sample images (and that of phones from other manufacturers). Even in photos taken in direct bright sunlight! I imagine it has an added side 'benefit' (due to the lack of noise/grain) of decreasing the images' sizes after compression.
- dannyw 5y ago
  That's exactly why I switched to a Pixel.
deleted 5y ago
[deleted]

hulitu 5y ago

micheljansen 5y ago

This gives me the same feeling as those ML-powered “enhanced zoom” features: where does the photograph end and the machine made-up fantasy start?

gsliepen 5y ago
Obligatory Red Dwarf reference: https://www.youtube.com/watch?v=2aINa6tg3fo https://www.youtube.com/watch?v=2aINa6tg3fo
- 14 5y ago
  Take my upvote. This has been posted to HN before but I love seeing it again. Red Dwarf was truly an amazing show way ahead of its time imo. For those who have never heard of it, it is a space based comedy I would recommend to all.
  - chris_wot 5y ago
    Definitely the best spoof of Blade Runner I've seen so far.
- sergiotapia 5y ago
  https://www.youtube.com/watch?v=T5tIDad3xxs https://www.youtube.com/watch?v=T5tIDad3xxs I love these guys so much
dathinab 5y ago
"ML-powered" enhanced zoom can't be trusted. The moment you need ML magic to magnify you are basically telling the ML "guess what is there based on what we can see and the training database you where trained on". Also due to the nature of such algorithm they do not have any form of common sense or "realistic general purpose" world view, and tend to be black box so it's hard to check if they learned something "stupid"/"ridiculous". So it's totally possible that somehow it learned to place knifes into the hyper-magnified hands if they have a specific skin color and a turquoise sleeve on a picture with a color scheme as if it's a cloudy day. Or similar arbitrary things. Sure it's supper unlikely but not impossible and there is not good way to find such "bugs". Worse multiple ML systems trained on the same data might share some mistakes and there are just that many huge comprehensive datasets.

Cerium 5y ago

"Cameras" making changes to the image like this make the discussion about the image processing pipeline during the Rittenhouse trial seem a little less bizarre.

tandymodel100 5y ago
No, not really
- kahrl 5y ago
  Yes, yes really. When real resolution is being substituted with the best guess of a completely closed source image processor, the court should be made aware of it.
  - tandymodel100 5y ago
    This sounds like a weird rationalization for an absurd case of technical ignorance. Like when people defended nuking hurricanes or using UV lights as a Covid therapy.
    - CamperBob2 5y ago
      Fact is, you have no idea what kind of unsolicited postprocessing the camera in the Rittenhouse trial might or might not have performed, and neither did the court. It's a huge potential problem, and getting worse by the day.
      - deleted 5y ago
        [deleted]
      - iszomer 5y ago
        Agreed as we've been living in the era of deep fakes for a while. I shudder to think how computational photography can advance to such a degree to blur the context to any unsuspecting user, whether accidental or intentional.

matsemann 5y ago

I hate how much phones lately alters the images. Of course it most of the time makes the images look better, and cameras are a big selling point on a phone. But I don't like how my photos of people suddenly have a filter applied to the faces, how a picture of leaves during fall have vibrance exaggerated, how the sky looks clearer than it really did.

akomtu 5y ago
Big tech has this uni-modal aporoach to users: they find what maximizes a metric, and works for 75% of the users, but roll it out to 100%. Dealing with the remaining 25% would have low ROI.
- hetspookjee 5y ago
  I never understood this one size fits all approach of a lot of companies. Controlled opposition would often result in higher market penetration and more net happiness
  - dylan604 5y ago
    Mr Smarty Pants has the solution for all cameras to be 100% accurate 100% of the time pleasing 100% of the users. How humble of you to chat with us here on HN. /s You sound like you have not dealt with humans at the scale of the number of smart phone users. I've never dealt with the numbers of something like an iPhone, but over the course of my career have had multiple SKUs totaling over 1 million units. No matter what, there are always "people on the internet" that are dissatisfied enough to go online to voice their opinions. That's the only 100% is that there will always be someone unhappy for whatever reason(s).
- amelius 5y ago
  Not only tech. Try finding shoes or pants when your size is 1.5 times sigma away from the mean.
  - iamacyborg 5y ago

layer8 5y ago

I wouldn‘t completely exclude the possibility that a random bit flip caused the ML processing to go haywire.

1_player 5y ago
The probability of a bit flip enabling the leaf-replacer logic instead of causing a weird heisenbug and just crashing the camera app is astronomically low.
- layer8 5y ago
  Just crashing the camera app won’t end up on Twitter and HN though.
- DSMan195276 5y ago
  Yeah but how many iPhone pictures are taken every day? I'm not saying I'm convinced, but even something with 1/1,000,000,000 odds isn't really out of the question for an action that must happen at least millions of times a day.
  - 1_player 5y ago
    Your probability estimate is off by a dozen orders of magnitude. As a back of the napkin estimate, an iPhone with 8GB has 64 billion bits that can be flipped. One bit flip per week per user would already be extremely bad, as it would mean on average one random crash or case of data corruption per week. If you suppose the bit flip happens in the CPU registers, the number of state changes on a modern CPU is so incredibly huge that a random bit flip doesn't just cause a memory load from a wrong address or a crash is, again, much less probable than 1/1e9 odds, by a lot. And as explained in my other comment, you'd have to show that this behaviour can be expressed in one bit. Is there a bit anywhere in the iOS hardware and software that by changing from 0 to 1 can run a code path that replaces your face with a leaf? I doubt that. I stand by Occam's razor. This happens because it's a bug introduced by its programmers rather than a random gamma ray that turned a bit to 1.

vorpalhex 5y ago

Is it possible to opt in to this feature? I would love it if any unauthorized photos of me had me replaced with background scenery.

NavinF 5y ago
Sounds like you need to watch the White Christmas episode of Black Mirror.

hughrr 5y ago

Well it’s in telephoto and 1/121 exposure so the photographer was probably wobbling around like mad when it was taken and the overlay and computational image stuff got confused. I’m fine with this. I use a mini tripod with my 13 pro on telephoto. Back in the old days this would just look like ass instead.

Dah00n 5y ago
I would rate any camera that does this as 1/100. It cannot possibly be a worse camera. How anyone that have an interest in photography can see swapping a head with a leaf as just fine is beyond me. It's no better than using Snapchat like filters and calling it raw. It's is utterly broken and should be removed ASAP from all iPhones.
- hughrr 5y ago
  I’ve taken about 8000 photos from iPhone 6 to 13pro and sifted through them all manually within a day or so of taking them and haven’t seen any anomalous things yet. The error margin is tiny and the benefits to the output are huge. My brain has more trouble accurately portraying things. This is not much of a problem. Reality is really transient and inconsistent anyway.
- dntrkv 5y ago
  Are you suggesting that if you have a product that performs better than the previous version 99.999999% of the time, but that 0.000001% of the time it performs terribly, in your opinion, the new version is worse?
jlnthws 5y ago
Yes that’s the correct explanation: 1/120 speed for 220mm focal length equivalent is far too slow, at least 1 EV. The stabilization is probably not on par with big pro lenses and SLR so you might offset 1 more EV. Now the way the AI is fixing this seems obviously wrong but it’s “best effort”. I’m guessing it has taken several pictures and merged them. This can’t be right 100% of the time. Maybe they should add a “no smart self-similarity merges” option, I would use it most of the time.

ineedasername 5y ago

Maybe iPhones are now making aesthetic decisions? "No, that person's face... Well, let's just cover it with a leaf." It puts the censorship of Renaissance paintings & statues with figleafs over the naughty bits in a new perspective.

dathinab 5y ago
Yes they have been doing aesthetic decisions since a while. What started out with "simple" image stabilizations, noise filtering etc. has long become a pipeline of "apply AI magic onto the image which makes it how people think it should look" (instead of how it actually looks). Like making the sky much more bluer then it is. Or edges much sharper then anything such a camera could see (or sometimes especially in combination with digital zoom anything a human with sharp healthy eyes could see). And in case of image stabilization one thing you tend to turn is to take multiple pictures in a row and interpolate. Like some pictures with leafs "besides" the head and some with them behind the head. And then "magic" the head becomes the leaf.

implying 5y ago

This reminds me of Huawei camera app detecting pictures of the moon and superimposing a clear stock photo into your picture: https://www.androidauthority.com/huawei-p30-pro-moon-mode-controversy-978486/ https://www.androidauthority.com/huawei-p30-pro-moon-mode-co...

Traubenfuchs 5y ago
That‘s hilarious. The way AI is developing we are turned into children full of magical thinking like „I made a great moonshot with my amazing device!“ when it‘s actually 99% AI handholding.
- therealdrag0 5y ago
  And I’m going to take thousands of pictures and never look at them.
  - feupan 5y ago
    Don’t worry I can now sync them to music and have other humanoids look at them instead while bored out of their minds.

alkonaut 5y ago

Those of us who have been shooting large digital cameras for the past decade and are some times sad that our photos often come out unsharp in poor light compared to smartphones can at least take some joy in this “no free lunch” demonstration. If this is due to stabilization and not some background blur face detection then it’s probably not something you can (or would want to) disable. Taking a telephoto shot with a tiny sensor in something other than great light (even a heavy overcast is often not enough) will require a lot of software processing. I’m not sure exactly what happened here but I’m pretty sure everyone asking for “unmodified raw photos” to be produced don’t understand what they are asking for. Those “unmodified” photos would be unusable in most cases outside very bright conditions.

Gigachad 5y ago
No free lunch, but an incredibly good value lunch. The quality that comes out of phone cameras is remarkable. And the times it messes up are so rare that it becomes a talking point worthy of hundreds of comments.
- GenerocUsername 5y ago
  You guys are conflating 'adjusting light levels and sharness' with 'replacing whole objects in one area of scene with other objects from the scene' I think this particular big demonstrates a massive gap in what people think photoprocessing is, and what it has very recently become
  - judge2020 5y ago
    Simply adjusting light levels and sharpness wouldn't produce this good/clear of a photo in the conditions presented - AI/ML image post-processing is a hard requirement for sensors like these.
    - creato 5y ago
      The "hard requirement" is just multi-frame noise reduction, which has been around for a lot longer than the AI/ML hype wave, and has always had a risk of producing similar artifacts to this one.

vardump 5y ago

I wonder if this has any implications over iPhone (or cellphones in general) photos in court. This might be brought up to overturn any photo evidence from phones.

hughrr 5y ago
All photos regardless of the source are a corruption of reality both semantically and technically speaking. I think they should always be used on a case by case basis with corroborating evidence.
- iamacyborg 5y ago
  I recall reading a lot of fantastic essays on the “reality” of photos when I studied photography at A Level. I’ll have to see if I can dig stuff up, but I recall there being a lot of controversy surrounding posing of bodies in early war photography.
  - roywiggins 5y ago
    Probably the most famous one: https://en.wikipedia.org/wiki/The_Falling_Soldier https://en.wikipedia.org/wiki/The_Falling_Soldier
    - iamacyborg 5y ago
      That one definitely but I was thinking of Alexander Gardner’s Home of a rebel sharpshooter.
fknorangesite 5y ago
...um: https://appleinsider.com/articles/21/11/11/defense-in-kyle-rittenhouse-trial-argues-apple-ai-manipulates-video-footage https://appleinsider.com/articles/21/11/11/defense-in-kyle-r... Regardless of the validity of the objection in this specific case, yes it's already happening.
post_break 5y ago

londons_explore 5y ago

The cause of this is image-stacking. The phone takes ~20 frames, over 0.2 seconds. In that time, lots of people and things in the frame move. Optical flow is used to track all moving parts of the image, and then 'undo' any movement, aligning all parts of the image. Then the frames are combined, usually by, for each pixel, taking something like the median or throwing out outliers and using the average. When the optical flow fails to track an object in more than half the frames, the 'outliers' that are thrown out can in fact be the image content you wanted. It happens with leaves a lot because they can flutter fast from one frame to the next, so tracking each individual leaf is hard. A few bad tracking results on more than half the frames, and all you end up seeing is leaves where there should be a face..

dylan604 5y ago
Oflow "glitches" are some of the most fun things in image processing, and definitely my favorite render glitches. When it works, it's amazing. When it doesn't, it's also just as amazing, but in a different manner.

deleted 5y ago

[deleted]

beervirus 5y ago

pryce 5y ago

how do we exclude the possibility that we are just seeing a leaf on its way falling (or blowing) between the subject that the photographer? Logically an event like that would be followed by the iphone not detecting a face, and therefore not applying its usual face-related black-box features? Supposing this is the case and is 'bad', what exactly do we expect 'better behaviour' would mean in this situation?

function_seven 5y ago
The leaf that takes the place of the face is still attached to its branch. It's not falling down, it's hanging from the tree. And the person's head is missing in the areas you'd expect to still see it if it was a falling leaf.
- dathinab 5y ago
  The was taken under zoom on a smartphone known to apply all kinds of "make thinks look nice AI magic". Both of following scenarios is possible: - Due to a combination of various post-processing steps including image stabilization the leaf replaced the head. - The leaf is on a branch which is above the head and was pushed down by the wind hiding the face, due to the unusual angel of the branch that is non-obvious. And the artifacts around the "face" come form the image sharpening/stabilization magic not knowing what to do with that pixels. - The leaf might also be falling and it's connection to the branch can be an optical illusion, that kind of leafs sometimes have a "mini branch part" attached to them when they fall. I would say both is likely, and given that it was supposedly done under high zoom I wouldn't be sure if the human doing the photo can/does see it correctly, because our brain also does "magic" to make the things we see look "better" (like our brain actually fills in details from memory/eperience in some situations).

dqpb 5y ago

[flagged]

avrionov 5y ago

I've been thinking a lot about this recently. The mobile phones photography went in a direction which makes the mobile photography quite questionable. ML algorithms can't be tested completely, there are going to be many corner cases where results like the above will be produced. The algorithms are trying to compensate for the limitations of the small sensors, limited lenses and the hands movement, but we end up getting images that are completely artificial in many cases. Multiple images are combined with data which just made up by the ML models. Better hardware (sensors) could solve some of this I hope.

syntaxing 5y ago

Wouldn’t looking at the RAW give insight on what’s happening here

chrisseaton 5y ago
The iPhone Camera app doesn't support RAW except on some specific Pro models, and then you have to turn it on - you can't get RAW retroactively.

xchaotic 5y ago

This is how it begins - SKYNET

beervirus 5y ago

osivertsson 5y ago

I sometimes hang out on mountainbike forums. A user posts that despite a new chain and setting up their rear derailleur according to instructions gear-changes are clunky or the chain skips when applying power on the pedals. The usual reply from the forum members is then to send a photo of the cassette since excessive wear there leads to these problems. This used to clearly show that the user needed to replace their worn cassette too. Lately these photos of cassettes can look all mangled. Like ML tried to “enhance” mountains or the back of an alligator or some other structure from nature on to the cassette. This has to be a problem in all sorts of technical trouble-shooting.

internet2000 5y ago
Focusing on the wrong thing? That’s not a ML issue.
wrboyce 5y ago
I’d be curious to see this, could you link to some examples? Cheers.
mc32 5y ago
They need to offer an option for a "raw" image as they used to be called --which were minimally processed and were not compressed.
- Razengan 5y ago
  iOS does
  - jakear 5y ago
    How do I get this? Sick of seeing oversaturated landscapes. Edit: why is this downvoted? honest question... I'm on iOS and didn't see anything in settings.
    - mc32 5y ago
      Some models of iPhones the "PROs"[1] support something they call "Pro Raw" which isn't actual RAW images: "Apple ProRAW combines the information of a standard RAW format along with iPhone image processing, which gives you more flexibility when editing the exposure, color, and white balance in your photo." So it's better than JPEG but not as good as RAW. Anyhow apparently Apple supports quasi raw at the system level but doesn't always get exposed to the user. but 3rd part apps can take advantage of this even for non "PRO" Apple telephones. [1]https://support.apple.com/en-us/HT211965 https://support.apple.com/en-us/HT211965

Zak 5y ago

This seems like it might be a case of the phone combining images taken with two or more of its cameras.

bendbro 5y ago

If anyone remembers the infamous "pickles" drama in the Rittenhouse case, this is another example of why it's so important to know exactly what technology is doing to a photograph. An aside, the judge's demonstrated understanding of pixels, despite being slimed by most media, was actually quite good. During the trial he demonstrated understanding of how an algorithm can take as input the original pixels and calculate estimated (or interpolated, whatever you want to call it generically) pixels from that input.

literallyaduck 5y ago

Remember the Rittenhouse trial and the big to do about the video being enhanced when zoomed? Digital evidence and photos are suspicious for the purpose of evidence. Not commenting on the results of the trial. Sidebar not specific to that trial, slow motion, and slowed video shouldn't be shown to jurors because it creates the illusion of having more time to think. Everything in real life happens at one speed, you can't back it up, slow it down, or armchair qb it then decide if an action was appropriate.

mikewhy 5y ago
> the Rittenhouse trial and the big to do about the video being enhanced when zoomed I keep thinking about this, and was already when the case was going on. If linear interpolated zoom isn't allowed, no iPhone photo after a certain time should be.
- wmf 5y ago
  If linear interpolated zoom isn't allowed... Think about analog/optical/chemical photography. In the old days did juries look at camera negatives with a loupe? Of course not, they looked at enlargements. What "algorithm" does an optical enlarger use?
  - tagoregrtst 5y ago
    Continuously linear implemented with an analog device vs digital “linear” (ie, watch your floats! Take care of quanization error! Have you kept the colors separated according to the Beyer pattern?). No one has a problem with a mathematically perfect linear transformation, and film enlargers come very close to that ideal (yes they distort, but in a very obvious way and by degrading detail not adding detail that isn't there) The analog picture is much harder doctor. More gracious artifacts in blow up (grains are random, versus sharp grid). Much more detail is recorded (by virtue of the size of the sensor and therefore the diffraction limit. Sensor resolution is not too useful).
    - sudosysgen 5y ago

RicoElectrico 5y ago

Block matching is a somewhat popular noise removal technique predating neural networks. https://en.wikipedia.org/wiki/Block-matching_and_3D_filtering https://en.wikipedia.org/wiki/Block-matching_and_3D_filterin...

kgen 5y ago

This is so interesting, I how you debug something like this (assuming there is no copy of the original pre-processed image as well)?

iandanforth 5y ago

Three dots instead of four.

rurp 5y ago

I recently got a Pixel and was disappointed to see that the stock camera app has no way to control the focus manually. I guess Google is so impressed with their post-processing they think users won't need any control. Turns out, their software isn't that great at focusing in many of the photos I take, so I need a different camera app. My only requirements are: 1. Manual focus 2. Wide angle lens support I thought a replacement would be trivial to find but I've tried a bunch of apps at this point and haven't found a single one that checks both boxes. Can anyone recommend a basic android camera? Preferably one that doesn't distort the world too much or replace faces with leaves. I've been using Open Camera which is pretty great, but doesn't support wide angle lens shots as far as I can tell.

TedShiller 5y ago
> Can anyone recommend a basic android camera? I can't, unfortunately
Zak 5y ago
Does Open Camera not show the switch multi-camera button on your phone? From https://opencamera.org.uk/help.html https://opencamera.org.uk/help.html Switch multi-camera icon - This icon only shows on devices with more than one front and/or back cameras, and allows you to switch between those cameras. For example, a device might have two back cameras, one standard and one ultra-wide, this icon will switch between the standard and ultra-wide camera. If Settings/On screen GUI/"Multiple cameras icon" is disabled, then this icon will not show; instead the "Switch camera" icon can by used to cycle through all the cameras. Note that some devices do not allow third party applications to access their multiple cameras, in which case Open Camera isn't able to use them.
- user_7832 5y ago
  Not the person you were replying to but I also have a Pixel 5 (still on Android 11). I didn't realize it until the previous comment but I also cannot access the wide angle camera. The button simply doesn't exist. Other 3rd party apps like Lightroom also couldn't find the 2nd camera. But a gcam mod finds it unsurprisingly. To answer op's grievance though the wide angle is fixed focus, so they could use gcam for the wide angle and open camera for the regular lens.

chrstphrknwtn 5y ago

At some point the images from phones are no longer photographs… but some kind of photo-illustration. Taking landscape shots with new iPhones creates some very intense over-saturation of skies and some general HDR-like balancing (even with HDR setting turned off). I understand that phone makers want to give people devices that “make the best photos” but it does sometimes feel like the image processing is going to far, and producing a representation of reality that is weirdly unrealistic.

anonnyj 5y ago

My wife's phone produced some odd results. We went out shooting Christmas decorations around town at night... The next day when we were looking at them they appeared very much like daytime photos. WTH

mr_toad 5y ago
The phone will take a lot of shots at different exposure levels and combine them. It lets you take shots at night without having lights overexposed and the background underexposed.

danShumway 5y ago

Before we criticize, before jumping to conclusions -- I just want to point out, it is a great leaf. Are we certain that the camera's decision was wrong? I mean, maybe we should consider the camera's judgement about what the best picture is. I'm not a photographer, if my camera tells me that my friend's head is best represented as a combination of sticks, leaves, and other natural objects, who am I to argue with that? I haven't been to photography college. This is a darn good picture of a leaf person.

lkois 5y ago
I also don't quite understand the leap to controversy, which seems to rest on the assumption that this person does not have a pile of leaves for a head.
- kingcharles 5y ago
  Right. To me the photo was set up for the retweets and likes. They got their leaf-headed friend to stand in front of some leaves and took a photo. Probably didn't even use an iPhone. This might have been some Samsung psy-op to discredit the iPhone neural processor.
  - yoav 5y ago
    I think it’s absurd to assume this guy has friends or that there are people with leaves for heads. These are clearly a couple of jackets stuffed with leaves. You don’t see the hands or the other friend except a sleeve. This is the same ploy I did many years ago trying to convince the internet I had friends.
Centmo 5y ago
Clearly, the ML algorithm had been given the goal of optimizing the happiness of the user. It had ascertained that the user was quite active on social media and had tied a significant amount of their self-worth to the number of likes received from posts and photos. It had correctly calculated that the user would extract more enjoyment from the photo by way of likes if their friend's face were replaced with an arrangement of foliage. The user clearly did not like that friend very much anyway due to the lack of engagement with their posts. Truly impressive technology.

725686 5y ago

Reminded me of the infamous Xerox copier bug that changed numbers in copies! https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...

arnaudsm 5y ago
The fact that a bug this severe and overengineered took so look to be noticed is both hilarious and extremely scary about how we trust modern software.

pvaldes 5y ago

anigbrowl 5y ago

Probably because she is Canadian

webkike 5y ago

Kinda looks like a water droplet or something

bargle0 5y ago

How long until your phone won’t let you take pictures of certain things?

deleted 5y ago

[deleted]

nutanc 5y ago

This is now popular enough that the owner should probably create an NFT out of this :)

peterkelly 5y ago

Autotune for photography.

lattice_epochs 5y ago

This just reminds me of the early iPhone panorama stitching.

mark-r 5y ago

The leaf over the face is much larger than the leaves in the background. That leads me to believe it was an actual leaf in the foreground hanging in an inconvenient place.

bitwize 5y ago

My guess is this is some sort of AI driven compression. The compressor presented the neural network with images of the woman's head and some background leaves and said: "Corporate needs you to tell the difference between these two pictures." The neural network happily replied, "They're the same picture". And so the compressor discarded the woman's head from the data and replaced it with the leaves, which have unerringly been found to be equivalent.

root_axis 5y ago

Why does everyone automatically assume this explanation is correct? Based on the photo, it looks to me like a real leaf in the foreground, probably having fallen at the perfect time to create this photo. I would be curious to hear Apple's explanation on this...

vmception 5y ago
> Why does everyone automatically assume this explanation is correct? because they want to. thats how the internet works right now, it mirrors your greatest gripe or nightmare, which is exactly when you should have the most skepticism.
- krapp 5y ago
  Except the leaf isn't so close to the foreground that it completely obscures the person's head, and there is clearly other distortion/artifacting in the image. Something more than that is going on. Useful skepticism actually takes evidence into account, rather than dismissing claims at a glance, or because "thats how the internet works right now."
  - vmception 5y ago
    https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026
    - krapp 5y ago
      Yep, fair enough I have to eat those words. Although to be fair(er), people didn't believe otherwise because, as you insisted, they were buying mindlessly into some internet hype/rage generator "mirroring their latest gripe or nightmare." The original photo did look odd and it generated a lot of interesting discussion. Honestly, it still does to me. While that reasoning was (kind of) correct (the leaf didn't fall at just the right time, it was attached to a branch,) both you and root_axis above assumed people were being driven by stupidity rather than curiosity, which is still an attitude we could use a lot less of.

omk 5y ago

Few things seem to be at play here. 1. iPhone uses face tracking to adjust focus on subject's face 2. Same face detection can end up detecting faces in arbitrary objects. Very common occurring. Here the leaf. 3. iPhone uses multiple lens to compose a single photo. 1 lens focused on the persons face while another with better focus on the face estimated the face to be within the leaves. 4. The composite photo now has picture of leaf replacing the person's face.

yuvalkarmi 5y ago

Plot twist: this person actually has a leaf for a head

dmitshur 5y ago

I had a feeling of recalling something relevant to this topic recently. Turns out it was a video by Marques Brownlee, "Smartphone Cameras vs Reality!". This instance would've been a great example there. The section at t=364 [1] is quite relevant. [1] https://www.youtube.com/watch?v=MZ8giCWDcyE&t=364 https://www.youtube.com/watch?v=MZ8giCWDcyE&t=364

alboy 5y ago

I take it the processing model was trained on a dataset of Magritte’s paintings?

dehrmann 5y ago

Remember that bit in the Kyle Rittenhouse trial about image resizing?

sathishmanohar 5y ago

She is a ghost! Duh.

ronenlh 5y ago

Can this be an argument for plausible deniability in court?

lloydjones 5y ago

I wonder whether there’s a point where photographic evidence can’t reasonably be used in court etc because of this? I’m aware that Photoshop has been around for an age.

gfykvfyxgc 5y ago

Apple would tell you this is actually the photo you wanted to take.

ezconnect 5y ago

This is worse than the Huawei moon scandal.

aimor 5y ago

I had a problem where letting iPhones automatically touch up portraits taken on a dedicated camera would result in black blobs being inserted over the whites of the subject's eyes. That's just to say that these problems are common and to watch out for it on your own photos. When you know it can happen in subtle ways you'll pay more attention and you'll start to see big errors in images you thought were fine at first glance.

netcan 5y ago

When I was 12 I learned to make autocorrect replace my 8 yr old brother's name with "poophead." It was a genius prank. If I worked at Apple today, I'd do the same thing with his 30 year old head.

tomduncalf 5y ago

I had something like this happen the either week - someone took a photo of a group of us against a wooden wall in low light with the front camera of the iPhone 13 Pro Max, and while the photo looked great zoomed out, if you looked closely it had applied the vertical line and texture of the wooden slats to my face, so I had a sort of wooden face! My assumption is that this is over enthusiastic ML processing trying to enhance photos in challenging conditions… makes me think of e.g. https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-added-ryan-goslings-face-to-this-photo/ https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a... Similarly, I’ve noticed that the iPhone 13 Pro tends to use the wide angle lens for 3x zoom photos in low light then upscale it. This makes some sense as it can capture more light and the photos look good zoomed out, but if you zoom in, you’ll see things like text and small faces have been replaced with random blurry “ML-looking” smudges. If you want to avoid this, right now I think the only way to do so is to use a third party camera app. I use Halide and it’s excellent, it will only use the lens you have selected even if it is in challenging lighting conditions, and it applies much less processing (and you can further disable more processing if you like, I’ve not had the need). The default camera is generally great for snapshots that you aren’t going to zoom into etc. though. Would be nice if they could tweak it to be a bit less aggressive in its processing, perhaps adding a “pro” mode or similar.

albert_e 5y ago
I worry that smartphone photos will become inadmissible in court as evidence ... and will also give all shady characters like politicians a cover to change narratives
- retube 5y ago
  I think we're already past that point....
- rowanG077 5y ago
  I hope that smartphones photos become inadmissable. Because photos taken don't reflect what really happened. I also hope smartphones start storing unprocessed version of every photo that will be admissable in court.

busia 5y ago

A.I. is watching you

rbrbr 5y ago

space_rock 5y ago

So now we can't trust our photos to be fake? Let's get rid of the ML. Optional post processing only. This is why an iPhone photo won't be able to be used in court

aloer 5y ago

I upgraded from an iPhone X to a 13 pro when it was released and since then it’s been a very mixed experience. The range of quality is incredibly frustrating. I enjoy the speed with which I can take pictures that are not blurry. But in most cases I wish I could reduce the “AI aggressiveness” since they all look so artificially sharp. But that’s only part of it. I have pictures of my family this Christmas where we posed in the same location, same light and with just a few seconds apart and the pictures look completely different when it comes to the colors. “Different AI interpretation” I tell those around me that haven’t come in contact with computational photography yet. Then I apologize, saying that I can’t know how the picture will turn out before it’s taken. And that there is no way to reanalyze. No way to feed RAW back to “the AI”. But such is the future… There’s a picture of me and my partner against a perfectly blue sky. I am a few cm behind her. She looks normal while my face looks half transparent / whitened like a ghost in the sky. There’s pictures of text and logos on billboard and such that are so smooth as if they are photoshopped on top. I often tell those people that not only do we need to accept that photos now are not the same concept as photos in the past, we also have to accept that the next iOS update could change the algorithm and photos from this Christmas will be different to photos from next Christmas even when shot with the same camera. Frustrating

emerged 5y ago

Well to be fair, its incredible how far head to leaf technology has come.

jinto36 5y ago

Here's a link to the image in question directly: https://pbs.twimg.com/media/FH0N9HNWQAE0n9R?format=jpg&name=large https://pbs.twimg.com/media/FH0N9HNWQAE0n9R?format=jpg&name=...

Jenda_ 5y ago

So… when are we going to see made up numbers in photographed documents again? https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... Superimposing other's people face onto a photo from a crime scene? Or, as part of my work, I photograph electric switchboards for documentation and diagnostic purposes. Hopefully I won't get any hallucinated connections, right?

GoToRO 5y ago

Impossible to take photo of stars on iphone. Whatever sw they use, it will create random stars in the photo. Like instead of 1 star you get ten.

mitchcohentoo 5y ago

Mystery solved! https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026

cdaringe 5y ago

Man takes photo of leaf, surprised to see leaf in photo

rrauenza 5y ago

There was an update -- sounds like a leaf was really in the shot: > Big news! I sent @sdw the original image. He theorized a leaf from a foreground tree obscured the face. I didn’t think anything was in view, and the closest tree is a Japanese Maple (smaller leaves). But he’s right! Here’s a video I just shot, showing the parallax. Wow!

tqi 5y ago
This thread has been an amazing example of confirmation bias.
darkwizard42 5y ago
LOL amazing that the explanation was so simple!
roryokane 5y ago
Link to that tweet: https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026. Nitter proxy: https://nitter.net/mitchcohen/status/1476951534160257026 https://nitter.net/mitchcohen/status/1476951534160257026
agar 5y ago
I wish this could be pinned to the top. Perhaps a follow-up post to HN would be worthwhile. So many people will leave this thread incorrectly believing this was an iPhone problem. It could easily become the data point people use to cast doubt on all pictures they don't like.
davidbarker 5y ago
I'm the OP (of the HN submission — not the tweet). Really interesting discussions. I've upvoted this — hopefully it makes it to the top. Here's the link to the HN discussion on the update: https://news.ycombinator.com/item?id=29750660 https://news.ycombinator.com/item?id=29750660
ffggvv 5y ago
even if its a foreground tree it still looks blurry and strange doesn't it?

resters 5y ago

It was actually a leaf from a foreground tree, not an image processing artifact: https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026

bereasonable 5y ago

He posted an update. It was a leaf on a tree in front of the woman and it obscured her face. https://twitter.com/mitchcohen/status/1476951534160257026 https://twitter.com/mitchcohen/status/1476951534160257026

davidhariri 5y ago

This title should be updated. The author has since figured out that it was in fact a leaf obstructing the person’s face.