20 ms·

The limits of "computational photography"

278 points by fabfabfab 4y ago

michrassena 4y ago
I haven't finished the article, but it seems like using the flash on the iPhone might have been enough to lower the ISO for the photo. Lower ISO = lower noise. The end results look like a typical noise reduction process. For screen-sized images, the new phones do quite well. But zoom in and it's often a painterly-blur.
- nomel 4y ago
  Or, use the tools provided, to get the raw image: https://support.apple.com/guide/iphone/take-apple-proraw-photos-iphae1e882a3/ios https://support.apple.com/guide/iphone/take-apple-proraw-pho...
neilpanchal 4y ago
> Ultimately, we are still beholden to the pigeonhole principle, and we cannot create information out of thin air. *Looks up pigeonhole principle*: https://en.wikipedia.org/wiki/Pigeonhole_principle https://en.wikipedia.org/wiki/Pigeonhole_principle > If 5 pigeons occupy 4 holes, then there must be some hole with at least 2 pigeons. This is so obvious. > This seemingly obvious statement, a type of counting argument, can be used to demonstrate possibly unexpected results. For example, given that the population of London is greater than the maximum number of hairs that can be present on a human's head, then the pigeonhole principle requires that there must be at least two people in London who have the same number of hairs on their heads. Oh...
- kridsdale1 4y ago
  Yes, it’s zero, and there are half a million of them.
  - eru 4y ago
    The proof works, even if you exclude bald people.
- deleted 4y ago
  [deleted]
bediger4000 4y ago
This article mystified me until I realized it was Computational Photography not Philosophy.
- moistly 4y ago
  Computational Philosophy: “the use of mechanized computational techniques to instantiate, extend, and amplify philosophical research. Computational philosophy is not philosophy of computers or computational techniques; it is rather philosophy using computers and computational techniques. The idea is simply to apply advances in computer technology and techniques to advance discovery, exploration and argument within any philosophical area.” The word “simply” is doing a lot of work in that last sentence, I’m sure! https://plato.stanford.edu/entries/computational-philosophy/ https://plato.stanford.edu/entries/computational-philosophy/ (BTW, I just posted same to front page, if the subject interests you & we’re lucky, it’ll generate some discussion)
  - eru 4y ago
    Compare also https://scottaaronson.blog/?p=735 https://scottaaronson.blog/?p=735
- swayvil 4y ago
  Sounds like an application for one of those new chatbots.
swayvil 4y ago
So it's an edge case fail for the noise reduction alg. How far can a noise reduction algorithm go? Can we use a white painted wall as a mirror?
- michrassena 4y ago
  I don't think we're that point yet, but there's some amazing work on deconvolution for seeing around corners.
  - swayvil 4y ago
    If we could use a rock one light year away as a mirror then we could see 2 years into the past. That's a practical application.
bee_rider 4y ago
The iPhone definitely does some extra processing on text. I’m 99% sure that it recognizes letters and fills in “creatively.” I noticed this while taking some photos of text in low-light. Could barely see it with my eyeballs but the phone worked it out.
- pancrufty 4y ago
  To be fair, sensors have the luxury of time that our eyes don’t have. See astrophotography for an example of “could barely see it but the sensor worked it out.”
  - nine_k 4y ago
    Sensors may have the time, but our hands do not; they are unsteady in subtle ways. When making photos in low light, I always try to lean my phone against something (a bench, a lamppost, a tree, a building) to let the longer exposure be sharper.
    - adgjlsfhk1 4y ago
      as long as there is anything to key onto it's possible to remove the shake algorithmically
      - eru 4y ago
        > as long as there is anything to key onto it's possible to remove the shake algorithmically Why the requirement? The phone already has accelerometers, doesn't it? In any case, less hake should still be easier for the phone to deal with. Those algorithms aren't flawless.
        sebzim4500 4y ago
        There's no way that the accelerators/gyrocscope would be accurate enough to remove the shake. According to google, the iphone gyroscope is accurate to within about 0.5 degrees, roughly two orders of magnitude away from being pixel accurate in a 1x zoom image.
kelsolaar 4y ago
I always shoot in Raw (using the Lightroom iPhone App) to make sure that this kind of defects never occur. Noise is generally preferable and acceptable that the disaster trail left by denoising et al. At least you can do it yourself in a way that pleases you instead of having a ruined photograph.
- ecpottinger 4y ago
  That is what I was thinking. The poster said the photo looked okay at the moment they took a picture and the phone's processing takes over and he then gets junk. I wonder how hard to is to take 'RAW' photos without adding an app first.
  - nomel 4y ago
    You can use the camera app. It’s all in the settings menu for Pro models: https://support.apple.com/guide/iphone/take-apple-proraw-photos-iphae1e882a3/ios https://support.apple.com/guide/iphone/take-apple-proraw-pho... This is a “there’s already a solution, but the average consumer wouldn’t know about it, because the defaults are made for them” type of problem. One could claim that it's a UI problem, and should be exposed in the Camera app. This may be true, but the files are 10 to 12 times larger, with a real "quality hit", as perceived by the average user, for overall aesthetics. I personally think it should be in the settings menu. It's not something you would want enabled without understanding and intent. It’s a little frustrating that apple added this feature, for this exact kind of thing, and they’re, inadvertently or not, getting a little dumped on, do to lack of knowledge/research. These features (Google also attempted to standardize it, not sure they succeeded) were a big deal in the photography world.
    - ecpottinger 4y ago
      Sorry, I do not have an iPhone. How easy is the 'RAW' feature to find? I understand what you mean about most users do not need such a feature. But while supporting computers I am surprise how many features of programs users do not know exist not just because they do not use them but also how hard it is to get at some features. Personally, I seen too many programs where you need to know to turn OFF certain functions before you can enable some other feature. The users often never see what they needed because it is so hard to enable that feature without know they need to turn something else OFF.
epicycles33 4y ago
Interesting article. I think the real question however is whether imposing complex priors (say driven by a neural net) makes images better _on average_ even if has some failure modes. My guess would be that a fairly weak prior trained on a diverse enough dataset would lead to better average image quality(as judged by everyday people in diverse scenarios) and that's why they are used.
- wyager 4y ago
  I think it could absolutely make images better on average, assuming their prior is at all representative. The question, though, is whether the expected benefit to the photographer in cases where it improves the image (usually somewhat small) outweighs the costs when it screws up (perhaps relatively large). Now, are most people going to notice that the iPhone wrecked the text on their subject? Probably not. But they probably also wouldn't notice if the model wasn't applied to the image at all. The median consumer probably mostly benefits from (in terms of how much they like the photo) AE, a bit of curve reshaping (using a smoothed histogram CDF algorithm or something), and maybe some extra saturation.
delta_p_delta_x 4y ago
I would like to see computational photography applied to raw images from DSLRs and MILCs with APS-C and larger sensors. Perhaps Canon, Nikon, Sony, and Fujifilm could have built-in options in their cameras for ‘social media mode’, with a modicum of noise reduction (honestly unnecessary at ISOs lower than about 1600 for modern cameras), but drastically improved HDR and white balance. Many of these cameras are able to take bracketed[1] exposures, and the SNR in even just one image from such sensors is immense compared to the tiny sensors in phones. Surely with this much more data to work with, HDR is much nicer and without the edge brightening typically seen in phone HDR images. [1]: https://www.nikonusa.com/en/learn-and-explore/a/tips-and-techniques/exposure-bracketing-the-creative-insurance-policy.html https://www.nikonusa.com/en/learn-and-explore/a/tips-and-tec...
- haswell 4y ago
  I’d love to see this on “adventures” cameras like the Olympus OM-D line. This would also pair well with Fujifilm’s lineup which already includes camera features focused on in-camera processing.
  - rimliu 4y ago
    I think Olympus was a pioneer at least in some aspects in computation photography. At least "live composite" and "focus stacking" were not very common at the time Olympus introduced them.
- jlarocco 4y ago
  I'm not exactly sure what you're asking for here. In my experience (and as seen in the article) the image processing in most digital cameras will already blow an iPhone out of the water. As far as I know, iPhone and Android aren't doing anything that isn't already done by digital cameras. They ramp up the settings on things like noise reduction and sharpness to balance out their tiny sensors, but it's more or less the same algorithms that the cameras are using. Good cameras even allow you to tweak the settings and control RAW conversion right on the camera. The author could have botched the noise reduction on his Fujifilm to match the iPhone if he wanted to. [0] [0] https://www.jmpeltier.com/fujifilm-in-camera-raw-converter/ https://www.jmpeltier.com/fujifilm-in-camera-raw-converter/
indianmouse 4y ago
The smartphone cameras have improved a lot in the recent years, but they cannot compete or match a full frame sensor provided the limitations. The size of the sensor and the optics play a major role in the final image quality and one can only do so much with the computational photography or whatever method. Especially the iPhone photography and videography is always overrated by the fanbois and some of the "professionals". While it might look good on "some" pictures with the heavy post processing, it just doesn't have any details. It might just appeal fine for a 100% view of the picture as is and even the slightest post processing or editing done on the output pictures ruins them a lot. One has to depend upon what the developer of the application or the manufacturer thinks is the right picture (and who the hell are they to decide what my photo should look like?) and most of the time they are terribly wrong. Apple is just overrated and for that matter, even some of the Android's as well. Raw pics from a full frame sensors hold the fort and will continue to hold for a longtime to come unless the phones match DSLR in terms of sensor size and optics size. Until then "computational photography" will make the pictures look terrible and dictate how it has to look like. I see a lot of comments where folks talk about RAW. But seriously, how does it matter for any normal user who tends to click a pic using the phone instead of a DSLR? If one is photographer, it makes sense, else it is additional workflow to get it in RAW and do the post processing on a computer... I'm just saying... Thoughts welcome...
- jiggawatts 4y ago
  My $0.02 as some one with an expensive full-frame DSLR and and the latest iPhone Pro: There are entire categories of image quality that only Apple seems to bother even trying to improve — and then they leapt past everyone. A few years ago if you wanted to make a HDR, 4K, 60 fps Dolby Vision wide-gamut video… That would have cost you. Tens of thousands on cameras, displays, and software. It would have been a serious undertaking involving a lot of “pro” tools and baroque workflows optimised for Hollywood movie production. With an iPhone I can just hit the record button and it does all of that for me, on the phone! Did you notice that it also does shake reduction? It’s staggeringly good, about the same as GoPro. Just setting up the stabilisation in DaVinci is half an hour of faffing around. The iPhone just has it “on”. I could go on. A challenge I give people is to take a still photo and send it to someone else that is wide gamut, 10 bit, and HDR, any method they prefer. Outside of the Apple ecosystem this is basically impossible in the general case. Everything everywhere is 8-bit SDR sRGB. Heck, even professional print shops still request files in sRGB! So yes, the software in the Apple ecosystem does have a big impact on the end result of photography. I can take a 14-bit dynamic range picture with my Nikon, but I can’t show it to anyone in that quality because of shitty Windows and Linux software, so what’s the point? I take pics with my Apple iPhone instead. All the people I want to show pictures to have iDevices, so I can share the full HDR quality that the phone camera is capable of, not some SDR version.
jlarocco 4y ago
I love my big, heavy Nikon DSLR, and there's really no comparing the images it takes with the ones from my iPhone. Especially in "tricky" lighting situations they're not even in the same ballpark. That said, there can be just as much (or more) "computational photography" going on with a digital camera as there is with a modern phone, the difference is that cameras and processing software give control to the user, and phones typically do not.
- wombat_trouble 4y ago
  "Real" cameras do a lot of postprocessing too, but it's generally oriented at producing faithful results. They might remove unambiguous and correctable issues such as vignetting or lens distortion, but they don't cross the line of inventing new details to make the photo look good. Computational photography techniques on smartphones, on the other hand, were always designed around squishy "user perception" goals to make photos look impressive, details be damned.
  - jlarocco 4y ago
    I didn't see any invented "new details" in the article's iPhone photos. Phones have small sensors and crap lenses, so they ramp up noise reduction and sharpness to make up for it. Turn up the ISO and max out the NR on the Fujifilm and the results would be nearly as bad.
    - wombat_trouble 4y ago
      I see invented texture and layering here: https://yager.io/comp/mi.jpeg https://yager.io/comp/mi.jpeg
      - kaba0 4y ago
        That seems just like edge sharpening.
      - jefftk 4y ago
        It's pretty hard to tell without knowing what sort of image was input to the computational photography process
mortenjorck 4y ago
This is an interesting edge case, and the author makes some instructive observations on the physics side. It would be really interesting, though, to see an image signal processing expert weigh in on what the algorithm(s) are actually doing in this case.
andreareina 4y ago
There’s some criticism about the event horizon photo of M87 because they had to do a lot of filling in based on a model of how black holes behave. IIRC they ran a hyperparameter search and picked the ones that were most consistent with the actual photons received.
- poulpy123 4y ago
  Interferometers usually don't produce image but take power and phase measurements in the frequency plane. If there are enough points taken an image can be "reconstructed" if not the scientists will do model fitting
zeckalpha 4y ago
The inverse has been how wowed I am when a camera is better than my vision. Night mode and giant aperture ratio lenses both wow me. Mirrorless cameras may have been delayed if it weren't for the competition with phones. DSLRs were only around a few years before camera phones.
- Tepix 4y ago
  Lenses in general can make things larger than with normal vision, so why are you so surprised?
  - zeckalpha 4y ago
    Not larger or smaller, I mean better in low light. Large aperture is very different from large focal length.
AstixAndBelix 4y ago
Computational photography excels at certain uses. Noise reduction can be miraculous as of recently, exposure bracketing and automatic merging allows to take good pictures of a scene with a bright sky without obscuring everything, lens distortion, vignetting and chromatic aberration corrections work really well. Of course you cannot really compensate for the lens not resolving enough detail, or not focusing close enough; but since the almost totality of photos taken on a phone will be seen on another mobile device these are the less important bits of the equation. Correct exposure and good colors always look good, regardless of how much you zoom the photo. OP's use case is very limited, and unfortunately didn't provide enough context about the nature of the photo.
killjoywashere 4y ago
> the relevant metric is what I call “photographic bandwidth” - the information-theoretic limit on the amount of optical data that can be absorbed by the camera under given photographic conditions (ambient light, exposure time, etc.). You mean, "resolution"?
- jsmith99 4y ago
  'resolution' is usually used just to mean the number of pixels: nothing about how much ligh they capture or what sort of lens and processing.
  - wnkrshm 4y ago
    Talking about optics, optical resolution is also a thing, i.e. whether you can resolve a certain target like a grid of micrometer-sized bars under a microscope. Edit: That kind of resolution is induced by the optics and not by the sensor (if your optics can resolve the target, you can always add more optics to magnify the image if you have a low-pixel-resolution sensor). Edit2: The poster you replied to is right that optical resolution is a constraint in terms of information that can be reconstructed after being imaged through a specific optical system. An optical system filters light in phase space (imagine a space of position, angle and intensity of light in each point of the optical system, in a geometrical optics picture) and since some components are cut off, you cannot reconstruct an image to arbitrary fidelity, you lose information (or are stuck with a certain optical resolution).
Llamamoe 4y ago
I'm surprised the author is unfamiliar with Google Camera and its super-resolution features[1,2], which uses actually clever algorithms to push digital photography beyond what would be physically possible to get out of a naive set of HDR exposures, both in terms of resolution and dynamic range. It's literal magic. [1] https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html?m=1 https://ai.googleblog.com/2018/10/see-better-and-further-wit... [2] https://petapixel.com/2019/05/28/how-googles-handheld-multi-frame-super-resolution-tech-works/ https://petapixel.com/2019/05/28/how-googles-handheld-multi-...
- smusamashah 4y ago
  I have used 3 pixels and never seen any of this bad post processing as shown by author. I never thought iPhones camera could do bad post processing like this. Seen this in cheap point and shoot cameras and cheap chinese phones though.
- datagram 4y ago
  The author spends a whole paragraph talking about this category of techniques: > Slightly more objectionable, but still mostly reasonable, examples of computational photography are those which try to make more creative use of available information. For example, by stitching together multiple dark images to try to make a brighter one. (Dedicated cameras tend to have better-quality but conceptually similar options like long exposures with physical IS.) However, we are starting to introduce the core sin of modern computational photography: imposing a prior on the image contents. In particular, when we do something like stitch multiple images together, we are making an assumption: the contents of the image have moved only in a predictable way in between frames. If you’re taking a picture of a dark subject that is also moving multiple pixels per frame, the camera can’t just straightforwardly stitch the photos together - it has to either make some assumptions about what the subject is doing, or accept a blurry image. Their point is that it's not magic; these techniques rely on assumptions about the subject being photographed. As soon as those assumptions no longer hold, you start getting weird outputs.
ipsum2 4y ago
Is the full sized image posted anywhere? This could just be the limits of dynamic range blowing out the text in the iPhone pics.
onphonenow 4y ago
A very long article but I’m slightly confused. Is he using proraw? Can you not get unprocessed images from proraw? I’m not sure what the pipeline looks but I thought this type of situation was where pro raw was supposed to be used?
- CarVac 4y ago
  Afaik proraw has the same computational reconstruction, but none of the baked-in contrast, lighting, or color tweaks. This gives you a clean low-noise image with editing flexibility but it does have the flaws of deconvolution and stacking and AI denoising. Actual raw from a cell phone is insanely noisy and hideously soft from diffraction in the best of cases.
  - onphonenow 4y ago
    Thanks - very interesting
  - foldr 4y ago
    >Actual raw from a cell phone is insanely noisy and hideously soft from diffraction in the best of cases. You can get single shot RAW output from Halide and other third party apps on iPhones. It's actually perfectly usable and not particularly noisy or soft. I haven't personally had any problems with the output of ProRAW (which applies far less aggressive sharpening than the standard JPEG processing). I'm pretty sure the photo in the article would have come out fine if it had been shot using ProRAW.
    - CarVac 4y ago
      Do you have any full resolution examples to share? I'm comparing to large-sensor cameras on large 4k screens, of course. What is "usable" or not varies dramatically depending on how large you display it.
      - foldr 4y ago
        I’m comparing it to multi shot raw (i.e. ProRAW). What I mean to say is that if you would consider using a typical ProRAW or JPEG shot from an iPhone (both based on multiple exposures), then you would also consider using a single-shot RAW. At daylight ISOs the difference in noise and sharpness is small. Here's a JPEG from a single shot 12MP RAW made with Halide (can't shoot 48MP single shot RAW for some reason): https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG7BXmH/view?usp=share_link https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG... Here's a JPEG from a ProRAW 48MP made with the camera app: https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG7BXmH/view?usp=share_link https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG... Here's a JPEG from a ProRAW 48MP made with Halide: https://drive.google.com/file/d/13GW_CIIvOSFEsKcON28Y34NgQAAD-EUk/view?usp=share_link https://drive.google.com/file/d/13GW_CIIvOSFEsKcON28Y34NgQAA... All images are shot on an iPhone 14 Pro Max and processed in Lightroom (60 on the sharpening slider, no noise reduction). There are certainly many differences between the images, but it's not as if the single shot RAW is a total mess. In fact, in this case the single shot RAW is cleaner and better looking than the ProRAW output of Apple's camera app for the most part. It even has more textural detail in a couple of areas where Apple's processing has smoodged things. This could partly be because I was able to select ISO 57 using Halide, whereas the camera app chose to use ISO 100. (All images were shot on a tripod, so there was no real need to use a higher ISO.) As you can see, Halide's ProRAW doesn't smoodge quite as much. I generally prefer the Halide ProRAW to the single shot RAW, even though it looks a little more processed when pixel peeping. There is clearly more noise in the single shot RAW (as you'd expect). However, bear in mind that the JPEG above shows the result of doing fairly heavy sharpening and no noise reduction whatsoever. With more balanced processing the noise is much less noticeable. Here's an example: https://drive.google.com/file/d/1PQKAcE-Cr-M6Uz-rcXTDt3OM0EjsB3Uf/view?usp=share_link https://drive.google.com/file/d/1PQKAcE-Cr-M6Uz-rcXTDt3OM0Ej...
account42 4y ago
> For example, by stitching together multiple dark images to try to make a brighter one. (Dedicated cameras tend to have better-quality but conceptually similar options like long exposures with physical IS.) However, we are starting to introduce the core sin of modern computational photography: imposing a prior on the image contents. In particular, when we do something like stitch multiple images together, we are making an assumption: the contents of the image have moved only in a predictable way in between frames. If you’re taking a picture of a dark subject that is also moving multiple pixels per frame, the camera can’t just straightforwardly stitch the photos together - it has to either make some assumptions about what the subject is doing, or accept a blurry image. This applies to dedicated cameras too though - physical image stabilization can compensate for camera motion but not for subject motion. The difference is that a) physical IS can compensate throughout each exposure, not just between exposure and b) the photographer is not bound to a black box algorithm but can instead use his own a priori knowledge to align the images if needed.
- wyager 4y ago
  Yes, I meant to imply that dedicated cameras are committing the same "sin" here. I only meant physical IS is better because you don't need to do things like periodically read out the sensor. You are getting a true full-duration exposure that won't produce artifacts like tearing or skips.
  - foldr 4y ago
    The iPhone has physical IS too. It's very good. (Image stabilisation tends to work better for smaller sensors, as you have less weight to move around and less far to move.)
- foldr 4y ago
  >In particular, when we do something like stitch multiple images together, we are making an assumption: the contents of the image have moved only in a predictable way in between frames From this way of looking at things a normal long exposure also imposes a prior assumption (that nothing is moving). It's just that we're used to the artefact that's generated when this prior isn't true (motion blur).
Traubenfuchs 4y ago
iOS postprocessing is garbage and must be changed. It's an embarrassment. The most infuriating thing is that you can usually see the image before post processing if you are quick enough and those look sharp and good, but this trash software can't be turned off.
- zimpenfish 4y ago
  > but this trash software can't be turned off. Are you talking about proRAW? Or JPEGs straight from the Camera app?
wnkrshm 4y ago
There are many more clever methods that one can use with CMOS that fall under computational photography or optics. One very interesting one is ptychography (in microscopy often Fourier ptychography, since you can use Fourier optics to describe the optical system [0]), which uses a model of an optical system to get an image (iirc x7-x10 resolution) out of many blurry images, while knowing a bit about the optics in front of your image sensor - it can also work in remote sensing to some degree (better with coherent illumination though). Edit: This is not just averaging or maxing pixels, it reconstructs the image using reconstructed phase information from having low-res pictures with different, known illumination or camera positions. [0] https://www.youtube.com/watch?v=hece_x37ITg https://www.youtube.com/watch?v=hece_x37ITg
- londons_explore 4y ago
  I think ptychography is the future of phone cameras... You'll see phones with 1000 lenses and 1000 CCD's, across the whole back of the phone. They'll all be manufactured as a one piece glass moulding and single CCD chip - and the whole thing will be very cheap to make, having moved all the difficulty into software.
whywhywhywhy 4y ago
Haven’t felt like the camera on my iPhone 13 is significantly better than the one on my iPhone 7 at all in terms of basic quality. My shots look about the same. As someone who upgrades every several years I’ve been wondering how people who upgrade every year and rave about the camera being better are even seeing at this point. (Stills only I’m talking about)
- substation13 4y ago
  Have you done a side-by-side comparison? I have noticed huge improvements.
  - shinycode 4y ago
    Exactly, HDR, colors, shadows, night shots all of that makes a huge difference. Take a night shot side by side between the two phones you listed. If there is no difference it’s because you never took those pictures. Maybe a camera it’s just a camera for you. Point shoot done. The difference in various lightning situation are huge but you don’t take that much photos or care enough ?
- npteljes 4y ago
  I'm noticing the same, looking at GSMArena's comparison shots. We opted for an iPhone 11 back then, and out of curiosity, I keep an eye out for camera improvements, and I don't see that too much is happening. Comparing the low-light shots of the 11 and the 14 pro max, there is some extra detail, but the post-processing is also noticeably heavier.
- Terretta 4y ago
  Telephoto and low light Also, the camera, lenses, and sensors don't all update every year. Early on in Apple's tick-tock approach to design iteration, camera updates were the "s" models ("tock") in release cycle. Now they seem to be just incrementing the number and you have to pay attention to what if any changes they make. This time they did 4x pixels and do pixel binning for regular shots and low light.
MarkusWandel 4y ago
I don't have any examples from my own film photography handy, but a quick google brings up https://www.35mmc.com/10/01/2015/low-light-fun-ilford-hp5-ei3200-ilfotec-dd-x/ https://www.35mmc.com/10/01/2015/low-light-fun-ilford-hp5-ei... 3200ISO on black-and-white film was pushing it pretty hard. Yet these pictures look good, in a noisy kind of way. Let an algorithm loose on them and it'll "fix" things, first and foremost by smoothing the skin. Even older low-end dedicated digital cameras do this, some brands more than others. The pictures in low light feel more like a badly done painting than a good, honest, albeit noisy photo. One possibility is that the noise from a digital sensor is not as uniformly pleasing as that from film, so it must be masked.
hilbert42 4y ago
I accept that there is a place for computational/algorithmic photography but I remain deeply sceptical of its actual benefits (in its current incarnation), moreover my recent bad experiences with it have only strengthened my conviction. I have previously discussed having taken photos with a smartphone where certain objects within some images have been so modified by the processing algorithm as to be almost unrecognizable so I won't repeat those various scenarios here. Instead, I'd like to dwell on the implications algorithmic image processing for a moment. Let's briefly look at the issues: 1. Despite a recent announcement by Canon about a large increase in dynamic range in imaging, (https://news.ycombinator.com/item?id=34527687 https://news.ycombinator.com/item?id=34527687), I'm unaware of any current imaging sensor breakthrough that would vastly improve both resolution and dynamic range. Thus, essentially, we have to live with what we're already capable of physically squeezing into our present smartphones. 2. Manufacturers are improving both image sensors and optics but only incrementally. Thus, with current tech and absence of truly significant breakthroughs, we have to live with the limitations as outlined in the article (aberrations, lens flare, sensor insensitivity etc.). 3. Essentially, we're stymied both by the limitations of current tech and physical (smartphone) size. Usually, to overcome such limitations, we'd fall back on the old truism 'there's no substitute for capacity' and just make things bigger as we did with photographic emulsions, past camera lenses, loudspeakers, pipe organs, etc. but that's not possible here. 4. Outside incremental improvements in hardware—the Law of Diminishing (hardware) Returns having arrived—manufacturers have had to resort to computational methods. The trouble is that it seems with the present algorithms that the Law of Diminishing (computational) Returns is also already upon us, so what does this mean? Quo vadis? 5. Clearly, in its current form computational/algorithmic processing has hit a stumbling block or at least a major hiatus. Here, further incremental improvements are likely using current methods and there's little doubt that they'll be applied to recreational photography (smartphones and such), however, unfortunately, we now have a serious (and very obvious) problem with the authenticity of images taken by these cameras. Simply, when software starts guessing what's within images then we've not only lost visual authentication but we have serious downstream issues. It raises questions about whether or not photographic evidence based on computational imaging can be relied upon—or even submitted—as evidence in a court of law (I'd reckon, without ancillary cooperating/conjunctive evidence, such images would not muster if the Rules of Evidence tests were applied. How serious is this? Clearly, it depends on circumstance but long before 'guessing-what's-in-the-image' became in vogue simple compression was 'suspect' in, for example, serious surveillance work—because compression artifacts in an image raised doubts as to what objects actually were—simply, could objects be identified with 100% certainty, if not then what figure could be placed on such measurements/identifications. (Such matters are not hypotheticals or idle speculation, I recall in nuclear safeguards a debate over compression artifacts in remote monitoring equipment. Here, authenticating and identifying objects must meet strict criteria and a failure to authenticate (fully identify) them means a failure of surveillance which is a big deal! For example, the failure to distinguish between, say, round cables and pipes with 100% certainty could be a serious problem, as the latter could be used to transport nuclear materials—thus it'd be deemed a failure of surveillance. That's not out of the bounds of possability in a reprocessing plant.) Obviously, the need to authenticate what's in an image with 100% certainty isn't a daily occurrence for most of us but as these tiny cameras become more and more important and ubiquitous then we'll start seeing them used in areas where their images must be able to be authenticated. Post haste, we need rules and standards about how these computational algorithms process images and how they should be applied. 6. What's the future. On the hardware side we need better sensors with higher resolution and more sensitivity and improved optics (that, say, use metamaterials etc.). Such developments are on their way but don't hold your breath. Computational/algorithmic processing has the potential to do much, much better, but again don't hold your breath. There's considerable potential to correct focus and aberration problems etc. using both front-end and back-end computational methods ('front-end correcting lenses etc. on-the-fly and back-end as post-image processing) but much work still has to be done. Note: such methods also don't rely on guessing. What people often forget is that when a lens cannot fully focus or suffers aberrations, etc. information in the incoming light is not lost—it's just jumbled up (remember your quantum information theory). In the past untangling this mess has been seen as an almost insurmountable problem and it's still a very, very difficult one to resolve. Nevertheless, I'd wager that eventually computational processing of this order will be commonplace, moreover, it'll likely provide some of the most significant advances in imaging we're ever likely to witness.
kblev 4y ago
Is there a way to disable this extra processing?
- npteljes 4y ago
  You can go around some of the extra processing by using other camera apps, for example "Open Camera". It can even shoot RAW photos, so that the least amount of processing is applied to the image. Unfortunately, you can't disable all of the processing, because some of it happen on the hardware, or in the camera's kernel module. https://opencamera.org.uk/ https://opencamera.org.uk/
- hapticmonkey 4y ago
  Shoot in RAW mode. Either with the various third party camera apps, or Apple’s built in “Pro RAW” mode in the iOS camera app.
  - astrange 4y ago
    Third party camera apps don't use the Camera app processing whether or not they're shooting raw mode. You can shoot JPEGs all day.
- wyager 4y ago
  Sort of - (I assume for PR/marketing reasons) Apple doesn't let apps get access to actual raw sensor data. It may be possible to skip the steps that are causing the most trouble here.
  - foldr 4y ago
    Why don't you just try shooting it ProRAW, then try shooting a single shot RAW using a third-party app such as Halide? The latter option certainly removes any fancy computational photography from the pipeline.
worewood 4y ago
There was a very good video from Marques Brownlee about the issue [1]. IPhone cameras are getting worse. [1] https://youtu.be/88kd9tVwkH8 https://youtu.be/88kd9tVwkH8
- WaffleIronMaker 4y ago
  And, in a similar vein, I enjoyed his video on The Best Smartphone Camera of 2022, where he applied a scientific ranking system taking 21.2 million votes from 600,00 users. I had previously assumed, due to Apple's reputation, that iPhones would take pictures that people like more, but that was not the case. [1] https://youtu.be/LQdjmGimh04 https://youtu.be/LQdjmGimh04
macshome 4y ago
A photographer friend had a good way of framing this once for me... "Phones take amazing snapshots, but dedicated cameras can make better photographs." The new smartphone cameras are capable of pretty amazing things and they can extend taking good pictures to a whole new audience. If you need the control though that large sensors and specific lenses can bring then you will need a dedicated camera.
manv1 4y ago
The limits are because at this point there's no way to tell the tool what you're trying to do. The various photo modes are a step in that direction, but they've been stuck. Once they find a way to interact with the processing engine then the quality will jump again. For the vast majority of users, the phone camera is super awesome and just fine.
hedgehog 4y ago
The camera software on the 14 Pro is pretty bad, so that's part of their problem. Almost enough to make me return mine.
matheweis 4y ago
This is not a new problem and can sometimes have disastrous results. 10 years or so ago a variation of this made headlines all over as certain Xerox Workcentres were transposing numbers during scans, due to a compression algorithm that was sometimes matching a different number than the one actually scanned. https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
- antegamisou 4y ago
  The same algorithm was part of the legendary NSO Group's FORCEDENTRY zero click exploit. https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-into-nso-zero-click.html https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i...
fleddr 4y ago
Yep, smartphone cameras are optically terrible, which is then compensated for with clever tricks. These tricks optimize for popular use cases: people, food, etc. One aspect that is little discussed is the inflated quality perception of such a photo when seen on the actual device, an iPhone in this case. iPhones have an incredible screen. OLED, wide gamut color, high PPI. A photo looks radically better on an iPhone compared to opening the same photo on a standard monitor.
- wyager 4y ago
  Apple also uses non-standard HEIF tags to allow for HDR photo display of photos taken by Apple devices. Last time I checked, you couldn't (easily) take a photo from a dedicated camera (which has more than enough dynamic range to justify HDR) and turn it into a file that would get rendered as HDR on iPhone.
smusamashah 4y ago
This looks like iPhone postprocessing problem more than anything else. I have used 3 Google pixel phones (up to pixel 4) and none of them does bad post-processing. In fact, it improves the resolution of whatever you are taking picture of https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html https://ai.googleblog.com/2018/10/see-better-and-further-wit... I never saw any of these phones altering the details like in article.
- shiftpgdn 4y ago
  Google Pixel phones still do "deep fusion" processing like an iphone, but instead with Google's secret sauce. The photo your phone is showing you is what machine learning thinks the picture should look like, and not the picture you took.
  - smusamashah 4y ago
    But unlike examples in the article, whatever magic sauce Google uses, end result does not look different from actual thing.
alistairSH 4y ago
Does anybody know what post-processing is applied to ProRaw images from iOS? I'm guessing true raw (Halide, etc) have none at all, but I recall reading the ProRaw had some applied. I just haven't seen a summary of which steps are applied. 90% of the time, my iPhone photos are fine straight out of the camera as HEIC. But every once in a while, I get something like is described here (or in several other recent similar articles).
- _aavaa_ 4y ago
  They are already demosaiced (this I know for sure and is easy to verify). I also believe that they are the result of stacking several photos to increase the dynamic range and reduce noise (see [0][1]). Even for the "true raw" ones, I don't know if they're truly raw. Do they have distortion and light fall-off correction applied? [0]: https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pixel-phones.html https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pix... [1]: https://dl.acm.org/doi/10.1145/3355089.3356508 https://dl.acm.org/doi/10.1145/3355089.3356508
pvillano 4y ago
There's an important piece of background to understand why computational methods cannot completely correct chromatic aberration. A photon can be any color of the rainbow. The reason ink and TVs can get away with using only 3 colors is because our eyes only have 3 types of receptors (cone cells). Each receptor responds to a range of wavelengths. "In-between" wavelengths will trigger multiple receptors. For example, a TV can send a mix of red and green photons and create the same brain signals as yellow photons would. Animals with more types of receptors, such as bees or the mantis shrimp, wouldn't be fooled by a TV with only three base colors. A camera's sensor performs the same lossy compression as our eyes. Light comes into the camera in a range of wavelengths, and triggers each type of pixel a different amount. Each type of pixel has a sensitivity curve engineered to resemble the sensitivity curve of one of the cone cells in our eyes. Understanding that natural light isn't just red, green, and blue makes it clear why chromatic aberration can't be fixed computionally. A green pixel can't know when it's receiving green photons that are perfectly aligned, or yellow light that needs to be destorted. P.S. There cameras that can "see" a greater range of colors. Search for "spectral cameras" and "infra-red goggles" PPS This is also why a RGB light strip might look white, but objects illuminated by it might look odd. You might be familiar with the fact that a blue object illuminated by a red light will look black. For the same reason, it's possible for a yellow object to be eliminated by red, green, and blue light and still look black. PPPS This is also why custom wall paints are a mixture of more than three colors. Two paints may look completely the same, but objects illuminated by the light bounced off the walls look completely different. PPPPS This is also why high-CRI lightbulbs are a thing. If you get something hot, like the sun or a tungsten filament, it will release photons with a wide range of wavelengths. Neon tubes and LEDs emit a single wavelength, so they must be coated with phosphors that fluoresce — emit light at a different wavelength than they absorbed. Using more kinds of phosphors is more expensive, but makes it more likely that whatever object is illuminated gets all the wavelengths it is able to reflect.
abc_lisper 4y ago
Why is the X-T5 picture so grainy? Was it taken in dark? Or through a microscope?
aaroninsf 4y ago
Q: do apps like Halide and/or "shooting RAW", allow bypassing the post-processing on iOS?

278 points by fabfabfab 4y ago

I haven't finished the article, but it seems like using the flash on the iPhone might have been enough to lower the ISO for the photo. Lower ISO = lower noise. The end results look like a typical noise reduction process. For screen-sized images, the new phones do quite well. But zoom in and it's often a painterly-blur.

nomel 4y ago
Or, use the tools provided, to get the raw image: https://support.apple.com/guide/iphone/take-apple-proraw-photos-iphae1e882a3/ios https://support.apple.com/guide/iphone/take-apple-proraw-pho...

neilpanchal 4y ago

> Ultimately, we are still beholden to the pigeonhole principle, and we cannot create information out of thin air. *Looks up pigeonhole principle*: https://en.wikipedia.org/wiki/Pigeonhole_principle https://en.wikipedia.org/wiki/Pigeonhole_principle > If 5 pigeons occupy 4 holes, then there must be some hole with at least 2 pigeons. This is so obvious. > This seemingly obvious statement, a type of counting argument, can be used to demonstrate possibly unexpected results. For example, given that the population of London is greater than the maximum number of hairs that can be present on a human's head, then the pigeonhole principle requires that there must be at least two people in London who have the same number of hairs on their heads. Oh...

kridsdale1 4y ago
Yes, it’s zero, and there are half a million of them.
- eru 4y ago
  The proof works, even if you exclude bald people.
deleted 4y ago
[deleted]

bediger4000 4y ago

This article mystified me until I realized it was Computational Photography not Philosophy.

moistly 4y ago
Computational Philosophy: “the use of mechanized computational techniques to instantiate, extend, and amplify philosophical research. Computational philosophy is not philosophy of computers or computational techniques; it is rather philosophy using computers and computational techniques. The idea is simply to apply advances in computer technology and techniques to advance discovery, exploration and argument within any philosophical area.” The word “simply” is doing a lot of work in that last sentence, I’m sure! https://plato.stanford.edu/entries/computational-philosophy/ https://plato.stanford.edu/entries/computational-philosophy/ (BTW, I just posted same to front page, if the subject interests you & we’re lucky, it’ll generate some discussion)
- eru 4y ago
  Compare also https://scottaaronson.blog/?p=735 https://scottaaronson.blog/?p=735
swayvil 4y ago
Sounds like an application for one of those new chatbots.

swayvil 4y ago

So it's an edge case fail for the noise reduction alg. How far can a noise reduction algorithm go? Can we use a white painted wall as a mirror?

michrassena 4y ago
I don't think we're that point yet, but there's some amazing work on deconvolution for seeing around corners.
- swayvil 4y ago
  If we could use a rock one light year away as a mirror then we could see 2 years into the past. That's a practical application.

bee_rider 4y ago

The iPhone definitely does some extra processing on text. I’m 99% sure that it recognizes letters and fills in “creatively.” I noticed this while taking some photos of text in low-light. Could barely see it with my eyeballs but the phone worked it out.

pancrufty 4y ago
To be fair, sensors have the luxury of time that our eyes don’t have. See astrophotography for an example of “could barely see it but the sensor worked it out.”
- nine_k 4y ago
  Sensors may have the time, but our hands do not; they are unsteady in subtle ways. When making photos in low light, I always try to lean my phone against something (a bench, a lamppost, a tree, a building) to let the longer exposure be sharper.
  - adgjlsfhk1 4y ago
    as long as there is anything to key onto it's possible to remove the shake algorithmically
    - eru 4y ago
      > as long as there is anything to key onto it's possible to remove the shake algorithmically Why the requirement? The phone already has accelerometers, doesn't it? In any case, less hake should still be easier for the phone to deal with. Those algorithms aren't flawless.
      - sebzim4500 4y ago
        There's no way that the accelerators/gyrocscope would be accurate enough to remove the shake. According to google, the iphone gyroscope is accurate to within about 0.5 degrees, roughly two orders of magnitude away from being pixel accurate in a 1x zoom image.

kelsolaar 4y ago

I always shoot in Raw (using the Lightroom iPhone App) to make sure that this kind of defects never occur. Noise is generally preferable and acceptable that the disaster trail left by denoising et al. At least you can do it yourself in a way that pleases you instead of having a ruined photograph.

ecpottinger 4y ago
That is what I was thinking. The poster said the photo looked okay at the moment they took a picture and the phone's processing takes over and he then gets junk. I wonder how hard to is to take 'RAW' photos without adding an app first.
- nomel 4y ago
  You can use the camera app. It’s all in the settings menu for Pro models: https://support.apple.com/guide/iphone/take-apple-proraw-photos-iphae1e882a3/ios https://support.apple.com/guide/iphone/take-apple-proraw-pho... This is a “there’s already a solution, but the average consumer wouldn’t know about it, because the defaults are made for them” type of problem. One could claim that it's a UI problem, and should be exposed in the Camera app. This may be true, but the files are 10 to 12 times larger, with a real "quality hit", as perceived by the average user, for overall aesthetics. I personally think it should be in the settings menu. It's not something you would want enabled without understanding and intent. It’s a little frustrating that apple added this feature, for this exact kind of thing, and they’re, inadvertently or not, getting a little dumped on, do to lack of knowledge/research. These features (Google also attempted to standardize it, not sure they succeeded) were a big deal in the photography world.
  - ecpottinger 4y ago
    Sorry, I do not have an iPhone. How easy is the 'RAW' feature to find? I understand what you mean about most users do not need such a feature. But while supporting computers I am surprise how many features of programs users do not know exist not just because they do not use them but also how hard it is to get at some features. Personally, I seen too many programs where you need to know to turn OFF certain functions before you can enable some other feature. The users often never see what they needed because it is so hard to enable that feature without know they need to turn something else OFF.

epicycles33 4y ago

Interesting article. I think the real question however is whether imposing complex priors (say driven by a neural net) makes images better _on average_ even if has some failure modes. My guess would be that a fairly weak prior trained on a diverse enough dataset would lead to better average image quality(as judged by everyday people in diverse scenarios) and that's why they are used.

wyager 4y ago
I think it could absolutely make images better on average, assuming their prior is at all representative. The question, though, is whether the expected benefit to the photographer in cases where it improves the image (usually somewhat small) outweighs the costs when it screws up (perhaps relatively large). Now, are most people going to notice that the iPhone wrecked the text on their subject? Probably not. But they probably also wouldn't notice if the model wasn't applied to the image at all. The median consumer probably mostly benefits from (in terms of how much they like the photo) AE, a bit of curve reshaping (using a smoothed histogram CDF algorithm or something), and maybe some extra saturation.

delta_p_delta_x 4y ago

I would like to see computational photography applied to raw images from DSLRs and MILCs with APS-C and larger sensors. Perhaps Canon, Nikon, Sony, and Fujifilm could have built-in options in their cameras for ‘social media mode’, with a modicum of noise reduction (honestly unnecessary at ISOs lower than about 1600 for modern cameras), but drastically improved HDR and white balance. Many of these cameras are able to take bracketed[1] exposures, and the SNR in even just one image from such sensors is immense compared to the tiny sensors in phones. Surely with this much more data to work with, HDR is much nicer and without the edge brightening typically seen in phone HDR images. [1]: https://www.nikonusa.com/en/learn-and-explore/a/tips-and-techniques/exposure-bracketing-the-creative-insurance-policy.html https://www.nikonusa.com/en/learn-and-explore/a/tips-and-tec...

haswell 4y ago
I’d love to see this on “adventures” cameras like the Olympus OM-D line. This would also pair well with Fujifilm’s lineup which already includes camera features focused on in-camera processing.
- rimliu 4y ago
  I think Olympus was a pioneer at least in some aspects in computation photography. At least "live composite" and "focus stacking" were not very common at the time Olympus introduced them.
jlarocco 4y ago
I'm not exactly sure what you're asking for here. In my experience (and as seen in the article) the image processing in most digital cameras will already blow an iPhone out of the water. As far as I know, iPhone and Android aren't doing anything that isn't already done by digital cameras. They ramp up the settings on things like noise reduction and sharpness to balance out their tiny sensors, but it's more or less the same algorithms that the cameras are using. Good cameras even allow you to tweak the settings and control RAW conversion right on the camera. The author could have botched the noise reduction on his Fujifilm to match the iPhone if he wanted to. [0] [0] https://www.jmpeltier.com/fujifilm-in-camera-raw-converter/ https://www.jmpeltier.com/fujifilm-in-camera-raw-converter/

indianmouse 4y ago

The smartphone cameras have improved a lot in the recent years, but they cannot compete or match a full frame sensor provided the limitations. The size of the sensor and the optics play a major role in the final image quality and one can only do so much with the computational photography or whatever method. Especially the iPhone photography and videography is always overrated by the fanbois and some of the "professionals". While it might look good on "some" pictures with the heavy post processing, it just doesn't have any details. It might just appeal fine for a 100% view of the picture as is and even the slightest post processing or editing done on the output pictures ruins them a lot. One has to depend upon what the developer of the application or the manufacturer thinks is the right picture (and who the hell are they to decide what my photo should look like?) and most of the time they are terribly wrong. Apple is just overrated and for that matter, even some of the Android's as well. Raw pics from a full frame sensors hold the fort and will continue to hold for a longtime to come unless the phones match DSLR in terms of sensor size and optics size. Until then "computational photography" will make the pictures look terrible and dictate how it has to look like. I see a lot of comments where folks talk about RAW. But seriously, how does it matter for any normal user who tends to click a pic using the phone instead of a DSLR? If one is photographer, it makes sense, else it is additional workflow to get it in RAW and do the post processing on a computer... I'm just saying... Thoughts welcome...

jiggawatts 4y ago
My $0.02 as some one with an expensive full-frame DSLR and and the latest iPhone Pro: There are entire categories of image quality that only Apple seems to bother even trying to improve — and then they leapt past everyone. A few years ago if you wanted to make a HDR, 4K, 60 fps Dolby Vision wide-gamut video… That would have cost you. Tens of thousands on cameras, displays, and software. It would have been a serious undertaking involving a lot of “pro” tools and baroque workflows optimised for Hollywood movie production. With an iPhone I can just hit the record button and it does all of that for me, on the phone! Did you notice that it also does shake reduction? It’s staggeringly good, about the same as GoPro. Just setting up the stabilisation in DaVinci is half an hour of faffing around. The iPhone just has it “on”. I could go on. A challenge I give people is to take a still photo and send it to someone else that is wide gamut, 10 bit, and HDR, any method they prefer. Outside of the Apple ecosystem this is basically impossible in the general case. Everything everywhere is 8-bit SDR sRGB. Heck, even professional print shops still request files in sRGB! So yes, the software in the Apple ecosystem does have a big impact on the end result of photography. I can take a 14-bit dynamic range picture with my Nikon, but I can’t show it to anyone in that quality because of shitty Windows and Linux software, so what’s the point? I take pics with my Apple iPhone instead. All the people I want to show pictures to have iDevices, so I can share the full HDR quality that the phone camera is capable of, not some SDR version.

jlarocco 4y ago

I love my big, heavy Nikon DSLR, and there's really no comparing the images it takes with the ones from my iPhone. Especially in "tricky" lighting situations they're not even in the same ballpark. That said, there can be just as much (or more) "computational photography" going on with a digital camera as there is with a modern phone, the difference is that cameras and processing software give control to the user, and phones typically do not.

wombat_trouble 4y ago
"Real" cameras do a lot of postprocessing too, but it's generally oriented at producing faithful results. They might remove unambiguous and correctable issues such as vignetting or lens distortion, but they don't cross the line of inventing new details to make the photo look good. Computational photography techniques on smartphones, on the other hand, were always designed around squishy "user perception" goals to make photos look impressive, details be damned.
- jlarocco 4y ago
  I didn't see any invented "new details" in the article's iPhone photos. Phones have small sensors and crap lenses, so they ramp up noise reduction and sharpness to make up for it. Turn up the ISO and max out the NR on the Fujifilm and the results would be nearly as bad.
  - wombat_trouble 4y ago
    I see invented texture and layering here: https://yager.io/comp/mi.jpeg https://yager.io/comp/mi.jpeg
    - kaba0 4y ago
      That seems just like edge sharpening.
    - jefftk 4y ago
      It's pretty hard to tell without knowing what sort of image was input to the computational photography process

mortenjorck 4y ago

This is an interesting edge case, and the author makes some instructive observations on the physics side. It would be really interesting, though, to see an image signal processing expert weigh in on what the algorithm(s) are actually doing in this case.

andreareina 4y ago

There’s some criticism about the event horizon photo of M87 because they had to do a lot of filling in based on a model of how black holes behave. IIRC they ran a hyperparameter search and picked the ones that were most consistent with the actual photons received.

poulpy123 4y ago
Interferometers usually don't produce image but take power and phase measurements in the frequency plane. If there are enough points taken an image can be "reconstructed" if not the scientists will do model fitting

zeckalpha 4y ago

The inverse has been how wowed I am when a camera is better than my vision. Night mode and giant aperture ratio lenses both wow me. Mirrorless cameras may have been delayed if it weren't for the competition with phones. DSLRs were only around a few years before camera phones.

Tepix 4y ago
Lenses in general can make things larger than with normal vision, so why are you so surprised?
- zeckalpha 4y ago
  Not larger or smaller, I mean better in low light. Large aperture is very different from large focal length.

AstixAndBelix 4y ago

Computational photography excels at certain uses. Noise reduction can be miraculous as of recently, exposure bracketing and automatic merging allows to take good pictures of a scene with a bright sky without obscuring everything, lens distortion, vignetting and chromatic aberration corrections work really well. Of course you cannot really compensate for the lens not resolving enough detail, or not focusing close enough; but since the almost totality of photos taken on a phone will be seen on another mobile device these are the less important bits of the equation. Correct exposure and good colors always look good, regardless of how much you zoom the photo. OP's use case is very limited, and unfortunately didn't provide enough context about the nature of the photo.

killjoywashere 4y ago

> the relevant metric is what I call “photographic bandwidth” - the information-theoretic limit on the amount of optical data that can be absorbed by the camera under given photographic conditions (ambient light, exposure time, etc.). You mean, "resolution"?

jsmith99 4y ago
'resolution' is usually used just to mean the number of pixels: nothing about how much ligh they capture or what sort of lens and processing.
- wnkrshm 4y ago
  Talking about optics, optical resolution is also a thing, i.e. whether you can resolve a certain target like a grid of micrometer-sized bars under a microscope. Edit: That kind of resolution is induced by the optics and not by the sensor (if your optics can resolve the target, you can always add more optics to magnify the image if you have a low-pixel-resolution sensor). Edit2: The poster you replied to is right that optical resolution is a constraint in terms of information that can be reconstructed after being imaged through a specific optical system. An optical system filters light in phase space (imagine a space of position, angle and intensity of light in each point of the optical system, in a geometrical optics picture) and since some components are cut off, you cannot reconstruct an image to arbitrary fidelity, you lose information (or are stuck with a certain optical resolution).

Llamamoe 4y ago

I'm surprised the author is unfamiliar with Google Camera and its super-resolution features[1,2], which uses actually clever algorithms to push digital photography beyond what would be physically possible to get out of a naive set of HDR exposures, both in terms of resolution and dynamic range. It's literal magic. [1] https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html?m=1 https://ai.googleblog.com/2018/10/see-better-and-further-wit... [2] https://petapixel.com/2019/05/28/how-googles-handheld-multi-frame-super-resolution-tech-works/ https://petapixel.com/2019/05/28/how-googles-handheld-multi-...

smusamashah 4y ago
I have used 3 pixels and never seen any of this bad post processing as shown by author. I never thought iPhones camera could do bad post processing like this. Seen this in cheap point and shoot cameras and cheap chinese phones though.
datagram 4y ago
The author spends a whole paragraph talking about this category of techniques: > Slightly more objectionable, but still mostly reasonable, examples of computational photography are those which try to make more creative use of available information. For example, by stitching together multiple dark images to try to make a brighter one. (Dedicated cameras tend to have better-quality but conceptually similar options like long exposures with physical IS.) However, we are starting to introduce the core sin of modern computational photography: imposing a prior on the image contents. In particular, when we do something like stitch multiple images together, we are making an assumption: the contents of the image have moved only in a predictable way in between frames. If you’re taking a picture of a dark subject that is also moving multiple pixels per frame, the camera can’t just straightforwardly stitch the photos together - it has to either make some assumptions about what the subject is doing, or accept a blurry image. Their point is that it's not magic; these techniques rely on assumptions about the subject being photographed. As soon as those assumptions no longer hold, you start getting weird outputs.

ipsum2 4y ago

Is the full sized image posted anywhere? This could just be the limits of dynamic range blowing out the text in the iPhone pics.

onphonenow 4y ago

A very long article but I’m slightly confused. Is he using proraw? Can you not get unprocessed images from proraw? I’m not sure what the pipeline looks but I thought this type of situation was where pro raw was supposed to be used?

CarVac 4y ago
Afaik proraw has the same computational reconstruction, but none of the baked-in contrast, lighting, or color tweaks. This gives you a clean low-noise image with editing flexibility but it does have the flaws of deconvolution and stacking and AI denoising. Actual raw from a cell phone is insanely noisy and hideously soft from diffraction in the best of cases.
- onphonenow 4y ago
  Thanks - very interesting
- foldr 4y ago
  >Actual raw from a cell phone is insanely noisy and hideously soft from diffraction in the best of cases. You can get single shot RAW output from Halide and other third party apps on iPhones. It's actually perfectly usable and not particularly noisy or soft. I haven't personally had any problems with the output of ProRAW (which applies far less aggressive sharpening than the standard JPEG processing). I'm pretty sure the photo in the article would have come out fine if it had been shot using ProRAW.
  - CarVac 4y ago
    Do you have any full resolution examples to share? I'm comparing to large-sensor cameras on large 4k screens, of course. What is "usable" or not varies dramatically depending on how large you display it.
    - foldr 4y ago
      I’m comparing it to multi shot raw (i.e. ProRAW). What I mean to say is that if you would consider using a typical ProRAW or JPEG shot from an iPhone (both based on multiple exposures), then you would also consider using a single-shot RAW. At daylight ISOs the difference in noise and sharpness is small. Here's a JPEG from a single shot 12MP RAW made with Halide (can't shoot 48MP single shot RAW for some reason): https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG7BXmH/view?usp=share_link https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG... Here's a JPEG from a ProRAW 48MP made with the camera app: https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG7BXmH/view?usp=share_link https://drive.google.com/file/d/1oqQB_UbdBaaoM3vC2IG_9jzP2AG... Here's a JPEG from a ProRAW 48MP made with Halide: https://drive.google.com/file/d/13GW_CIIvOSFEsKcON28Y34NgQAAD-EUk/view?usp=share_link https://drive.google.com/file/d/13GW_CIIvOSFEsKcON28Y34NgQAA... All images are shot on an iPhone 14 Pro Max and processed in Lightroom (60 on the sharpening slider, no noise reduction). There are certainly many differences between the images, but it's not as if the single shot RAW is a total mess. In fact, in this case the single shot RAW is cleaner and better looking than the ProRAW output of Apple's camera app for the most part. It even has more textural detail in a couple of areas where Apple's processing has smoodged things. This could partly be because I was able to select ISO 57 using Halide, whereas the camera app chose to use ISO 100. (All images were shot on a tripod, so there was no real need to use a higher ISO.) As you can see, Halide's ProRAW doesn't smoodge quite as much. I generally prefer the Halide ProRAW to the single shot RAW, even though it looks a little more processed when pixel peeping. There is clearly more noise in the single shot RAW (as you'd expect). However, bear in mind that the JPEG above shows the result of doing fairly heavy sharpening and no noise reduction whatsoever. With more balanced processing the noise is much less noticeable. Here's an example: https://drive.google.com/file/d/1PQKAcE-Cr-M6Uz-rcXTDt3OM0EjsB3Uf/view?usp=share_link https://drive.google.com/file/d/1PQKAcE-Cr-M6Uz-rcXTDt3OM0Ej...

account42 4y ago

> For example, by stitching together multiple dark images to try to make a brighter one. (Dedicated cameras tend to have better-quality but conceptually similar options like long exposures with physical IS.) However, we are starting to introduce the core sin of modern computational photography: imposing a prior on the image contents. In particular, when we do something like stitch multiple images together, we are making an assumption: the contents of the image have moved only in a predictable way in between frames. If you’re taking a picture of a dark subject that is also moving multiple pixels per frame, the camera can’t just straightforwardly stitch the photos together - it has to either make some assumptions about what the subject is doing, or accept a blurry image. This applies to dedicated cameras too though - physical image stabilization can compensate for camera motion but not for subject motion. The difference is that a) physical IS can compensate throughout each exposure, not just between exposure and b) the photographer is not bound to a black box algorithm but can instead use his own a priori knowledge to align the images if needed.

wyager 4y ago
Yes, I meant to imply that dedicated cameras are committing the same "sin" here. I only meant physical IS is better because you don't need to do things like periodically read out the sensor. You are getting a true full-duration exposure that won't produce artifacts like tearing or skips.
- foldr 4y ago
  The iPhone has physical IS too. It's very good. (Image stabilisation tends to work better for smaller sensors, as you have less weight to move around and less far to move.)
foldr 4y ago
>In particular, when we do something like stitch multiple images together, we are making an assumption: the contents of the image have moved only in a predictable way in between frames From this way of looking at things a normal long exposure also imposes a prior assumption (that nothing is moving). It's just that we're used to the artefact that's generated when this prior isn't true (motion blur).

Traubenfuchs 4y ago

iOS postprocessing is garbage and must be changed. It's an embarrassment. The most infuriating thing is that you can usually see the image before post processing if you are quick enough and those look sharp and good, but this trash software can't be turned off.

zimpenfish 4y ago
> but this trash software can't be turned off. Are you talking about proRAW? Or JPEGs straight from the Camera app?

wnkrshm 4y ago

There are many more clever methods that one can use with CMOS that fall under computational photography or optics. One very interesting one is ptychography (in microscopy often Fourier ptychography, since you can use Fourier optics to describe the optical system [0]), which uses a model of an optical system to get an image (iirc x7-x10 resolution) out of many blurry images, while knowing a bit about the optics in front of your image sensor - it can also work in remote sensing to some degree (better with coherent illumination though). Edit: This is not just averaging or maxing pixels, it reconstructs the image using reconstructed phase information from having low-res pictures with different, known illumination or camera positions. [0] https://www.youtube.com/watch?v=hece_x37ITg https://www.youtube.com/watch?v=hece_x37ITg

londons_explore 4y ago
I think ptychography is the future of phone cameras... You'll see phones with 1000 lenses and 1000 CCD's, across the whole back of the phone. They'll all be manufactured as a one piece glass moulding and single CCD chip - and the whole thing will be very cheap to make, having moved all the difficulty into software.

whywhywhywhy 4y ago

Haven’t felt like the camera on my iPhone 13 is significantly better than the one on my iPhone 7 at all in terms of basic quality. My shots look about the same. As someone who upgrades every several years I’ve been wondering how people who upgrade every year and rave about the camera being better are even seeing at this point. (Stills only I’m talking about)

substation13 4y ago
Have you done a side-by-side comparison? I have noticed huge improvements.
- shinycode 4y ago
  Exactly, HDR, colors, shadows, night shots all of that makes a huge difference. Take a night shot side by side between the two phones you listed. If there is no difference it’s because you never took those pictures. Maybe a camera it’s just a camera for you. Point shoot done. The difference in various lightning situation are huge but you don’t take that much photos or care enough ?
npteljes 4y ago
I'm noticing the same, looking at GSMArena's comparison shots. We opted for an iPhone 11 back then, and out of curiosity, I keep an eye out for camera improvements, and I don't see that too much is happening. Comparing the low-light shots of the 11 and the 14 pro max, there is some extra detail, but the post-processing is also noticeably heavier.
Terretta 4y ago
Telephoto and low light Also, the camera, lenses, and sensors don't all update every year. Early on in Apple's tick-tock approach to design iteration, camera updates were the "s" models ("tock") in release cycle. Now they seem to be just incrementing the number and you have to pay attention to what if any changes they make. This time they did 4x pixels and do pixel binning for regular shots and low light.

MarkusWandel 4y ago

I don't have any examples from my own film photography handy, but a quick google brings up https://www.35mmc.com/10/01/2015/low-light-fun-ilford-hp5-ei3200-ilfotec-dd-x/ https://www.35mmc.com/10/01/2015/low-light-fun-ilford-hp5-ei... 3200ISO on black-and-white film was pushing it pretty hard. Yet these pictures look good, in a noisy kind of way. Let an algorithm loose on them and it'll "fix" things, first and foremost by smoothing the skin. Even older low-end dedicated digital cameras do this, some brands more than others. The pictures in low light feel more like a badly done painting than a good, honest, albeit noisy photo. One possibility is that the noise from a digital sensor is not as uniformly pleasing as that from film, so it must be masked.

hilbert42 4y ago

I accept that there is a place for computational/algorithmic photography but I remain deeply sceptical of its actual benefits (in its current incarnation), moreover my recent bad experiences with it have only strengthened my conviction. I have previously discussed having taken photos with a smartphone where certain objects within some images have been so modified by the processing algorithm as to be almost unrecognizable so I won't repeat those various scenarios here. Instead, I'd like to dwell on the implications algorithmic image processing for a moment. Let's briefly look at the issues: 1. Despite a recent announcement by Canon about a large increase in dynamic range in imaging, (https://news.ycombinator.com/item?id=34527687 https://news.ycombinator.com/item?id=34527687), I'm unaware of any current imaging sensor breakthrough that would vastly improve both resolution and dynamic range. Thus, essentially, we have to live with what we're already capable of physically squeezing into our present smartphones. 2. Manufacturers are improving both image sensors and optics but only incrementally. Thus, with current tech and absence of truly significant breakthroughs, we have to live with the limitations as outlined in the article (aberrations, lens flare, sensor insensitivity etc.). 3. Essentially, we're stymied both by the limitations of current tech and physical (smartphone) size. Usually, to overcome such limitations, we'd fall back on the old truism 'there's no substitute for capacity' and just make things bigger as we did with photographic emulsions, past camera lenses, loudspeakers, pipe organs, etc. but that's not possible here. 4. Outside incremental improvements in hardware—the Law of Diminishing (hardware) Returns having arrived—manufacturers have had to resort to computational methods. The trouble is that it seems with the present algorithms that the Law of Diminishing (computational) Returns is also already upon us, so what does this mean? Quo vadis? 5. Clearly, in its current form computational/algorithmic processing has hit a stumbling block or at least a major hiatus. Here, further incremental improvements are likely using current methods and there's little doubt that they'll be applied to recreational photography (smartphones and such), however, unfortunately, we now have a serious (and very obvious) problem with the authenticity of images taken by these cameras. Simply, when software starts guessing what's within images then we've not only lost visual authentication but we have serious downstream issues. It raises questions about whether or not photographic evidence based on computational imaging can be relied upon—or even submitted—as evidence in a court of law (I'd reckon, without ancillary cooperating/conjunctive evidence, such images would not muster if the Rules of Evidence tests were applied. How serious is this? Clearly, it depends on circumstance but long before 'guessing-what's-in-the-image' became in vogue simple compression was 'suspect' in, for example, serious surveillance work—because compression artifacts in an image raised doubts as to what objects actually were—simply, could objects be identified with 100% certainty, if not then what figure could be placed on such measurements/identifications. (Such matters are not hypotheticals or idle speculation, I recall in nuclear safeguards a debate over compression artifacts in remote monitoring equipment. Here, authenticating and identifying objects must meet strict criteria and a failure to authenticate (fully identify) them means a failure of surveillance which is a big deal! For example, the failure to distinguish between, say, round cables and pipes with 100% certainty could be a serious problem, as the latter could be used to transport nuclear materials—thus it'd be deemed a failure of surveillance. That's not out of the bounds of possability in a reprocessing plant.) Obviously, the need to authenticate what's in an image with 100% certainty isn't a daily occurrence for most of us but as these tiny cameras become more and more important and ubiquitous then we'll start seeing them used in areas where their images must be able to be authenticated. Post haste, we need rules and standards about how these computational algorithms process images and how they should be applied. 6. What's the future. On the hardware side we need better sensors with higher resolution and more sensitivity and improved optics (that, say, use metamaterials etc.). Such developments are on their way but don't hold your breath. Computational/algorithmic processing has the potential to do much, much better, but again don't hold your breath. There's considerable potential to correct focus and aberration problems etc. using both front-end and back-end computational methods ('front-end correcting lenses etc. on-the-fly and back-end as post-image processing) but much work still has to be done. Note: such methods also don't rely on guessing. What people often forget is that when a lens cannot fully focus or suffers aberrations, etc. information in the incoming light is not lost—it's just jumbled up (remember your quantum information theory). In the past untangling this mess has been seen as an almost insurmountable problem and it's still a very, very difficult one to resolve. Nevertheless, I'd wager that eventually computational processing of this order will be commonplace, moreover, it'll likely provide some of the most significant advances in imaging we're ever likely to witness.

kblev 4y ago

Is there a way to disable this extra processing?

npteljes 4y ago
You can go around some of the extra processing by using other camera apps, for example "Open Camera". It can even shoot RAW photos, so that the least amount of processing is applied to the image. Unfortunately, you can't disable all of the processing, because some of it happen on the hardware, or in the camera's kernel module. https://opencamera.org.uk/ https://opencamera.org.uk/
hapticmonkey 4y ago
Shoot in RAW mode. Either with the various third party camera apps, or Apple’s built in “Pro RAW” mode in the iOS camera app.
- astrange 4y ago
  Third party camera apps don't use the Camera app processing whether or not they're shooting raw mode. You can shoot JPEGs all day.
wyager 4y ago
Sort of - (I assume for PR/marketing reasons) Apple doesn't let apps get access to actual raw sensor data. It may be possible to skip the steps that are causing the most trouble here.
- foldr 4y ago
  Why don't you just try shooting it ProRAW, then try shooting a single shot RAW using a third-party app such as Halide? The latter option certainly removes any fancy computational photography from the pipeline.

worewood 4y ago

There was a very good video from Marques Brownlee about the issue [1]. IPhone cameras are getting worse. [1] https://youtu.be/88kd9tVwkH8 https://youtu.be/88kd9tVwkH8

WaffleIronMaker 4y ago
And, in a similar vein, I enjoyed his video on The Best Smartphone Camera of 2022, where he applied a scientific ranking system taking 21.2 million votes from 600,00 users. I had previously assumed, due to Apple's reputation, that iPhones would take pictures that people like more, but that was not the case. [1] https://youtu.be/LQdjmGimh04 https://youtu.be/LQdjmGimh04

macshome 4y ago

A photographer friend had a good way of framing this once for me... "Phones take amazing snapshots, but dedicated cameras can make better photographs." The new smartphone cameras are capable of pretty amazing things and they can extend taking good pictures to a whole new audience. If you need the control though that large sensors and specific lenses can bring then you will need a dedicated camera.

manv1 4y ago

The limits are because at this point there's no way to tell the tool what you're trying to do. The various photo modes are a step in that direction, but they've been stuck. Once they find a way to interact with the processing engine then the quality will jump again. For the vast majority of users, the phone camera is super awesome and just fine.

hedgehog 4y ago

The camera software on the 14 Pro is pretty bad, so that's part of their problem. Almost enough to make me return mine.

matheweis 4y ago

This is not a new problem and can sometimes have disastrous results. 10 years or so ago a variation of this made headlines all over as certain Xerox Workcentres were transposing numbers during scans, due to a compression algorithm that was sometimes matching a different number than the one actually scanned. https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...

antegamisou 4y ago
The same algorithm was part of the legendary NSO Group's FORCEDENTRY zero click exploit. https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-into-nso-zero-click.html https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i...

fleddr 4y ago

Yep, smartphone cameras are optically terrible, which is then compensated for with clever tricks. These tricks optimize for popular use cases: people, food, etc. One aspect that is little discussed is the inflated quality perception of such a photo when seen on the actual device, an iPhone in this case. iPhones have an incredible screen. OLED, wide gamut color, high PPI. A photo looks radically better on an iPhone compared to opening the same photo on a standard monitor.

wyager 4y ago
Apple also uses non-standard HEIF tags to allow for HDR photo display of photos taken by Apple devices. Last time I checked, you couldn't (easily) take a photo from a dedicated camera (which has more than enough dynamic range to justify HDR) and turn it into a file that would get rendered as HDR on iPhone.

smusamashah 4y ago

This looks like iPhone postprocessing problem more than anything else. I have used 3 Google pixel phones (up to pixel 4) and none of them does bad post-processing. In fact, it improves the resolution of whatever you are taking picture of https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html https://ai.googleblog.com/2018/10/see-better-and-further-wit... I never saw any of these phones altering the details like in article.

shiftpgdn 4y ago
Google Pixel phones still do "deep fusion" processing like an iphone, but instead with Google's secret sauce. The photo your phone is showing you is what machine learning thinks the picture should look like, and not the picture you took.
- smusamashah 4y ago
  But unlike examples in the article, whatever magic sauce Google uses, end result does not look different from actual thing.

alistairSH 4y ago

Does anybody know what post-processing is applied to ProRaw images from iOS? I'm guessing true raw (Halide, etc) have none at all, but I recall reading the ProRaw had some applied. I just haven't seen a summary of which steps are applied. 90% of the time, my iPhone photos are fine straight out of the camera as HEIC. But every once in a while, I get something like is described here (or in several other recent similar articles).

_aavaa_ 4y ago
They are already demosaiced (this I know for sure and is easy to verify). I also believe that they are the result of stacking several photos to increase the dynamic range and reduce noise (see [0][1]). Even for the "true raw" ones, I don't know if they're truly raw. Do they have distortion and light fall-off correction applied? [0]: https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pixel-phones.html https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pix... [1]: https://dl.acm.org/doi/10.1145/3355089.3356508 https://dl.acm.org/doi/10.1145/3355089.3356508

pvillano 4y ago

There's an important piece of background to understand why computational methods cannot completely correct chromatic aberration. A photon can be any color of the rainbow. The reason ink and TVs can get away with using only 3 colors is because our eyes only have 3 types of receptors (cone cells). Each receptor responds to a range of wavelengths. "In-between" wavelengths will trigger multiple receptors. For example, a TV can send a mix of red and green photons and create the same brain signals as yellow photons would. Animals with more types of receptors, such as bees or the mantis shrimp, wouldn't be fooled by a TV with only three base colors. A camera's sensor performs the same lossy compression as our eyes. Light comes into the camera in a range of wavelengths, and triggers each type of pixel a different amount. Each type of pixel has a sensitivity curve engineered to resemble the sensitivity curve of one of the cone cells in our eyes. Understanding that natural light isn't just red, green, and blue makes it clear why chromatic aberration can't be fixed computionally. A green pixel can't know when it's receiving green photons that are perfectly aligned, or yellow light that needs to be destorted. P.S. There cameras that can "see" a greater range of colors. Search for "spectral cameras" and "infra-red goggles" PPS This is also why a RGB light strip might look white, but objects illuminated by it might look odd. You might be familiar with the fact that a blue object illuminated by a red light will look black. For the same reason, it's possible for a yellow object to be eliminated by red, green, and blue light and still look black. PPPS This is also why custom wall paints are a mixture of more than three colors. Two paints may look completely the same, but objects illuminated by the light bounced off the walls look completely different. PPPPS This is also why high-CRI lightbulbs are a thing. If you get something hot, like the sun or a tungsten filament, it will release photons with a wide range of wavelengths. Neon tubes and LEDs emit a single wavelength, so they must be coated with phosphors that fluoresce — emit light at a different wavelength than they absorbed. Using more kinds of phosphors is more expensive, but makes it more likely that whatever object is illuminated gets all the wavelengths it is able to reflect.

abc_lisper 4y ago

Why is the X-T5 picture so grainy? Was it taken in dark? Or through a microscope?

aaroninsf 4y ago

Q: do apps like Halide and/or "shooting RAW", allow bypassing the post-processing on iOS?