3 ms·

We seem to mostly agree. Here’s the line—on one side of it are phones, able to produce decent images despite hardware limitations by applying black-box magic wh

We seem to mostly agree. Here’s the line—on one side of it are phones, able to produce decent images despite hardware limitations by applying black-box magic when interpreting raw captures (but not faring that well when used to shoot raw); on the other side are cameras, employing advanced tech (and sometimes, indeed, magic) to assist you with capturing raw signal of much higher fidelity and yielding full control as to its further interpretation. That’s what I meant by cameras being in a qualitatively different category, and why I wrote my comment arguing against the next iPhone somehow rendering mirrorless cameras useless. (The only point where it might become more of a debate in my eyes is if phones start applying similar extensive processing that they do, but at raw levels. There could be genuine gains there—but also unwelcome unpredictable adulteration of DNG captures.)

orbital-decay 5y ago
My main objection is this: >able to produce decent images despite hardware limitations by applying black-box magic when interpreting raw captures (but not faring that well when used to shoot raw) Not really, the quality boost in recent smartphones comes exactly from bayer domain stacking, which is scene-referred capture. You can absolutely get precise photometric values from this process, as precise as your hardware and lighting allows you to, and as long as you calibrate your device. Those apps contain profiles for specific sensors precisely for this reason, albeit you can do it yourself better, as many people do just like they do it with proper cameras. There's no black box magic here, the output is just the same ordinary DNG with the CFA photometric interpretation, but with significantly lowered noise floor and better dynamic range. Moreover, the pipeline is strictly separated into two parts: bayer domain operations, which give the quality boost, and post-demosaicing processing (auto white balance, tonemapping, skin smoothing, other features aimed at point-and-shoot users - what you call magic). I'm not claiming smartphones are somehow better, which would be silly. I'm objecting to a claim that apps do some opaque magic behind your back when writing RAWs. They don't, you can get entirely normal RAWs with all the quality benefits of stacking (and its limitations). And to be honest, I feel like our discussion is creeping into a no true scotsman/audiophile territory a bit. There's nothing sacred or qualitatively different about single-shot capture, no modern normal-sized sensor gives you truly unprocessed internal bucket values, and even what is counted as single shot is often unclear (many sensors are capable of simultaneous multiple exposure capture requiring the same exposure fusion to get the result; is it single shot or not?). What smartphones, and some cameras, do is essentially temporal denoising, it's a routine imaging technique employed by videographers and in technical imaging (which is as precise and scene-referred as you can get), and I'm not sure what's magical here. Same with temporal super-resolution that many cameras do.
- goblin89 5y ago
  Sure, some of the black box processing that phone camera does relies on known algorithms. The difference is that it also makes use of ML, which IMO either plays a significant role or will do so in near term. > auto white balance, tonemapping, skin smoothing, other features This, for example, I see as an opaque step with nondeterministic outcomes, which if you take away from phones and shoot raw reveals how far they are away from mirrorless cameras. (Cameras that shoot raw do not do this all that well, nor are they expected to.) > There's nothing sacred or qualitatively different about single-shot capture I stack exposures too. Whatever process works, works. The difference is that my flow is A -> N -> X, my camera’s built-in processing is A -> M -> Y, while ML-powered black-box compute in a phone does A -> ? -> R today or A -> ? -> Z tomorrow. Due to its automatic nature, I see this processing as taking more of the creative part of photographer’s role, unlike compute in cameras where it is a capture-time assistant. It will have to play a bigger and bigger role in how phone camera output improves in near future, as there is not that much space for enlarging the sensor or the lens; and as a user you either let it do the magic or you end up with a very mediocre camera. It is not inherently better or worse, but it relates to photography in the sense that I appreciate it in kind of as Nvidia Canvas et al. relate to hand-created art: different tools, different goals.
  - orbital-decay 5y ago
    It still seems we're talking about completely different things, or maybe you're trying to derail the conversation... Both ML-based processing and tonemapping take place after demosaicing to produce the final JPEG. RAW capture is just your typical laplacian pyramid stacking/exposure fusion with motion compensation and sensor-specific calibration. That's the main contributing factor that somewhat compensates for the sensor size, not some kind of ML processing. If you think that ML is somehow involved here, I have zero idea where you are getting it from. The entire point of pre-demosaicing pipeline is to robustly converge the captured stack into a neutral scene-referred baseline with provably better SNR and precise photometric parameters. And that's what you get in a resulting DNG, not some neural tonemapped RGB image (which is not even created at this point). Perhaps you should look at the pipeline yourself, here's the source code for a typical "computational imaging" camera app that produces stacked RAWs, for example. [1] And here's what Google Camera does [2] [3] (you can ignore everything that comes after demosaicing). Google's particular method is also partially implemented for arbitrary RAWs as third-party software, so you can stack any burst capture from your camera with the same method smartphones do. [4] (there's much less benefit for normal sized sensors, for obvious reasons) [1] https://github.com/mirsadm/motioncam/ https://github.com/mirsadm/motioncam/ [2] https://static.googleusercontent.com/media/hdrplusdata.org/en//hdrplus.pdf https://static.googleusercontent.com/media/hdrplusdata.org/e... (PDF) [3] https://static.googleusercontent.com/media/hdrplusdata.org/en//hdrplus_supp.pdf https://static.googleusercontent.com/media/hdrplusdata.org/e... (PDF) [4] https://github.com/martin-marek/hdr-plus-swift https://github.com/martin-marek/hdr-plus-swift

orbital-decay 5y ago

My main objection is this: >able to produce decent images despite hardware limitations by applying black-box magic when interpreting raw captures (but not faring that well when used to shoot raw) Not really, the quality boost in recent smartphones comes exactly from bayer domain stacking, which is scene-referred capture. You can absolutely get precise photometric values from this process, as precise as your hardware and lighting allows you to, and as long as you calibrate your device. Those apps contain profiles for specific sensors precisely for this reason, albeit you can do it yourself better, as many people do just like they do it with proper cameras. There's no black box magic here, the output is just the same ordinary DNG with the CFA photometric interpretation, but with significantly lowered noise floor and better dynamic range. Moreover, the pipeline is strictly separated into two parts: bayer domain operations, which give the quality boost, and post-demosaicing processing (auto white balance, tonemapping, skin smoothing, other features aimed at point-and-shoot users - what you call magic). I'm not claiming smartphones are somehow better, which would be silly. I'm objecting to a claim that apps do some opaque magic behind your back when writing RAWs. They don't, you can get entirely normal RAWs with all the quality benefits of stacking (and its limitations). And to be honest, I feel like our discussion is creeping into a no true scotsman/audiophile territory a bit. There's nothing sacred or qualitatively different about single-shot capture, no modern normal-sized sensor gives you truly unprocessed internal bucket values, and even what is counted as single shot is often unclear (many sensors are capable of simultaneous multiple exposure capture requiring the same exposure fusion to get the result; is it single shot or not?). What smartphones, and some cameras, do is essentially temporal denoising, it's a routine imaging technique employed by videographers and in technical imaging (which is as precise and scene-referred as you can get), and I'm not sure what's magical here. Same with temporal super-resolution that many cameras do.

goblin89 5y ago
Sure, some of the black box processing that phone camera does relies on known algorithms. The difference is that it also makes use of ML, which IMO either plays a significant role or will do so in near term. > auto white balance, tonemapping, skin smoothing, other features This, for example, I see as an opaque step with nondeterministic outcomes, which if you take away from phones and shoot raw reveals how far they are away from mirrorless cameras. (Cameras that shoot raw do not do this all that well, nor are they expected to.) > There's nothing sacred or qualitatively different about single-shot capture I stack exposures too. Whatever process works, works. The difference is that my flow is A -> N -> X, my camera’s built-in processing is A -> M -> Y, while ML-powered black-box compute in a phone does A -> ? -> R today or A -> ? -> Z tomorrow. Due to its automatic nature, I see this processing as taking more of the creative part of photographer’s role, unlike compute in cameras where it is a capture-time assistant. It will have to play a bigger and bigger role in how phone camera output improves in near future, as there is not that much space for enlarging the sensor or the lens; and as a user you either let it do the magic or you end up with a very mediocre camera. It is not inherently better or worse, but it relates to photography in the sense that I appreciate it in kind of as Nvidia Canvas et al. relate to hand-created art: different tools, different goals.
- orbital-decay 5y ago
  It still seems we're talking about completely different things, or maybe you're trying to derail the conversation... Both ML-based processing and tonemapping take place after demosaicing to produce the final JPEG. RAW capture is just your typical laplacian pyramid stacking/exposure fusion with motion compensation and sensor-specific calibration. That's the main contributing factor that somewhat compensates for the sensor size, not some kind of ML processing. If you think that ML is somehow involved here, I have zero idea where you are getting it from. The entire point of pre-demosaicing pipeline is to robustly converge the captured stack into a neutral scene-referred baseline with provably better SNR and precise photometric parameters. And that's what you get in a resulting DNG, not some neural tonemapped RGB image (which is not even created at this point). Perhaps you should look at the pipeline yourself, here's the source code for a typical "computational imaging" camera app that produces stacked RAWs, for example. [1] And here's what Google Camera does [2] [3] (you can ignore everything that comes after demosaicing). Google's particular method is also partially implemented for arbitrary RAWs as third-party software, so you can stack any burst capture from your camera with the same method smartphones do. [4] (there's much less benefit for normal sized sensors, for obvious reasons) [1] https://github.com/mirsadm/motioncam/ https://github.com/mirsadm/motioncam/ [2] https://static.googleusercontent.com/media/hdrplusdata.org/en//hdrplus.pdf https://static.googleusercontent.com/media/hdrplusdata.org/e... (PDF) [3] https://static.googleusercontent.com/media/hdrplusdata.org/en//hdrplus_supp.pdf https://static.googleusercontent.com/media/hdrplusdata.org/e... (PDF) [4] https://github.com/martin-marek/hdr-plus-swift https://github.com/martin-marek/hdr-plus-swift