8 ms·

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of ph

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowledged by that understanding. The technology is good, but it won't change the way photographers deal with photography, at best, it will give them a better tool to work with. I also wonder how much impact this could have. Foveon Inc. had developed a great sensor that was going to be revolutionary, about 20 years ago. Today only Sigma uses it, and the UX is so bad in those cameras that the (very good) technology they had their hands on never got a chance to shine. Also, is this implementation very different from having a 16-bit ADC in normal sensors? The end results would be the same: higher dynamic range. And the increase in bits seems a lot closer (because it would mean little changes to the current manufacture processes) than a whole new type of sensor being use in cameras.

kps 11y ago
Sure, photographs will still control aperture and exposure time — this just removes a constraint at one end: if you want shallow depth of field and motion blur, you could use this method (and get extra bits) rather than an ND filter.
- StavrosK 11y ago
  Exactly. This is huge, and I was wondering why people didn't do it already. Or, for example, why don't cameras currently take multiple snapshots of the sensor? For example, if I'm exposing for 2 seconds, read the sensor at 0.1 sec (for the very bright lights) and again at 2 seconds, for the darker parts. That way, I can make an HDR image without having to take another photo. Is reading the sensor destructive to the data, somehow?
  - EvanAnderson 11y ago
    Readout of a CCD is destructive. There are types of sensors that have non-destructive readout (Charge Injection Devices being one example: https://www.cis.rit.edu/research/CID/a_cid_is.htm https://www.cis.rit.edu/research/CID/a_cid_is.htm).
    - StavrosK 11y ago
      Very interesting, thank you.
    - kps 11y ago
      A sensor with per-pixel reset to zero on readout would seem to be ideal — then you could get cumulative counts without wasting valuable photon landing space on the die.
    - kuschku 11y ago
      Okay, but can’t I take 8 raw images and merge them into one image with 8 times higher dynamic range?
      - KingMob 11y ago
        Yes, but if I understand correctly, unless those images occur at the same time (or really, really close) you run into issues involving subject and camera motion.
      - altcognito 11y ago
        Yes, that's what modern cameras with the built in feature do. (although I'm not sure about 8 times) The article talks about this feature.
        kuschku 11y ago
        Yes, but why can’t the camera do it continuously? Film at 240fps and provide a 60fps output with 10bit color depth.
        11y ago
- jpablo 11y ago
  That's just really a completely misguided statement. No mater how much you fumble with aperture and shutter speed there's simply no way to expand the camera sensor dynamic range without taking multiple exposures. Besides I have never seen "ordinary people" fumble with aperture size and exposure length so that's a solved problem. Sometimes it gives "wrong" results, like exposing for the highlights rather than the shadows, that's what this technology solves.
  - kps 11y ago
    > there's simply no way to expand the camera sensor dynamic range without taking multiple exposures That's the point of this paper†. They expand dynamic range by having the detectors wrap around (discarding high bits) and then recover the high bits computationally. † here http://web.media.mit.edu/~hangzhao/modulo.html http://web.media.mit.edu/~hangzhao/modulo.html (linked by putterson in another comment)
    - jpablo 11y ago
      I know, I just don't see how it solves this issue. You couldn't solve the issue before "fumbling with aperture size and exposure length" The only thing I would consider that solves "fumble with aperture size and exposure length" would be huge ISO range (both ways) without any noise. > No more will photographers or even ordinary people have to fumble with aperture size and exposure length.
      - wanderingstan 11y ago
        > No mater how much you fumble with aperture and shutter speed there's simply no way to expand the camera sensor dynamic range without taking multiple exposures. > I just don't see how it solves this issue. http://rationalwiki.org/wiki/Argument_from_incredulity http://rationalwiki.org/wiki/Argument_from_incredulity
  - dr_zoidberg 11y ago
    Some DSLRs and MILCs (and I think some semi-pro compacts too) have a wide enough dynamic range that they can work and HDR image from a single RAW file. Catch is, you have to shoot RAW and then process it with specific software. When it's done in-camera, it always is through multiple exposures. Even without specific software, you can work the effect from a single RAW file to get "multiple exposures" from it and then combine them into an HDR image.
trop 11y ago
> is this implementation very different from having a 16-bit ADC in normal sensors 16-bit ADCs increases tonal granularity in the readout, which can help in the darker tones. A sensor site may be capable of differentiating about 12 bits worth of meaningful information, hence there's a good argument for overshooting that in the ADC. Our eyes, even if not sensitive to more than 8 bits worth of tonal gradation, are perceiving in a logarithmic space (vs. the sensor recording in a linear space), hence extra information in the shadows can be quite good. Plus the extra bits give more room in which to tune a captured image. An article such as http://www.scantips.com/basics14.html http://www.scantips.com/basics14.html gives a rundown on this. The extra bits don't help if the incoming light exceeds the sensor's recording capacity, hence the interest of this "modulo" discussion for over-exposed images.
ISL 11y ago
It sounds from the implementation summary that it's equivalent to a higher bit-depth ADC, but that the higher-value bits are inferred by software somehow. I don't think it would properly recover an image with a dark field with the exception of a single pixel with a value of 10^7 above the dark noise. What would truly solve the dynamic-range problem is a logarithmic readout CCD. (Thanks for the Foveon pointer; it's an elegant approach to color.)
- paulmd 11y ago
  It's actually inferred by hardware - the paper handwaves it because they wanted to focus on the software, but basically they're talking about implementing a low-accuracy micro-ADC at every site that automatically wipes the charge when it hits a trigger voltage and counts every time it does so. Number of counts = site signal level. Not quite sure that's practical or how you'd implement it on a destructive-readout sensor, though. Right now the sensors with enough accuracy and sensitivity are all destructive.
- coldtea 11y ago
  >I don't think it would properly recover an image with a dark field with the exception of a single pixel with a value of 10^7 above the dark noise. Then again it doesn't matter, because that's a contrived example (the photography equivalent of a "microbenchmark").
dharma1 11y ago
The Foveon sensor is different - the claimed benefit there has to do with stacked silicon and lack of debayering, and the benefit is having better resolving power with the same pixel count. Sony is now researching similar stacked chips so we'll probably see this idea return. The innovation with the tech in the article is in the dynamic range, current CMOS sensors have a fairly limited maximum dynamic range because of the well saturation described in the article, usually around 12-14ev DR. For high contrast situations that isn't enough, and photographers will definitely appreciate increased dynamic range, not to mention the other applications they listed, like astronomy. On the display side most displays also have fairly limited DR and cannot reproduce the dynamic range in HDR images, so we end up having to tonemap images to get the perceived dynamic range back - however I think HDR monitors will break through in the next couple of years as many HDR TVs are hitting the market this year.
- ancientworldnow 11y ago
  Worth noting that those "HDR" televisions are still only Rec709 which is limited to about 6 stops of dynamic range. Sure some are approaching rec2020 color gamut, but we're still cramming all that gamma down into 6-9 stops. Even a consumer DSLR can shoot 12. I work in post and have moved our workflow over to the Academy Color Encoding Specification (ACES) which supports up to 27 stops (more than the eye can see which is about 24 best case scenario). ACES allows us to dynamically change our output to rec709, rec2020, P3 etc and while 2020 has better colors, it's definitely still a highly compressed image (in terms of dynamic range).
  - dharma1 11y ago
    Interesting, will look into ACES. What do you use for the files, openEXR? Yep first "HDR" tv's will be rec709 but higher DR is definitely coming, both to home TVs and cinemas (via super bright laser projectors) http://blog.abelcine.com/2015/03/06/high-dynamic-range-explained/ http://blog.abelcine.com/2015/03/06/high-dynamic-range-expla... I'm actually not convinced most people will want 5-100x brighter pictures than we currently have! Dynamic range of the human eye is also related to foveal/peripheral vision, so dynamic range in the real world probably doesn't map 1:1 to a small screen experience - if that makes sense. http://www.rags-int-inc.com/PhotoTechStuff/CameraEye/ http://www.rags-int-inc.com/PhotoTechStuff/CameraEye/ I am very interested in "natural", halo free tone mapping algorithms, squeezing that extended DR into current displays. I really like the results Adobe is getting with laplacian pyramids in Camera Raw for highlight/shadow recovery. Do you know of other tone mapping algos that look natural?
Tloewald 11y ago
> This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. The actual headline of the article is actually more informative and less misleading than the subhead: "Unbounded High Dynamic Range Photography Using a Modulo Camera" In photographer's terms, they're referring to ending blown highlights (caused by filling the wells in a sensel) not saturated images (caused by exaggerating color data). There's a school of photography that basically lives and dies by over-saturating images (e.g. http://kenrockwell.com http://kenrockwell.com ) and they'd likely have a cow over the headline, but understand and be totally onboard for the actual goal. Depending on how quickly the well can reset and how many times resets can be counted this will either be better or worse than simply improving existing approaches. The state of the art in full frame sensors is just under 15-bits of dynamic range. So to equal it an 8-bit sensor would need to be able to count resets at least 128 times AND be able to do this in 1/8000 of a second, at the same sensel density and efficiency! Don't hold your breath. http://www.dxomark.com/Cameras/Nikon/D810 http://www.dxomark.com/Cameras/Nikon/D810 (I'd also suggest that trends towards using multiple sensors to assemble high resolution images will easily blow this away since they can use high- and low- sensitivity sensors to synthesize more dynamic range and resolution. Also see recent Olympus patents to capture polarization data at the same time.) BTW: assuming they can do this stuff, they presumably can read the sensor pretty darn fast — so they should be able to eliminate image "tearing" in digital video and the need for mechanical shutters altogether. That's probably a bigger issue than dynamic range. I suspect that clearing a well in < 1/10,000,000 of a second (which would only lose 10% of the photon detection time during a 1/8000s exposure) is going to be tough. Assuming 25% of the sensel real estate needs to be sacrificed to the modulo circuitry, that's a loss of 32.5% of photon data which is about the same loss as for pellicle mirrors (as seen in Sony's pretty unsuccessful SLT cameras) for a feature that won't be much use in many situations.
- paulmd 11y ago
  I identified the same issue with cycle time, but to me the biggest problem is the assumption of an accurate, sensitive, non-destructive-read sensor. They're assuming that they can charge a capacitor from a photodiode accurately, and also hook a voltage comparator (which is effectively the same as an ADC) to it constantly to get an accurate pulse to count. Assuming you have an accurate sensor that can be read non-destructively, then none of this modulus stuff is necessary anyway. Read once to set the proper gain on the ADC, then read again to get the fine bits. That way you still get the "unbounded HDR", but you don't need an ADC for every single receptor site, and you don't waste space on the counter or comparator. Instead, all of that lives on the image processor where it's not going to cause heat noise. That's such an incredibly enormous hand-wave to me that I feel I've got to be missing something. Why make it so complex with all the modulus stuff if we have a magic sensor?
mortenjorck 11y ago
I'm surprised no one in the comments so far has used the word "clipping", which is what the real breakthrough here is in. I've been doing street photography somewhat seriously for about five years now, and always shoot full-manual: I rarely have problems with motion blur because I control the shutter speed; I rarely have problems with focus because I use assisted manual; the only thing that gets me with any regularity is highlight or shadow clipping. I often shoot with a histogram overlay which helps a lot, but there are still high-contrast scenes where I have to decide where I'm going to bias the exposure. I can usually tease out some more detail in Lightroom from the RAW file, but direct sunlight plus shadow, or a quick-draw moment where I fumbled the exposure to make the shot, often leave some clipped areas. Not having to worry about clipping would be an amazing breakthrough, and I'd even accept some other limitations in a camera that could manage this.
- SonicSoul 11y ago
  Being about year in with street photography this is very insightful. Can you elaborate on any tricks you do in Lightroom in this area ??
- semi-extrinsic 11y ago
  FWIW, analog film doesn't clip in the same way as digital, but instead has a nonlinear roll-of on either end of the brightness scale. Still there is a limit to the maximum contrast, of course. If you're interested, try Tri-X for BW and Fuji Velvia 50 for color. Any late 70s Nikon will do well as a camera, the old 50mm f1.4 is particularly good and cheap. Tri-X is best developed yourself, but the Velvia needs sending to a lab that specializes in positive film development.

kps 11y ago

Sure, photographs will still control aperture and exposure time — this just removes a constraint at one end: if you want shallow depth of field and motion blur, you could use this method (and get extra bits) rather than an ND filter.

StavrosK 11y ago
Exactly. This is huge, and I was wondering why people didn't do it already. Or, for example, why don't cameras currently take multiple snapshots of the sensor? For example, if I'm exposing for 2 seconds, read the sensor at 0.1 sec (for the very bright lights) and again at 2 seconds, for the darker parts. That way, I can make an HDR image without having to take another photo. Is reading the sensor destructive to the data, somehow?
- EvanAnderson 11y ago
  Readout of a CCD is destructive. There are types of sensors that have non-destructive readout (Charge Injection Devices being one example: https://www.cis.rit.edu/research/CID/a_cid_is.htm https://www.cis.rit.edu/research/CID/a_cid_is.htm).
  - StavrosK 11y ago
    Very interesting, thank you.
  - kps 11y ago
    A sensor with per-pixel reset to zero on readout would seem to be ideal — then you could get cumulative counts without wasting valuable photon landing space on the die.
  - kuschku 11y ago
    Okay, but can’t I take 8 raw images and merge them into one image with 8 times higher dynamic range?
    - KingMob 11y ago
      Yes, but if I understand correctly, unless those images occur at the same time (or really, really close) you run into issues involving subject and camera motion.
    - altcognito 11y ago
      Yes, that's what modern cameras with the built in feature do. (although I'm not sure about 8 times) The article talks about this feature.
      - kuschku 11y ago
        Yes, but why can’t the camera do it continuously? Film at 240fps and provide a 60fps output with 10bit color depth.
        11y ago
jpablo 11y ago
That's just really a completely misguided statement. No mater how much you fumble with aperture and shutter speed there's simply no way to expand the camera sensor dynamic range without taking multiple exposures. Besides I have never seen "ordinary people" fumble with aperture size and exposure length so that's a solved problem. Sometimes it gives "wrong" results, like exposing for the highlights rather than the shadows, that's what this technology solves.
- kps 11y ago
  > there's simply no way to expand the camera sensor dynamic range without taking multiple exposures That's the point of this paper†. They expand dynamic range by having the detectors wrap around (discarding high bits) and then recover the high bits computationally. † here http://web.media.mit.edu/~hangzhao/modulo.html http://web.media.mit.edu/~hangzhao/modulo.html (linked by putterson in another comment)
  - jpablo 11y ago
    I know, I just don't see how it solves this issue. You couldn't solve the issue before "fumbling with aperture size and exposure length" The only thing I would consider that solves "fumble with aperture size and exposure length" would be huge ISO range (both ways) without any noise. > No more will photographers or even ordinary people have to fumble with aperture size and exposure length.
    - wanderingstan 11y ago
      > No mater how much you fumble with aperture and shutter speed there's simply no way to expand the camera sensor dynamic range without taking multiple exposures. > I just don't see how it solves this issue. http://rationalwiki.org/wiki/Argument_from_incredulity http://rationalwiki.org/wiki/Argument_from_incredulity
- dr_zoidberg 11y ago
  Some DSLRs and MILCs (and I think some semi-pro compacts too) have a wide enough dynamic range that they can work and HDR image from a single RAW file. Catch is, you have to shoot RAW and then process it with specific software. When it's done in-camera, it always is through multiple exposures. Even without specific software, you can work the effect from a single RAW file to get "multiple exposures" from it and then combine them into an HDR image.

trop 11y ago

> is this implementation very different from having a 16-bit ADC in normal sensors 16-bit ADCs increases tonal granularity in the readout, which can help in the darker tones. A sensor site may be capable of differentiating about 12 bits worth of meaningful information, hence there's a good argument for overshooting that in the ADC. Our eyes, even if not sensitive to more than 8 bits worth of tonal gradation, are perceiving in a logarithmic space (vs. the sensor recording in a linear space), hence extra information in the shadows can be quite good. Plus the extra bits give more room in which to tune a captured image. An article such as http://www.scantips.com/basics14.html http://www.scantips.com/basics14.html gives a rundown on this. The extra bits don't help if the incoming light exceeds the sensor's recording capacity, hence the interest of this "modulo" discussion for over-exposed images.

ISL 11y ago

It sounds from the implementation summary that it's equivalent to a higher bit-depth ADC, but that the higher-value bits are inferred by software somehow. I don't think it would properly recover an image with a dark field with the exception of a single pixel with a value of 10^7 above the dark noise. What would truly solve the dynamic-range problem is a logarithmic readout CCD. (Thanks for the Foveon pointer; it's an elegant approach to color.)

paulmd 11y ago
It's actually inferred by hardware - the paper handwaves it because they wanted to focus on the software, but basically they're talking about implementing a low-accuracy micro-ADC at every site that automatically wipes the charge when it hits a trigger voltage and counts every time it does so. Number of counts = site signal level. Not quite sure that's practical or how you'd implement it on a destructive-readout sensor, though. Right now the sensors with enough accuracy and sensitivity are all destructive.
coldtea 11y ago
>I don't think it would properly recover an image with a dark field with the exception of a single pixel with a value of 10^7 above the dark noise. Then again it doesn't matter, because that's a contrived example (the photography equivalent of a "microbenchmark").

dharma1 11y ago

The Foveon sensor is different - the claimed benefit there has to do with stacked silicon and lack of debayering, and the benefit is having better resolving power with the same pixel count. Sony is now researching similar stacked chips so we'll probably see this idea return. The innovation with the tech in the article is in the dynamic range, current CMOS sensors have a fairly limited maximum dynamic range because of the well saturation described in the article, usually around 12-14ev DR. For high contrast situations that isn't enough, and photographers will definitely appreciate increased dynamic range, not to mention the other applications they listed, like astronomy. On the display side most displays also have fairly limited DR and cannot reproduce the dynamic range in HDR images, so we end up having to tonemap images to get the perceived dynamic range back - however I think HDR monitors will break through in the next couple of years as many HDR TVs are hitting the market this year.

ancientworldnow 11y ago
Worth noting that those "HDR" televisions are still only Rec709 which is limited to about 6 stops of dynamic range. Sure some are approaching rec2020 color gamut, but we're still cramming all that gamma down into 6-9 stops. Even a consumer DSLR can shoot 12. I work in post and have moved our workflow over to the Academy Color Encoding Specification (ACES) which supports up to 27 stops (more than the eye can see which is about 24 best case scenario). ACES allows us to dynamically change our output to rec709, rec2020, P3 etc and while 2020 has better colors, it's definitely still a highly compressed image (in terms of dynamic range).
- dharma1 11y ago
  Interesting, will look into ACES. What do you use for the files, openEXR? Yep first "HDR" tv's will be rec709 but higher DR is definitely coming, both to home TVs and cinemas (via super bright laser projectors) http://blog.abelcine.com/2015/03/06/high-dynamic-range-explained/ http://blog.abelcine.com/2015/03/06/high-dynamic-range-expla... I'm actually not convinced most people will want 5-100x brighter pictures than we currently have! Dynamic range of the human eye is also related to foveal/peripheral vision, so dynamic range in the real world probably doesn't map 1:1 to a small screen experience - if that makes sense. http://www.rags-int-inc.com/PhotoTechStuff/CameraEye/ http://www.rags-int-inc.com/PhotoTechStuff/CameraEye/ I am very interested in "natural", halo free tone mapping algorithms, squeezing that extended DR into current displays. I really like the results Adobe is getting with laplacian pyramids in Camera Raw for highlight/shadow recovery. Do you know of other tone mapping algos that look natural?

Tloewald 11y ago

> This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. The actual headline of the article is actually more informative and less misleading than the subhead: "Unbounded High Dynamic Range Photography Using a Modulo Camera" In photographer's terms, they're referring to ending blown highlights (caused by filling the wells in a sensel) not saturated images (caused by exaggerating color data). There's a school of photography that basically lives and dies by over-saturating images (e.g. http://kenrockwell.com http://kenrockwell.com ) and they'd likely have a cow over the headline, but understand and be totally onboard for the actual goal. Depending on how quickly the well can reset and how many times resets can be counted this will either be better or worse than simply improving existing approaches. The state of the art in full frame sensors is just under 15-bits of dynamic range. So to equal it an 8-bit sensor would need to be able to count resets at least 128 times AND be able to do this in 1/8000 of a second, at the same sensel density and efficiency! Don't hold your breath. http://www.dxomark.com/Cameras/Nikon/D810 http://www.dxomark.com/Cameras/Nikon/D810 (I'd also suggest that trends towards using multiple sensors to assemble high resolution images will easily blow this away since they can use high- and low- sensitivity sensors to synthesize more dynamic range and resolution. Also see recent Olympus patents to capture polarization data at the same time.) BTW: assuming they can do this stuff, they presumably can read the sensor pretty darn fast — so they should be able to eliminate image "tearing" in digital video and the need for mechanical shutters altogether. That's probably a bigger issue than dynamic range. I suspect that clearing a well in < 1/10,000,000 of a second (which would only lose 10% of the photon detection time during a 1/8000s exposure) is going to be tough. Assuming 25% of the sensel real estate needs to be sacrificed to the modulo circuitry, that's a loss of 32.5% of photon data which is about the same loss as for pellicle mirrors (as seen in Sony's pretty unsuccessful SLT cameras) for a feature that won't be much use in many situations.

paulmd 11y ago
I identified the same issue with cycle time, but to me the biggest problem is the assumption of an accurate, sensitive, non-destructive-read sensor. They're assuming that they can charge a capacitor from a photodiode accurately, and also hook a voltage comparator (which is effectively the same as an ADC) to it constantly to get an accurate pulse to count. Assuming you have an accurate sensor that can be read non-destructively, then none of this modulus stuff is necessary anyway. Read once to set the proper gain on the ADC, then read again to get the fine bits. That way you still get the "unbounded HDR", but you don't need an ADC for every single receptor site, and you don't waste space on the counter or comparator. Instead, all of that lives on the image processor where it's not going to cause heat noise. That's such an incredibly enormous hand-wave to me that I feel I've got to be missing something. Why make it so complex with all the modulus stuff if we have a magic sensor?

mortenjorck 11y ago

I'm surprised no one in the comments so far has used the word "clipping", which is what the real breakthrough here is in. I've been doing street photography somewhat seriously for about five years now, and always shoot full-manual: I rarely have problems with motion blur because I control the shutter speed; I rarely have problems with focus because I use assisted manual; the only thing that gets me with any regularity is highlight or shadow clipping. I often shoot with a histogram overlay which helps a lot, but there are still high-contrast scenes where I have to decide where I'm going to bias the exposure. I can usually tease out some more detail in Lightroom from the RAW file, but direct sunlight plus shadow, or a quick-draw moment where I fumbled the exposure to make the shot, often leave some clipped areas. Not having to worry about clipping would be an amazing breakthrough, and I'd even accept some other limitations in a camera that could manage this.

SonicSoul 11y ago
Being about year in with street photography this is very insightful. Can you elaborate on any tricks you do in Lightroom in this area ??
semi-extrinsic 11y ago
FWIW, analog film doesn't clip in the same way as digital, but instead has a nonlinear roll-of on either end of the brightness scale. Still there is a limit to the maximum contrast, of course. If you're interested, try Tri-X for BW and Fuji Velvia 50 for color. Any late 70s Nikon will do well as a camera, the old 50mm f1.4 is particularly good and cheap. Tri-X is best developed yourself, but the Velvia needs sending to a lab that specializes in positive film development.