9 ms·

Unbounded High Dynamic Range Photography Using a Modulo Camera

140 points by dnt404-1 11y ago

jcr 11y ago
The original title of the article is, "Unbounded High Dynamic Range Photography Using a Modulo Camera"
Mithaldu 11y ago
What's with the completely useless thumbnail at the bottom? That said, the technological explanation sounds simple and solid. Why aren't cameras doing that already?
- putterson 11y ago
  I wondered the same; it takes two hyperlinks to get to the project's home page where they have a larger version. http://web.media.mit.edu/~hangzhao/modulo.html http://web.media.mit.edu/~hangzhao/modulo.html
  - Mithaldu 11y ago
    Nice, thank you!
- vlamingsjef 11y ago
  High resolution image here: http://web.media.mit.edu/~hangzhao/images/modulo_teaser.png http://web.media.mit.edu/~hangzhao/images/modulo_teaser.png
  - stevensanderson 11y ago
    Note that this is a simulated example, not one actually taken with a modulo camera (if such a thing has been built yet). That image of Tintern Abbey comes from this 2011 web page: http://people.csail.mit.edu/sparis/publi/2011/siggraph/additional_results/tone_mapping_large/tinterna.html http://people.csail.mit.edu/sparis/publi/2011/siggraph/addit...
jchrisa 11y ago
Reminds me of the implementation of the TRNG from yesterday's front page.
darkmighty 11y ago
The title here is terrible because over-saturation is an effect of poor tone mapping, which is inevitable if you want to display an HDR image in a low dynamic range display. An accurate title would be the end of over-exposed images, but why not just keep the original?
- kps 11y ago
  Two different senses of the word ‘saturation’. The subtitle of the article (current title here) is unfortunate.
  - tyingq 11y ago
    Yes. I expected to see an article about stock images being overused across lots of publications, and whatever the fix for that is.
- IshKebab 11y ago
  They're talking about the pixel brightness values getting saturated (255), not colour saturation. But I've never heard over-saturation being used to refer to brightness saturation - brightness is either saturated or not.
  - ssalazar 11y ago
    I'd guess whoever wrote the article subtitle comes from an EE or DSP background, where saturation refers to blowing out the ends of the dynamic range and clipping/distorting.
- sctb 11y ago
  We updated the submission title to the article's.
soylentcola 11y ago
Looks to be more related to exposure than saturation but still, it's an interesting new take on dynamic range. I'm more a hobbyist than an expert but current tone mapping techniques (at least anything mostly automated) can often leave you with "halos" around objects where the software has feathered the edges between lighter and darker areas. Then the article mentions the issues with using multiple shots for tone mapping. I still try to just expose "properly" for the effect I want and shoot RAW to give me a little leeway in tweaking shadows and highlights but I'd be interested in trying out software that uses this technique if only for something new to mess with.
- darkmighty 11y ago
  Halos are a problem, but they can mostly be removed with good post-processing. Note they're not a fundamental aspect of tone mapping. Tone mapping is just whatever algorithm you chose to compress the HDR information into a low dynamic range we have for displays. The great thing about this camera compared to multi-exposure HDR (apart from the halo effect you mentioned) is you can actually take pictures of moving objects or even make HDR movies (the temporal variation should actually help with better phase unwrapping). That would be amazing for the film industry I think.
  - dharma1 11y ago
    You can actually already do this with current cameras, using 2 exposures per frame, works for most types of action. Red Epic does this with HDR-X and hacked Canon cameras with Magic Lantern with Dual ISO
    - darkmighty 11y ago
      As explained in the article, this is different to combining exposures, which is what conventional HDR does too. The important thing to consider here is how does this scheme affect the SNR of the sensor. This scheme encourages keeping essentially maximum possible sensitivity (ISO), but this "modulo sensor" circuit may introduce some additional noise.
  - 11y ago
baldeagle 11y ago
It seems like this would make big glass even more important. This solution protects for over exposure, but doesn't address under exposure on a scene that moves (think indoor bar photography). So it isn't a end all solution that will everyone capture reality with their cell phones, but it is an awesome step in that direction.
dr_zoidberg 11y ago
> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowledged by that understanding. The technology is good, but it won't change the way photographers deal with photography, at best, it will give them a better tool to work with. I also wonder how much impact this could have. Foveon Inc. had developed a great sensor that was going to be revolutionary, about 20 years ago. Today only Sigma uses it, and the UX is so bad in those cameras that the (very good) technology they had their hands on never got a chance to shine. Also, is this implementation very different from having a 16-bit ADC in normal sensors? The end results would be the same: higher dynamic range. And the increase in bits seems a lot closer (because it would mean little changes to the current manufacture processes) than a whole new type of sensor being use in cameras.
- kps 11y ago
  Sure, photographs will still control aperture and exposure time — this just removes a constraint at one end: if you want shallow depth of field and motion blur, you could use this method (and get extra bits) rather than an ND filter.
  - StavrosK 11y ago
    Exactly. This is huge, and I was wondering why people didn't do it already. Or, for example, why don't cameras currently take multiple snapshots of the sensor? For example, if I'm exposing for 2 seconds, read the sensor at 0.1 sec (for the very bright lights) and again at 2 seconds, for the darker parts. That way, I can make an HDR image without having to take another photo. Is reading the sensor destructive to the data, somehow?
    - EvanAnderson 11y ago
      Readout of a CCD is destructive. There are types of sensors that have non-destructive readout (Charge Injection Devices being one example: https://www.cis.rit.edu/research/CID/a_cid_is.htm https://www.cis.rit.edu/research/CID/a_cid_is.htm).
kefka 11y ago
I had an idea similar to this that solved the oversaturation issue. Use 2 cameras. Have them at right angles to the shutter point, separated a few inches. At the T junction where the camera pointing would cross with the shutter area, put a prism there. The prism would give each camera 1/2 of the available photons. Now use any standard HDR processing technique.
- rasz_pl 11y ago
  prism = less than half signal at each end
- paulmd 11y ago
  Interestingly enough that's how Technicolor used to work. Three separate exposures from a single lens, split off by pellicle mirrors. The downside is that like any pellicle you are dividing your light between multiple exposures, meaning you some mixture of a longer exposure, a faster lens, or more lighting. Technicolor required special high-intensity arc lighting. Also, the pellicles must be kept spotless because dust shows up on the final image. Next-gen sensors will actually implement this physically - some receptor sites will be larger than others.
  - kefka 11y ago
    Admittedly, my understanding of traditional optics is very limited. I happened upon it in a dream whilst trying to figure out how to handle dynamic scenes while using SLAM. I was thinking of a prism (new word: pellicle mirror) like this one : http://static1.squarespace.com/static/54ca877ce4b014ea90e14bda/54ca9f5de4b021f8b6d68cc1/54caa186e4b021f8b6d6c6b8/1422565766946/penta_3.png?format=original http://static1.squarespace.com/static/54ca877ce4b014ea90e14b... Except I don't need the deconvolutional prism. I can handle that in software easily with a X or Y matrix flip. And I assume we are using the same CCD module for both areas; given that, I can sum the contrast between the pixels and calculate the min and max, as well as can calculate the HDR image from both sources. Calibration would be easy to handle, using a checkerboard to link the CCDs to each other (assume aligning errors will happen). I also see no reason why the pellicles wouldn't be kept clean. The same goes for the CCDs as well. I'd embed them in a dark acrylic box and try to seal them as best as can be.
brador 11y ago
This was the runner up prize winning paper, what won?
- Joeboy 11y ago
  https://twitter.com/ICCP_2015/status/592403545468448768/photo/1 https://twitter.com/ICCP_2015/status/592403545468448768/phot...
  - mjcohen 11y ago
    That's a picture. Where's the paper?
    - amritamaz 11y ago
      Here's the paper link: http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_ICCP15.pdf http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_ICC...
dharma1 11y ago
Read about something like this on http://image-sensors-world.blogspot.co.uk http://image-sensors-world.blogspot.co.uk a while ago - instead of saturating the CMOS when max brightness is exceeded, it just keeps going. Looking forward to seeing this commercialised.
paulmd 11y ago
Here's an idea, instead of separately tracking the number of "resets" we just add those bits on to the left hand side of the measurement. We could call it... having more bits in the ADC. Expose for the highlights and then pull more detail out of the shadows with your extra bits of ADC precision. There are not 1 million ADC units in a 1-megapixel camera, there's a few ADCs (a Canon 7D has 2 image processors with 4 ADCs each) that are iterated across the CCD sites to progressively read them out. To make this new sensor, you need to cram 50 million sets (50mp is state of the art) of voltage comparator/charge reset/digital counter circuits onto the CCD, and for N bits of rollover accuracy they need to be able to trigger, erase the charge, and resume exposure at least N times during the exposure time E (which is say 1/500th of a second, since they're complaining about movement during multiple HDR exposures). Practically speaking the trigger/operation/resume period must be significantly less than E/N since I see no way to retain the exposure during the period when the charge well is being drained to zero. If this time is non-trivial, that translates to losing the fine bits of your ADC accuracy. In their image they compare a 13-stop exposure (the current state of the art) to an 8-stop exposure (state of the art in 1900). So assuming they didn't just pull a photoshop out of their ass, they are overall claiming a 5 stop increase from state of the art. That's 5 extra bits of recovery (1 stop is double the range, i.e. an extra bit), so they think they can trigger the reset circuit 5 times during an exposure. That implies this circuit must have a minimum cycle time of 1/2500th of a second (practically speaking a lot less to give time for the actual exposure). With this math I'm also assuming that the comparator is perfect and doesn't lose any accuracy - variance in trigger threshold or trigger time translates to losing some of your accuracy again. They also need to generate little enough waste heat to avoid hot pixels (this is a major reason the ADC is on the image processor rather than the CCD), and you need 50 million sets of these on the CCD. If you can do it then go for it. It's a nice idea on paper but I think there's a lot of physical obstacles to overcome. If it were that easy someone would have done it already. It's especially difficult given that reading out the sensor is destructive - measuring it wipes the charge, so I'm not sure how any of this would work at all given that. Now what would actually be interesting is to apply the image-processing techniques to dual-DR imaging, as they mention in their "related works" section. The open-source Magic Lantern firmware for Canon DSLRs allows you to scan alternating rows at different ISOs, so effectively you can capture a lot more dynamic range at the expense of your vertical resolution. It's all volunteers and they probably haven't applied all the fancy image-processing magic with the convolutions and the hippity-hop rap music. How about reconstructing those overexposed lines instead? Or working on some of the sensors with physical implementations of dual-DR?
Tekker 11y ago
"Beginning of the end for over-saturated images" ? Well, then, say goodbye to the Disney channel, the most oversaturated channel out there.
IshKebab 11y ago
This is a great idea. It would be interesting to see the failure modes though because the phase-unwrapping surely can't handle every situation (e.g. adding a constant value >255 to the whole image).
salimmadjd 11y ago
As a photographer who at times needed to deal with HDR this is great news. In the past, I had often wondered why some scheme like this wasn't possible. However, the title of this post is horrible. I understand they were trying to market it in a way it was accessible to most readers. That said, photography has become quiet popular and most people now understand HDR. So they could have just presented as true HDR with only a single frame image.
tlb 11y ago
Title changed from "Beginning of the End for Oversaturated Images"
cozzyd 11y ago
Instagram and the like will ensure the future proliferation of shitty images, no matter the technology.
slr555 11y ago
I am probably less technical than most of you but the upside of this technology (at the point it is real world ready) might be producing what I would call a RAW file that is RAW(ER). By that I mean the file would simply contain a wider range of scene information that could be accessed by Adobe Camera Raw or the like. For the class of photographers that have no interest in RAW the information would help the JPEG engine in the camera make a "better" informed JPEG (or whatever the standard is at that time). To me capturing more scene info is better than less assuming there are minimal penalties being paid elsewhere (file size, burst shooting etc). As a side note, it seems to me that aperture and shutter speed would still be essential as they affect not only exposure but depth of field and subject motion. Perhaps a technique that allows proper exposure independent of fstop/shutter/ASA would allow exploration of extremely wide aperture with slow shutter but no need for ND filters. In any event it will be interesting to see what happens.
megrimlock 11y ago
Anyone know what they mean by an "inverse modulo" algorithm? That only seems meaningful for numbers coprime to the sensor ceiling value.
oppositelock 11y ago
I would love to feed a photo of an overexposed zebra to their reconstruction algorithm and see what happens. I think it will be spectacular!
deleted 11y ago
[deleted]
vernie 11y ago
Can somebody explain how this design differs from existing self-resetting pixel designs?
- oppositelock 11y ago
  I'm not sure what you mean by self resetting pixel designs. Do you perhaps means the self-shuttering (rolling shutters) on CMOS sensors? This is very different. If you oversaturate either a self shuttering sensor or a shuttered CCD, that information is gone. All you know is that this is "100% bright, or more". Your software which turns raw sensor readings into an image makes the area full brightness, and you get blow outs. With this approach, say the light is bright enough to make a pixel read 567% (if that as possible). A standard sensor would record 100%. This sensor would empty itself five times and record 67%. Then, the algorithmic magic happens which looks at neighboring areas and figures out how many roll-overs happened and adds the 500% back in. You can't do that perfectly (hence my comment before about an overexposed zebra), and it will introduce new forms of aliasing, kind of like de-bayering the bayer masks makes moiree patterns.
  - vernie 11y ago
    I've seen CMOS pixel designs in the past that can detect saturation and effectively dump the well (reset) repeatedly until saturation no longer occurs.
jlarocco 11y ago
I wonder how this compares to just adding a few extra bits to the sensor data. The paper (http://web.media.mit.edu/~hangzhao/papers/moduloUHDR.pdf http://web.media.mit.edu/~hangzhao/papers/moduloUHDR.pdf) compares against an "8-bit intensity camera", but most cameras today use 12-16 bits internally. Isn't "this pixel overflowed" essentially just an extra bit of information? The pictures using the "modulo camera" are definitely better quality, but don't look noticeably better than what I could do with a RAW from my camera and 2 seconds in Capture One or Lightroom adjusting the shadows and highlights.
iliis 11y ago
A related, but quite different technology I worked with are so-called Dynamic Vision Sensors (DVS, [1]): This is a CCD that was developed to mimic a biological retina and only reports changes in brightness, asynchronously for every pixel. Essentialy, you get a stream of short messages saying "Pixel XY just got a little brighter at time T". While such a thing is not very useful for taking pictures (pointing the camera at a static scene will generate no output at all) or videos (you don't get discrete frames, you get a continuous stream of differentiated pixels), it has a lot of potential for computer-vision tasks: For one, as the camera only reports changes, only the interesting information is transmitted which greatly reduces computational load (eg. no need for "yes, that white wall is still there"-style calculations for every frame). Secondly, because these changes are reported independently per pixel, they are very fast: Microsecond accuracy with a dozen or so uS delay are easily achievable. For comparison, a fast camera with 120 FPS (which will produce a lot more data to process) has an accuracy and latency of 1/120s = 8333uS. This also implicates that motion blur is basically nonexistent. And thirdly, as absolute intensity doesn't matter, you don't have any problems with high dynamic range either. The only downside is, that you don't really get a picture and most of the traditional computer-vision approaches are unusable ;) But still, such a camera is very neat, even if you just track the movement of your mice or want to balance a pencil on it's head[2]. The real deal however is full vision based 3D-SLAM, which would allow for very fast and robust movement without any external help from motion capturing systems or expensive (in money, weight and power) sensors like laser scanners. AFAIK, we're not there yet (see [3] for some work in that direction), but that would bring pizza delivery with a drone directly to your desk quite a bit closer to reality... -- [1] http://www.inilabs.com/products/davis http://www.inilabs.com/products/davis [2] http://www.ini.ch/~conradt/projects/PencilBalancer/ http://www.ini.ch/~conradt/projects/PencilBalancer/ [3] http://rpg.ifi.uzh.ch/research_dvs.html http://rpg.ifi.uzh.ch/research_dvs.html, also https://www.doc.ic.ac.uk/~ajd/Publications/kim_etal_bmvc2014.pdf https://www.doc.ic.ac.uk/~ajd/Publications/kim_etal_bmvc2014...

140 points by dnt404-1 11y ago

jcr 11y ago

The original title of the article is, "Unbounded High Dynamic Range Photography Using a Modulo Camera"

What's with the completely useless thumbnail at the bottom? That said, the technological explanation sounds simple and solid. Why aren't cameras doing that already?

putterson 11y ago
I wondered the same; it takes two hyperlinks to get to the project's home page where they have a larger version. http://web.media.mit.edu/~hangzhao/modulo.html http://web.media.mit.edu/~hangzhao/modulo.html
- Mithaldu 11y ago
  Nice, thank you!
vlamingsjef 11y ago
High resolution image here: http://web.media.mit.edu/~hangzhao/images/modulo_teaser.png http://web.media.mit.edu/~hangzhao/images/modulo_teaser.png
- stevensanderson 11y ago
  Note that this is a simulated example, not one actually taken with a modulo camera (if such a thing has been built yet). That image of Tintern Abbey comes from this 2011 web page: http://people.csail.mit.edu/sparis/publi/2011/siggraph/additional_results/tone_mapping_large/tinterna.html http://people.csail.mit.edu/sparis/publi/2011/siggraph/addit...

jchrisa 11y ago

Reminds me of the implementation of the TRNG from yesterday's front page.

darkmighty 11y ago

The title here is terrible because over-saturation is an effect of poor tone mapping, which is inevitable if you want to display an HDR image in a low dynamic range display. An accurate title would be the end of over-exposed images, but why not just keep the original?

kps 11y ago
Two different senses of the word ‘saturation’. The subtitle of the article (current title here) is unfortunate.
- tyingq 11y ago
  Yes. I expected to see an article about stock images being overused across lots of publications, and whatever the fix for that is.
IshKebab 11y ago
They're talking about the pixel brightness values getting saturated (255), not colour saturation. But I've never heard over-saturation being used to refer to brightness saturation - brightness is either saturated or not.
- ssalazar 11y ago
  I'd guess whoever wrote the article subtitle comes from an EE or DSP background, where saturation refers to blowing out the ends of the dynamic range and clipping/distorting.
sctb 11y ago
We updated the submission title to the article's.

soylentcola 11y ago

Looks to be more related to exposure than saturation but still, it's an interesting new take on dynamic range. I'm more a hobbyist than an expert but current tone mapping techniques (at least anything mostly automated) can often leave you with "halos" around objects where the software has feathered the edges between lighter and darker areas. Then the article mentions the issues with using multiple shots for tone mapping. I still try to just expose "properly" for the effect I want and shoot RAW to give me a little leeway in tweaking shadows and highlights but I'd be interested in trying out software that uses this technique if only for something new to mess with.

darkmighty 11y ago
Halos are a problem, but they can mostly be removed with good post-processing. Note they're not a fundamental aspect of tone mapping. Tone mapping is just whatever algorithm you chose to compress the HDR information into a low dynamic range we have for displays. The great thing about this camera compared to multi-exposure HDR (apart from the halo effect you mentioned) is you can actually take pictures of moving objects or even make HDR movies (the temporal variation should actually help with better phase unwrapping). That would be amazing for the film industry I think.
- dharma1 11y ago
  You can actually already do this with current cameras, using 2 exposures per frame, works for most types of action. Red Epic does this with HDR-X and hacked Canon cameras with Magic Lantern with Dual ISO
  - darkmighty 11y ago
    As explained in the article, this is different to combining exposures, which is what conventional HDR does too. The important thing to consider here is how does this scheme affect the SNR of the sensor. This scheme encourages keeping essentially maximum possible sensitivity (ISO), but this "modulo sensor" circuit may introduce some additional noise.
- 11y ago

baldeagle 11y ago

It seems like this would make big glass even more important. This solution protects for over exposure, but doesn't address under exposure on a scene that moves (think indoor bar photography). So it isn't a end all solution that will everyone capture reality with their cell phones, but it is an awesome step in that direction.

dr_zoidberg 11y ago

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowledged by that understanding. The technology is good, but it won't change the way photographers deal with photography, at best, it will give them a better tool to work with. I also wonder how much impact this could have. Foveon Inc. had developed a great sensor that was going to be revolutionary, about 20 years ago. Today only Sigma uses it, and the UX is so bad in those cameras that the (very good) technology they had their hands on never got a chance to shine. Also, is this implementation very different from having a 16-bit ADC in normal sensors? The end results would be the same: higher dynamic range. And the increase in bits seems a lot closer (because it would mean little changes to the current manufacture processes) than a whole new type of sensor being use in cameras.

kps 11y ago
Sure, photographs will still control aperture and exposure time — this just removes a constraint at one end: if you want shallow depth of field and motion blur, you could use this method (and get extra bits) rather than an ND filter.
- StavrosK 11y ago
  Exactly. This is huge, and I was wondering why people didn't do it already. Or, for example, why don't cameras currently take multiple snapshots of the sensor? For example, if I'm exposing for 2 seconds, read the sensor at 0.1 sec (for the very bright lights) and again at 2 seconds, for the darker parts. That way, I can make an HDR image without having to take another photo. Is reading the sensor destructive to the data, somehow?
  - EvanAnderson 11y ago
    Readout of a CCD is destructive. There are types of sensors that have non-destructive readout (Charge Injection Devices being one example: https://www.cis.rit.edu/research/CID/a_cid_is.htm https://www.cis.rit.edu/research/CID/a_cid_is.htm).

kefka 11y ago

I had an idea similar to this that solved the oversaturation issue. Use 2 cameras. Have them at right angles to the shutter point, separated a few inches. At the T junction where the camera pointing would cross with the shutter area, put a prism there. The prism would give each camera 1/2 of the available photons. Now use any standard HDR processing technique.

rasz_pl 11y ago
prism = less than half signal at each end
paulmd 11y ago
Interestingly enough that's how Technicolor used to work. Three separate exposures from a single lens, split off by pellicle mirrors. The downside is that like any pellicle you are dividing your light between multiple exposures, meaning you some mixture of a longer exposure, a faster lens, or more lighting. Technicolor required special high-intensity arc lighting. Also, the pellicles must be kept spotless because dust shows up on the final image. Next-gen sensors will actually implement this physically - some receptor sites will be larger than others.
- kefka 11y ago
  Admittedly, my understanding of traditional optics is very limited. I happened upon it in a dream whilst trying to figure out how to handle dynamic scenes while using SLAM. I was thinking of a prism (new word: pellicle mirror) like this one : http://static1.squarespace.com/static/54ca877ce4b014ea90e14bda/54ca9f5de4b021f8b6d68cc1/54caa186e4b021f8b6d6c6b8/1422565766946/penta_3.png?format=original http://static1.squarespace.com/static/54ca877ce4b014ea90e14b... Except I don't need the deconvolutional prism. I can handle that in software easily with a X or Y matrix flip. And I assume we are using the same CCD module for both areas; given that, I can sum the contrast between the pixels and calculate the min and max, as well as can calculate the HDR image from both sources. Calibration would be easy to handle, using a checkerboard to link the CCDs to each other (assume aligning errors will happen). I also see no reason why the pellicles wouldn't be kept clean. The same goes for the CCDs as well. I'd embed them in a dark acrylic box and try to seal them as best as can be.

brador 11y ago

This was the runner up prize winning paper, what won?

Joeboy 11y ago
https://twitter.com/ICCP_2015/status/592403545468448768/photo/1 https://twitter.com/ICCP_2015/status/592403545468448768/phot...
- mjcohen 11y ago
  That's a picture. Where's the paper?
  - amritamaz 11y ago
    Here's the paper link: http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_ICCP15.pdf http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nayar_ICC...

dharma1 11y ago

Read about something like this on http://image-sensors-world.blogspot.co.uk http://image-sensors-world.blogspot.co.uk a while ago - instead of saturating the CMOS when max brightness is exceeded, it just keeps going. Looking forward to seeing this commercialised.

paulmd 11y ago

Here's an idea, instead of separately tracking the number of "resets" we just add those bits on to the left hand side of the measurement. We could call it... having more bits in the ADC. Expose for the highlights and then pull more detail out of the shadows with your extra bits of ADC precision. There are not 1 million ADC units in a 1-megapixel camera, there's a few ADCs (a Canon 7D has 2 image processors with 4 ADCs each) that are iterated across the CCD sites to progressively read them out. To make this new sensor, you need to cram 50 million sets (50mp is state of the art) of voltage comparator/charge reset/digital counter circuits onto the CCD, and for N bits of rollover accuracy they need to be able to trigger, erase the charge, and resume exposure at least N times during the exposure time E (which is say 1/500th of a second, since they're complaining about movement during multiple HDR exposures). Practically speaking the trigger/operation/resume period must be significantly less than E/N since I see no way to retain the exposure during the period when the charge well is being drained to zero. If this time is non-trivial, that translates to losing the fine bits of your ADC accuracy. In their image they compare a 13-stop exposure (the current state of the art) to an 8-stop exposure (state of the art in 1900). So assuming they didn't just pull a photoshop out of their ass, they are overall claiming a 5 stop increase from state of the art. That's 5 extra bits of recovery (1 stop is double the range, i.e. an extra bit), so they think they can trigger the reset circuit 5 times during an exposure. That implies this circuit must have a minimum cycle time of 1/2500th of a second (practically speaking a lot less to give time for the actual exposure). With this math I'm also assuming that the comparator is perfect and doesn't lose any accuracy - variance in trigger threshold or trigger time translates to losing some of your accuracy again. They also need to generate little enough waste heat to avoid hot pixels (this is a major reason the ADC is on the image processor rather than the CCD), and you need 50 million sets of these on the CCD. If you can do it then go for it. It's a nice idea on paper but I think there's a lot of physical obstacles to overcome. If it were that easy someone would have done it already. It's especially difficult given that reading out the sensor is destructive - measuring it wipes the charge, so I'm not sure how any of this would work at all given that. Now what would actually be interesting is to apply the image-processing techniques to dual-DR imaging, as they mention in their "related works" section. The open-source Magic Lantern firmware for Canon DSLRs allows you to scan alternating rows at different ISOs, so effectively you can capture a lot more dynamic range at the expense of your vertical resolution. It's all volunteers and they probably haven't applied all the fancy image-processing magic with the convolutions and the hippity-hop rap music. How about reconstructing those overexposed lines instead? Or working on some of the sensors with physical implementations of dual-DR?

Tekker 11y ago

"Beginning of the end for over-saturated images" ? Well, then, say goodbye to the Disney channel, the most oversaturated channel out there.

IshKebab 11y ago

This is a great idea. It would be interesting to see the failure modes though because the phase-unwrapping surely can't handle every situation (e.g. adding a constant value >255 to the whole image).

salimmadjd 11y ago

As a photographer who at times needed to deal with HDR this is great news. In the past, I had often wondered why some scheme like this wasn't possible. However, the title of this post is horrible. I understand they were trying to market it in a way it was accessible to most readers. That said, photography has become quiet popular and most people now understand HDR. So they could have just presented as true HDR with only a single frame image.

tlb 11y ago

Title changed from "Beginning of the End for Oversaturated Images"

cozzyd 11y ago

Instagram and the like will ensure the future proliferation of shitty images, no matter the technology.

slr555 11y ago

I am probably less technical than most of you but the upside of this technology (at the point it is real world ready) might be producing what I would call a RAW file that is RAW(ER). By that I mean the file would simply contain a wider range of scene information that could be accessed by Adobe Camera Raw or the like. For the class of photographers that have no interest in RAW the information would help the JPEG engine in the camera make a "better" informed JPEG (or whatever the standard is at that time). To me capturing more scene info is better than less assuming there are minimal penalties being paid elsewhere (file size, burst shooting etc). As a side note, it seems to me that aperture and shutter speed would still be essential as they affect not only exposure but depth of field and subject motion. Perhaps a technique that allows proper exposure independent of fstop/shutter/ASA would allow exploration of extremely wide aperture with slow shutter but no need for ND filters. In any event it will be interesting to see what happens.

megrimlock 11y ago

Anyone know what they mean by an "inverse modulo" algorithm? That only seems meaningful for numbers coprime to the sensor ceiling value.

oppositelock 11y ago

I would love to feed a photo of an overexposed zebra to their reconstruction algorithm and see what happens. I think it will be spectacular!

deleted 11y ago

[deleted]

vernie 11y ago

Can somebody explain how this design differs from existing self-resetting pixel designs?

oppositelock 11y ago
I'm not sure what you mean by self resetting pixel designs. Do you perhaps means the self-shuttering (rolling shutters) on CMOS sensors? This is very different. If you oversaturate either a self shuttering sensor or a shuttered CCD, that information is gone. All you know is that this is "100% bright, or more". Your software which turns raw sensor readings into an image makes the area full brightness, and you get blow outs. With this approach, say the light is bright enough to make a pixel read 567% (if that as possible). A standard sensor would record 100%. This sensor would empty itself five times and record 67%. Then, the algorithmic magic happens which looks at neighboring areas and figures out how many roll-overs happened and adds the 500% back in. You can't do that perfectly (hence my comment before about an overexposed zebra), and it will introduce new forms of aliasing, kind of like de-bayering the bayer masks makes moiree patterns.
- vernie 11y ago
  I've seen CMOS pixel designs in the past that can detect saturation and effectively dump the well (reset) repeatedly until saturation no longer occurs.

jlarocco 11y ago

I wonder how this compares to just adding a few extra bits to the sensor data. The paper (http://web.media.mit.edu/~hangzhao/papers/moduloUHDR.pdf http://web.media.mit.edu/~hangzhao/papers/moduloUHDR.pdf) compares against an "8-bit intensity camera", but most cameras today use 12-16 bits internally. Isn't "this pixel overflowed" essentially just an extra bit of information? The pictures using the "modulo camera" are definitely better quality, but don't look noticeably better than what I could do with a RAW from my camera and 2 seconds in Capture One or Lightroom adjusting the shadows and highlights.

iliis 11y ago

A related, but quite different technology I worked with are so-called Dynamic Vision Sensors (DVS, [1]): This is a CCD that was developed to mimic a biological retina and only reports changes in brightness, asynchronously for every pixel. Essentialy, you get a stream of short messages saying "Pixel XY just got a little brighter at time T". While such a thing is not very useful for taking pictures (pointing the camera at a static scene will generate no output at all) or videos (you don't get discrete frames, you get a continuous stream of differentiated pixels), it has a lot of potential for computer-vision tasks: For one, as the camera only reports changes, only the interesting information is transmitted which greatly reduces computational load (eg. no need for "yes, that white wall is still there"-style calculations for every frame). Secondly, because these changes are reported independently per pixel, they are very fast: Microsecond accuracy with a dozen or so uS delay are easily achievable. For comparison, a fast camera with 120 FPS (which will produce a lot more data to process) has an accuracy and latency of 1/120s = 8333uS. This also implicates that motion blur is basically nonexistent. And thirdly, as absolute intensity doesn't matter, you don't have any problems with high dynamic range either. The only downside is, that you don't really get a picture and most of the traditional computer-vision approaches are unusable ;) But still, such a camera is very neat, even if you just track the movement of your mice or want to balance a pencil on it's head[2]. The real deal however is full vision based 3D-SLAM, which would allow for very fast and robust movement without any external help from motion capturing systems or expensive (in money, weight and power) sensors like laser scanners. AFAIK, we're not there yet (see [3] for some work in that direction), but that would bring pizza delivery with a drone directly to your desk quite a bit closer to reality... -- [1] http://www.inilabs.com/products/davis http://www.inilabs.com/products/davis [2] http://www.ini.ch/~conradt/projects/PencilBalancer/ http://www.ini.ch/~conradt/projects/PencilBalancer/ [3] http://rpg.ifi.uzh.ch/research_dvs.html http://rpg.ifi.uzh.ch/research_dvs.html, also https://www.doc.ic.ac.uk/~ajd/Publications/kim_etal_bmvc2014.pdf https://www.doc.ic.ac.uk/~ajd/Publications/kim_etal_bmvc2014...