6 ms·

Simultaneous Imaging and Energy Harvesting in CMOS Image Sensor Pixels

132 points by touristtam 9y ago

lawlessone 9y ago
"It turns out that photosensitive diodes aren’t totally opaque — in fact, quite a lot of light passes right through them. So putting the solar cell under the image sensor doesn’t actually deprive it of light." Does that also mean we could put multiple layers of photosensitive diodes under one another to perhaps get higher resolution images? or better images in the dark?
- m1el 9y ago
  Higher resolution - no. Resolution is limited due to diffraction, so adding more pixels doesn't necessarily add more details. Better images in the dark - possibly.
  - SiempreViernes 9y ago
    More important than the diffraction limit is probably the fact you put an image sensor in front of your image sensor, and it will distort the wavefront for the sensor behind it. Plastic bags are also quite transparent, but you never get a better picture by putting them in front of your camera.
    - amelius 9y ago
      Isn't it possible to computationally counter the effects of diffraction and distortion?
      - larkeith 9y ago
        Especially when you have a predistortion image from the frontmost sensor, and I would imagine that the diffraction would remain constant between photos - manufacture differences would presumably mean each sensor would need to be calibrated, but with the multiple sensors, it seems quite possible that could be done by the consumer.
      - Escapado 9y ago
        Generally speaking yes! That's what companies do that create masks for lithography so that the structures the manufacturers can create are smaller than the wavelength they use in the process. (See https://en.m.wikipedia.org/wiki/Computational_lithography https://en.m.wikipedia.org/wiki/Computational_lithography). The article states that generally this is rather computationally expensive and I would wager that an analogous concept would be too. Perhaps a Neural Network could be trained to infer corrections and maybe that would be feasible then but maybe someone else with more experience in the field can elaborate on that.
        jwilk 9y ago
        Non-mobline link: https://en.wikipedia.org/wiki/Computational_lithography https://en.wikipedia.org/wiki/Computational_lithography
        deepnotderp 9y ago
        Neural networks suffer from provability issues, you don't want to make a multi million dollar mask set only to find out your ConvNets were hallucinating. There are more standard computational lithography techniques such as OPC, RET and inverse lithography. With that being said, there's been some work done: https://ieeexplore.ieee.org/document/5726499/ https://ieeexplore.ieee.org/document/5726499/ https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10147/1/Accurate-lithography-simulation-model-based-on-convolutional-neural-networks/10.1117/12.2257871.short?SSO=1 https://www.spiedigitallibrary.org/conference-proceedings-of... http://iopscience.iop.org/article/10.1088/2040-8978/12/4/045601/meta http://iopscience.iop.org/article/10.1088/2040-8978/12/4/045... https://www.google.com/url?sa=t&source=web&rct=j&url=http://yibolin.com/publications/papers/DFM_ISPD2018_Lin.pdf&ved=2ahUKEwiHnePtsbDaAhUqs1QKHbqPDIsQFjAEegQIAxAB&usg=AOvVaw2bkntnIfn5oi_INTKovR-C https://www.google.com/url?sa=t&source=web&rct=j&url=http://...
      - gascan 9y ago
        They say the answer is "yes". It's even nominally offered as a feature in, e.g. Canon's RAW processing tool (called "DLO") Unfortunately, in practice it seems to behave more like smart sharpening than a discrete reverse convolution math op.
- janekm 9y ago
  Such a sensor (used to) exist, Foveon X3. It is very interesting as it also uses the differential absorption of different wavelengths of light at different depths in silicon to distinguish colours: https://en.m.wikipedia.org/wiki/Foveon_X3_sensor https://en.m.wikipedia.org/wiki/Foveon_X3_sensor It had various advantages, and some disadvantages related to distinguishing colours, but ultimately wasn’t very successful in the market despite getting shipped in some DSLRs, possibly due to the marketing challenges (Bayer array sensor manufacturers count each R/G/B sub pixel as part of their “megapixel” count, so the Foveon sensor appeared low resolution by that metric).
  - bitL 9y ago
    My Sigma SD1M still produces mind-blowing images I can't do with high-end Nikons and Canons. Foveon can truly make magical images, though there were so many issues with DSLRs it shipped on, e.g. they were almost always awfully slow and had very bad (but film-like) low-light performance. Also, original research suggested that to get realistic colors, up to 7 layers are needed. Bayers OTOH just interpolate and cook colors, but with the increase of resolution might at some point perform similarly, when lens limit is hit. Foveon was also always manufactured by a far worse process than e.g. Sony top-end sensors, and its quantum efficiency was orders of magnitude worse.
    - CarVac 9y ago
      Bayers don't cook colors when they interpolate, they cook edges.
  - teemwerk 9y ago
    I don't follow, Sigma hasn't stopped using Foveon sensors, they still use the Foveon Quattro stuff afaik. Now they use a 1:1:4 ratio for R/G/B layer resolution, where the blue layer also records luminance. They decided they wanted to confuse people more when they tried to compare it to a Bayer sensor.
    - janekm 9y ago
      Actually I didn't realise they still made them... Nice to know. I remembered the Foveon sensors from when they were first introduced, but didn't see any reference to them still being made when I did a quick search to refresh my memory, so they may have a bit of a marketing issue ;)
- rasz 9y ago
  Already on the market. Interview with Foveon cofounder Richard Lyon : https://www.youtube.com/watch?v=pIZp0qPZySw https://www.youtube.com/watch?v=pIZp0qPZySw Fun fact - Carver Mead was one of cofounders, of the Conway-Mead fame (founder of modern VLSI revolution). Another was Faggin, father of Intel 4004. Whole company was loaded by hardcore semiconductor process design engineers.
blauditore 9y ago
But a camera still needs some room (behind a point hole, and potentially some lens). At some point this will have higher impact than sensor size, no?
fredley 9y ago
Although I couldn't stand The Circle (book or film), it did raise questions about what happens to society when we can place extremely cheap cameras everywhere invisibly that require no battery, and hence can operate almost indefinitely. In The Circle they are owned by a pseudo Goog-book company that publishes (and records) every video feed (and its history) publicly.
- primitur 9y ago
  I started to wonder about this, but then I realized that what we're really witnessing right now is the birth of the primordial Grey Goo. I mean, omnipresent super-observation states may be one angle of Our Future Dystopia, but I think energy-harvesting super computers that can reproduce is right around the corner on this one ..
  - fredley 9y ago
    Reproduction is one thing I'm really not concerned about actually. Is it even possible to build any kind of silicon-based electronics outside of a fab?
    - javajosh 9y ago
      We've already entered into a period of symbiosis, where software controls human bodies. It seems unlikely that machines can ever do better than a human hand (but can do better almost everywhere else!)
      - fredley 9y ago
        That software is still created by humans, subject to human forces though. You can characterise many things as 'training' or 'evolving' humans to serve their purposes, e.g. gut bacteria, poppies, cows, dogs, toxoplasma gondii, hammers etc. The characterisation doesn't tell you much though.
- maurits 9y ago
  You might enjoy "The light of other days" [1] then. It envisions the world where wormhole cameras can view any event in the past, thus eliminating all secrecy and privacy overnight. [1]: https://en.wikipedia.org/wiki/The_Light_of_Other_Days https://en.wikipedia.org/wiki/The_Light_of_Other_Days
larkeith 9y ago
This seems like it could be combined with battery-free "no-power" WiFi tags [1] to provide functionally indefinite surveillance at a minimal price tag. While 1 kb/s bandwidth and a range of only 2 m limit the current applications of such a combination, the potential once both technologies become more mature is staggering (and rather terrifying). [1] https://www.sciencedaily.com/releases/2014/08/140804134215.htm https://www.sciencedaily.com/releases/2014/08/140804134215.h...
- brk 9y ago
  Doubtful. Several companies have already come and gone with solar-powered security cameras that never really worked as desired. If we can't power a simple low-res (720p) wifi camera reliably with a large (as compared to an image sensor) dedicated solar panel today, this technology is unlikely to change that in the next decade.
- awelkie 9y ago
  There has been some recent work[1] towards backscatter video (actually from the same lab you linked to). The basic idea is the same as the "great seal bug"[2] except for photodiodes. I say "towards" because much of the work they present is in the form of simulation. [1] https://homes.cs.washington.edu/~gshyam/Papers/videobackscatter.pdf https://homes.cs.washington.edu/~gshyam/Papers/videobackscat... [2] https://en.wikipedia.org/wiki/The_Thing_(listening_device) https://en.wikipedia.org/wiki/The_Thing_(listening_device)
- amelius 9y ago
  You still need to process or transmit the images, which might be costlier than recording the images.
  - cf498 9y ago
    For transmitting data on an energy harvesting budget there is stuff like LoRa, but Image is out of the question here because of the size. Much more likely is that such devices will be used for tracking, in situations where GPS is using to much power. Especially, as the calculations for that are done at the receivers instead of the transmitter.
nukeop 9y ago
There was a science fiction story about a society where privacy was completely impossible because air everywhere was permeated by nanoscale cameras that recorded everything everywhere from all perspectives at all times. Looks like we are approaching that future.
bitrazor123 9y ago
Bring it on
- quantized1 9y ago
  Yeah !
dzink 9y ago
Those would be useful for vascular and other medical imaging uses. Search and rescue bots as well.
bovermyer 9y ago
You know what a cool application of this would be? Cameras in your bloodstream that create a distributed 3D image of your entire circulation at once.
- albemuth 9y ago
  What would the light source be?
  - Scoundreller 9y ago
    A 50w LED that follows along externally?
Avery3R 9y ago
If you scroll down too far the website hijacks your back button

132 points by touristtam 9y ago

"It turns out that photosensitive diodes aren’t totally opaque — in fact, quite a lot of light passes right through them. So putting the solar cell under the image sensor doesn’t actually deprive it of light." Does that also mean we could put multiple layers of photosensitive diodes under one another to perhaps get higher resolution images? or better images in the dark?

m1el 9y ago
Higher resolution - no. Resolution is limited due to diffraction, so adding more pixels doesn't necessarily add more details. Better images in the dark - possibly.
- SiempreViernes 9y ago
  More important than the diffraction limit is probably the fact you put an image sensor in front of your image sensor, and it will distort the wavefront for the sensor behind it. Plastic bags are also quite transparent, but you never get a better picture by putting them in front of your camera.
  - amelius 9y ago
    Isn't it possible to computationally counter the effects of diffraction and distortion?
    - larkeith 9y ago
      Especially when you have a predistortion image from the frontmost sensor, and I would imagine that the diffraction would remain constant between photos - manufacture differences would presumably mean each sensor would need to be calibrated, but with the multiple sensors, it seems quite possible that could be done by the consumer.
    - Escapado 9y ago
      Generally speaking yes! That's what companies do that create masks for lithography so that the structures the manufacturers can create are smaller than the wavelength they use in the process. (See https://en.m.wikipedia.org/wiki/Computational_lithography https://en.m.wikipedia.org/wiki/Computational_lithography). The article states that generally this is rather computationally expensive and I would wager that an analogous concept would be too. Perhaps a Neural Network could be trained to infer corrections and maybe that would be feasible then but maybe someone else with more experience in the field can elaborate on that.
      - jwilk 9y ago
        Non-mobline link: https://en.wikipedia.org/wiki/Computational_lithography https://en.wikipedia.org/wiki/Computational_lithography
      - deepnotderp 9y ago
        Neural networks suffer from provability issues, you don't want to make a multi million dollar mask set only to find out your ConvNets were hallucinating. There are more standard computational lithography techniques such as OPC, RET and inverse lithography. With that being said, there's been some work done: https://ieeexplore.ieee.org/document/5726499/ https://ieeexplore.ieee.org/document/5726499/ https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10147/1/Accurate-lithography-simulation-model-based-on-convolutional-neural-networks/10.1117/12.2257871.short?SSO=1 https://www.spiedigitallibrary.org/conference-proceedings-of... http://iopscience.iop.org/article/10.1088/2040-8978/12/4/045601/meta http://iopscience.iop.org/article/10.1088/2040-8978/12/4/045... https://www.google.com/url?sa=t&source=web&rct=j&url=http://yibolin.com/publications/papers/DFM_ISPD2018_Lin.pdf&ved=2ahUKEwiHnePtsbDaAhUqs1QKHbqPDIsQFjAEegQIAxAB&usg=AOvVaw2bkntnIfn5oi_INTKovR-C https://www.google.com/url?sa=t&source=web&rct=j&url=http://...
    - gascan 9y ago
      They say the answer is "yes". It's even nominally offered as a feature in, e.g. Canon's RAW processing tool (called "DLO") Unfortunately, in practice it seems to behave more like smart sharpening than a discrete reverse convolution math op.
janekm 9y ago
Such a sensor (used to) exist, Foveon X3. It is very interesting as it also uses the differential absorption of different wavelengths of light at different depths in silicon to distinguish colours: https://en.m.wikipedia.org/wiki/Foveon_X3_sensor https://en.m.wikipedia.org/wiki/Foveon_X3_sensor It had various advantages, and some disadvantages related to distinguishing colours, but ultimately wasn’t very successful in the market despite getting shipped in some DSLRs, possibly due to the marketing challenges (Bayer array sensor manufacturers count each R/G/B sub pixel as part of their “megapixel” count, so the Foveon sensor appeared low resolution by that metric).
- bitL 9y ago
  My Sigma SD1M still produces mind-blowing images I can't do with high-end Nikons and Canons. Foveon can truly make magical images, though there were so many issues with DSLRs it shipped on, e.g. they were almost always awfully slow and had very bad (but film-like) low-light performance. Also, original research suggested that to get realistic colors, up to 7 layers are needed. Bayers OTOH just interpolate and cook colors, but with the increase of resolution might at some point perform similarly, when lens limit is hit. Foveon was also always manufactured by a far worse process than e.g. Sony top-end sensors, and its quantum efficiency was orders of magnitude worse.
  - CarVac 9y ago
    Bayers don't cook colors when they interpolate, they cook edges.
- teemwerk 9y ago
  I don't follow, Sigma hasn't stopped using Foveon sensors, they still use the Foveon Quattro stuff afaik. Now they use a 1:1:4 ratio for R/G/B layer resolution, where the blue layer also records luminance. They decided they wanted to confuse people more when they tried to compare it to a Bayer sensor.
  - janekm 9y ago
    Actually I didn't realise they still made them... Nice to know. I remembered the Foveon sensors from when they were first introduced, but didn't see any reference to them still being made when I did a quick search to refresh my memory, so they may have a bit of a marketing issue ;)
rasz 9y ago
Already on the market. Interview with Foveon cofounder Richard Lyon : https://www.youtube.com/watch?v=pIZp0qPZySw https://www.youtube.com/watch?v=pIZp0qPZySw Fun fact - Carver Mead was one of cofounders, of the Conway-Mead fame (founder of modern VLSI revolution). Another was Faggin, father of Intel 4004. Whole company was loaded by hardcore semiconductor process design engineers.

blauditore 9y ago

But a camera still needs some room (behind a point hole, and potentially some lens). At some point this will have higher impact than sensor size, no?

fredley 9y ago

Although I couldn't stand The Circle (book or film), it did raise questions about what happens to society when we can place extremely cheap cameras everywhere invisibly that require no battery, and hence can operate almost indefinitely. In The Circle they are owned by a pseudo Goog-book company that publishes (and records) every video feed (and its history) publicly.

primitur 9y ago
I started to wonder about this, but then I realized that what we're really witnessing right now is the birth of the primordial Grey Goo. I mean, omnipresent super-observation states may be one angle of Our Future Dystopia, but I think energy-harvesting super computers that can reproduce is right around the corner on this one ..
- fredley 9y ago
  Reproduction is one thing I'm really not concerned about actually. Is it even possible to build any kind of silicon-based electronics outside of a fab?
  - javajosh 9y ago
    We've already entered into a period of symbiosis, where software controls human bodies. It seems unlikely that machines can ever do better than a human hand (but can do better almost everywhere else!)
    - fredley 9y ago
      That software is still created by humans, subject to human forces though. You can characterise many things as 'training' or 'evolving' humans to serve their purposes, e.g. gut bacteria, poppies, cows, dogs, toxoplasma gondii, hammers etc. The characterisation doesn't tell you much though.
maurits 9y ago
You might enjoy "The light of other days" [1] then. It envisions the world where wormhole cameras can view any event in the past, thus eliminating all secrecy and privacy overnight. [1]: https://en.wikipedia.org/wiki/The_Light_of_Other_Days https://en.wikipedia.org/wiki/The_Light_of_Other_Days

larkeith 9y ago

This seems like it could be combined with battery-free "no-power" WiFi tags [1] to provide functionally indefinite surveillance at a minimal price tag. While 1 kb/s bandwidth and a range of only 2 m limit the current applications of such a combination, the potential once both technologies become more mature is staggering (and rather terrifying). [1] https://www.sciencedaily.com/releases/2014/08/140804134215.htm https://www.sciencedaily.com/releases/2014/08/140804134215.h...

brk 9y ago
Doubtful. Several companies have already come and gone with solar-powered security cameras that never really worked as desired. If we can't power a simple low-res (720p) wifi camera reliably with a large (as compared to an image sensor) dedicated solar panel today, this technology is unlikely to change that in the next decade.
awelkie 9y ago
There has been some recent work[1] towards backscatter video (actually from the same lab you linked to). The basic idea is the same as the "great seal bug"[2] except for photodiodes. I say "towards" because much of the work they present is in the form of simulation. [1] https://homes.cs.washington.edu/~gshyam/Papers/videobackscatter.pdf https://homes.cs.washington.edu/~gshyam/Papers/videobackscat... [2] https://en.wikipedia.org/wiki/The_Thing_(listening_device) https://en.wikipedia.org/wiki/The_Thing_(listening_device)
amelius 9y ago
You still need to process or transmit the images, which might be costlier than recording the images.
- cf498 9y ago
  For transmitting data on an energy harvesting budget there is stuff like LoRa, but Image is out of the question here because of the size. Much more likely is that such devices will be used for tracking, in situations where GPS is using to much power. Especially, as the calculations for that are done at the receivers instead of the transmitter.

nukeop 9y ago

There was a science fiction story about a society where privacy was completely impossible because air everywhere was permeated by nanoscale cameras that recorded everything everywhere from all perspectives at all times. Looks like we are approaching that future.

bitrazor123 9y ago

Bring it on

quantized1 9y ago
Yeah !

dzink 9y ago

Those would be useful for vascular and other medical imaging uses. Search and rescue bots as well.

bovermyer 9y ago

You know what a cool application of this would be? Cameras in your bloodstream that create a distributed 3D image of your entire circulation at once.

albemuth 9y ago
What would the light source be?
- Scoundreller 9y ago
  A 50w LED that follows along externally?

Avery3R 9y ago

If you scroll down too far the website hijacks your back button