7 ms·

Lensless camera creates detailed 3-D images without scanning

242 points by sgk284 9y ago

alanfalcon 9y ago
Brilliant and amazing. "This is a very powerful direction for imaging, but requires designers with optical and physics expertise as well as computational knowledge." It’s a little crazy to me how much can be accomplished by tackling hard problems in one domain by leveraging ideas and expertise from a seemingly unrelated[1], unexpected domain. Specialization is at the same time super important and a potential bottleneck to innovation. This fascinates me. [1]Not that physics expertise in imagery is unrelated, but I feel like it’s being used in very non-traditional ways here.
foota 9y ago
This is really neat, it seems to me like they use a see through material of some sort that scatters light randomly as a filter in front of the camera. They then move around a small light and use that to figure out the pattern that the light is scattered in by the material?
- eeZah7Ux 9y ago
  > They then move around a small light ...or wave around a checkerboard pattern of known size to make calibration faster perhaps?
  - solarkraft 9y ago
    Maybe, but they might need absolute positions.
ToJans 9y ago
TL;DR: this is an approach that simplifies the production of light field cameras (cameras that measure both color and angle of incoming light beams): instead of building a grid of microscopic lenses, you use a "random" piece of opaque plastic like scotch tape, and figure out how it modifies incoming light using a calibration phase.
- ClassyJacket 9y ago
  Could some middle-ground device exist where the lens is still manufactured for this use-case but still lower cost overall as the system has a high tolerance for quality/defects by implementing the calibration step?
  - icegreentea2 9y ago
    Sounds reasonable. They basically proved that the worse possible lens they can manufacture can still provide the information they need. I have no idea what the size of the relative trade offs involved are - ie, if you loosen the tolerance of your lens manufacturing by X%, what increase in computation is required. Exciting times!
- tischler 9y ago
  Here is a similar approach using water droplets instead of a diffuse film, but they don't go as far as performing 3D reconstruction from the light field and the ray directions are not calibrated by a calibration pattern but by inferring a 3D model of the droplets: https://light.cs.uni-bonn.de/4d-imaging-through-spray-on-optics/ https://light.cs.uni-bonn.de/4d-imaging-through-spray-on-opt...
  - frozenport 9y ago
    It's 2D, because they record a field U(x,y) and then propagate it up and down by convolving with a 2D function. This works for mesoscopic structures but not for small things like human cells.
deleted 9y ago
[deleted]
eeZah7Ux 9y ago
I wonder if unscrewing the lens on a cheap USB camera could lead to some interesting pictures. Any lensless camera Open Source project around? EDIT: A library, not the camera itself
- namibj 9y ago
  Axiom
- crusso 9y ago
  You still need the diffraction grating-like piece in front of the sensor array.
- solarkraft 9y ago
  Whew, we just learned about the research - but the paper is here. I'd be excited for such a project.
- lawall04 9y ago
  the paper itself has a link to the open-source code: http://www.laurawaller.com/research/diffusercam/ http://www.laurawaller.com/research/diffusercam/
mrow84 9y ago
Can someone explain why they can't apply the same reconstruction technique to the data that would be captured without a diffuser; i.e. why the diffuser is required?
- icegreentea2 9y ago
  I think the addition of the diffuser adds the 'depth response' to the camera. Imagine that you had two point light sources located some distance away from the sensor such that one is directly in front of the other. They're magical point light sources so the light from the rear one just passes through the front. If you didn't have anything in front of the sensor, there would be no way to distinguish that there are two point sources, as opposed to a single point source with a non uniform output.
- radarsat1 9y ago
  I think the diffuser ensures that there is a random distribution of incidence angle of photons. Then, if I understand, an optimisation is basically used to figure out what pixels correspond to what angles. I think the actual angle calculation is skipped and the reconstruction is done directly, but the "angle" concept is implicitly taken into account by the fact that it is needed for the 3D reconstruction to take place.
  - hammock 9y ago
    In essence, the diffuser is the lens (arranging the light in a usable way onto a sensor), just not a lens as we know it.
- frumiousirc 9y ago
  It's a good question and I think strikes at the heart of the idea. A lens transforms a family of rays to a pixel location. Given knowledge of that pixel's intensity there is a degenerate solution for the original ray (in terms of it's location and direction at some plane). This degeneracy is one thing that leads to blurry photos. The micro lens camera referenced in the article spreads this family over more pixels in a known, analytic way to make the solution more unique. In principle it suffers from the same degeneracy but each micro lens limits the possible location of rays so if any of the pixels under it are hit then the general location is set and the exact pixel in the group determines the direction. The diffuser works similarly but spreads direction and position location over many pixels and in a random way (seeded by the material and it's precise placement). While this spread can not be calculated it can be discovered through calibration with known point sources. In both these latter cases one inverts this analytic or calibrated ray->pixel matrix and applies that to the measured pixels to reconstruct the rays that have likely caused the measurement. In the case of the diffuse "lens", the required matrix inversion can be computationally expensive at best and impossible at worse. However, the methods of compressed sensing (in particular L1 regularization) allow an approximate inversion to be done in a relatively fast manner.
jaclaz 9y ago
For a moment I thought they could manage to have something like the ESPER machine in Blade Runner.
- sp332 9y ago
  Lytro's light-field cameras can do this. https://vimeo.com/102302646 https://vimeo.com/102302646
  - jaclaz 9y ago
    Yes, I know, thanks, but at the end of the day the Lytro nice technology allows for focus and depth of field correction (besides some slight shift in pespective), but once the "wow" effect has faded away (I mean for "plain" photography), that's it. For cinema, VR and CGI it is simply great of course.
    - sp332 9y ago
      Based on the movie, I'm not sure what features you want besides "move slightly to the right".
      - jaclaz 9y ago
        In the movie, from a plain photography, the ESPER machine manages, somehow, to enter into a reflection, and then expand in the reflection, seeing things that are "not there" in the original image, in practice in the fiction the whole 3d space (even what is on the side of the door) is navigable and viewable: http://thelegalgeeks.com/2017/06/28/admissibility-of-zhoras-photo-in-blade-runner/ http://thelegalgeeks.com/2017/06/28/admissibility-of-zhoras-... What I mean can be better appreciated in this reconstruction: https://typesetinthefuture.com/2016/06/19/bladerunner/ https://typesetinthefuture.com/2016/06/19/bladerunner/ https://typesetinthefuture.files.wordpress.com/2016/06/bladerunner_esper_room_layout_full.png https://typesetinthefuture.files.wordpress.com/2016/06/blade... https://vimeo.com/169392777 https://vimeo.com/169392777
        9y ago
crusso 9y ago
It depends on your definition of "lensless". You know those kids' books that have the bumpy plastic coating and when you turn the book one way you see one image - look at it from a different angle and you see another image? This is the same concept. They have a bumpy plastic coating that sends the incoming light in different directions. They do some processing on standard images to determine how the scattering works and then use that scattering pattern to reconstruct new images. I would view the bumpy coating as a myriad of lenses that change the character of the incoming light. We have one of those windows in our bathroom with the glass that is warped so that it breaks up the light so much that it gives you privacy. I've often thought that it would be a fun project to create a camera system that you could calibrate to decrypt that scattered image by placing a known image behind the window and pre-determining how the light waves are refracted. It's cool to see that someone implemented something similar.
- saycheese 9y ago
  On the same note, depends on how you define “scanning” sense they’re clearly scanning. You can do the same thing they’re doing with a bunch of pinhole, video feed with variation in the lighting, etc.
  - nwatson 9y ago
    I think "scanning" would imply compositing info gathered at different instances of time to reconstruct 3-D models ... e.g., LIDAR captures a point cloud with points measured at different times and tries to reconstruct objects from that point cloud. Perhaps the time it takes LIDAR to gather the whole point cloud is small, but objects will drift during that time and the reconstruction algorithm might need to account for this. On the other hand, the article's method captures all info in one instantaneous image and reconstructs based on the known, recorded, bias of the irregular lens and the implied 3-D locations of objects.
    - Retric 9y ago
      If you want to be really technical. Each data-point is not collected at the exact same instant.
saycheese 9y ago
Here’s the research paper: https://pdfs.semanticscholar.org/9cff/c0c80b1ae3c1b773b761f37c66e58890639e.pdf https://pdfs.semanticscholar.org/9cff/c0c80b1ae3c1b773b761f3...
bhouston 9y ago
Reminds me of insect eyes.
visarga 9y ago
I especially like the applications in cheap and compact 3D sensing and brain neural interfaces. Very exciting!
solarkraft 9y ago
Article and especially the title are of pretty low quality. Assuming the voxels are surface voxels, how is it even theoretically possible to turn 1 million pixels into 100 million voxels? This means you get 100x the xy resolution AND depth information out of this process. I'm sceptical of that claim. As crusso already mentioned lenses and scanning are essential parts of image capture, a lens being needed to direct the light to the sensor somehow and scanning to actually read out the image. "Using diffuse foils to replace microlens arrays" would probably be a more fitting and still teasing headline. Or "Diffuse foils can replace microlens arrays for 3D imaging", perhaps. Article aide, the research seems very sound and very cool. It demonstrates another case of extracting high quality information from low quality sensors - something I think we'll be seeing a lot more of. Another previously precisely manufactured piece of hardware is being replaced by a software-supported low-quality part through optimizations that in their spirit remind me of the Google Pixel's camera and that drone that can fly (steer) with one rotor.
- nsaote3ntean 9y ago
  > This means you get 100x the xy resolution AND depth information out of this process. How so? Isn't 1000 x 1000 = 1,000,000 So it's the same x/y resolution with 100x the z resolution. They extract the z by figuring out the different direction that light hitting each sensel is coming from. (And probably doing additional processing.)
- rleigh 9y ago
  We already have commercial systems which do this. See PhaseFocus for example (http://www.phasefocus.com/technology-virtual-lens/ http://www.phasefocus.com/technology-virtual-lens/). This uses proprietary deconvolution algorithms to reconstruct a 3D volume from diffraction patterns in unfocussed (or partially focussed) 2D planes. See also X-ray crystallography, which samples in Fourier space and uses a computed transform to reconstruct the 3D image. This new microlens technology is very neat, but the processing problem has been solved for several different applications for several years now. Reconstructing a 3D volume is absolutely possible, and while this new development will undoubtedly require new algorithms to work, it's already got a sound basis in existing technology in use today.
- 9y ago
punnerud 9y ago
The Research article (PDF): https://www.osapublishing.org/DirectPDFAccess/3ADDF00A-E071-C39B-B3DA82E301EA25C6_380297/optica-5-1-1.pdf?da=1&id=380297&seq=0 https://www.osapublishing.org/DirectPDFAccess/3ADDF00A-E071-...
kelvin0 9y ago
This could also be used to analyze some material used as the diffuser. The 'shape' of the caustics captured by the sensor with a given electromagnetic 'lightsource' can probably yield some interesting information regarding the diffuser. Kinda like an spectrograph works.
p1mrx 9y ago
Would it be possible to run this system in reverse, and make a holographic display?
FractalNerve 9y ago
This makes the Holographic_imager a reality, hooray! [1] http://memory-alpha.wikia.com/wiki/Holographic_imager http://memory-alpha.wikia.com/wiki/Holographic_imager To me this is the most major breakthrough I've heard in the recent years, which can and will hopefully affect everything. Using the extra CMOS Chip on your flagship smartphone will allow for taking 3D and soon Holographic pictures! How Amazing! I remember there was an AI trained to turn 2D pictures into 3D [2], combining that with the NPU Chip on smartphones can truly make this happens very soon. [2] http://www.dailymail.co.uk/sciencetech/article-4904298/The-AI-turn-selfie-3D-image.html http://www.dailymail.co.uk/sciencetech/article-4904298/The-A...

242 points by sgk284 9y ago

Brilliant and amazing. "This is a very powerful direction for imaging, but requires designers with optical and physics expertise as well as computational knowledge." It’s a little crazy to me how much can be accomplished by tackling hard problems in one domain by leveraging ideas and expertise from a seemingly unrelated[1], unexpected domain. Specialization is at the same time super important and a potential bottleneck to innovation. This fascinates me. [1]Not that physics expertise in imagery is unrelated, but I feel like it’s being used in very non-traditional ways here.

foota 9y ago

This is really neat, it seems to me like they use a see through material of some sort that scatters light randomly as a filter in front of the camera. They then move around a small light and use that to figure out the pattern that the light is scattered in by the material?

eeZah7Ux 9y ago
> They then move around a small light ...or wave around a checkerboard pattern of known size to make calibration faster perhaps?
- solarkraft 9y ago
  Maybe, but they might need absolute positions.

ToJans 9y ago

TL;DR: this is an approach that simplifies the production of light field cameras (cameras that measure both color and angle of incoming light beams): instead of building a grid of microscopic lenses, you use a "random" piece of opaque plastic like scotch tape, and figure out how it modifies incoming light using a calibration phase.

ClassyJacket 9y ago
Could some middle-ground device exist where the lens is still manufactured for this use-case but still lower cost overall as the system has a high tolerance for quality/defects by implementing the calibration step?
- icegreentea2 9y ago
  Sounds reasonable. They basically proved that the worse possible lens they can manufacture can still provide the information they need. I have no idea what the size of the relative trade offs involved are - ie, if you loosen the tolerance of your lens manufacturing by X%, what increase in computation is required. Exciting times!
tischler 9y ago
Here is a similar approach using water droplets instead of a diffuse film, but they don't go as far as performing 3D reconstruction from the light field and the ray directions are not calibrated by a calibration pattern but by inferring a 3D model of the droplets: https://light.cs.uni-bonn.de/4d-imaging-through-spray-on-optics/ https://light.cs.uni-bonn.de/4d-imaging-through-spray-on-opt...
- frozenport 9y ago
  It's 2D, because they record a field U(x,y) and then propagate it up and down by convolving with a 2D function. This works for mesoscopic structures but not for small things like human cells.

deleted 9y ago

[deleted]

eeZah7Ux 9y ago

I wonder if unscrewing the lens on a cheap USB camera could lead to some interesting pictures. Any lensless camera Open Source project around? EDIT: A library, not the camera itself

namibj 9y ago
Axiom
crusso 9y ago
You still need the diffraction grating-like piece in front of the sensor array.
solarkraft 9y ago
Whew, we just learned about the research - but the paper is here. I'd be excited for such a project.
lawall04 9y ago
the paper itself has a link to the open-source code: http://www.laurawaller.com/research/diffusercam/ http://www.laurawaller.com/research/diffusercam/

mrow84 9y ago

Can someone explain why they can't apply the same reconstruction technique to the data that would be captured without a diffuser; i.e. why the diffuser is required?

icegreentea2 9y ago
I think the addition of the diffuser adds the 'depth response' to the camera. Imagine that you had two point light sources located some distance away from the sensor such that one is directly in front of the other. They're magical point light sources so the light from the rear one just passes through the front. If you didn't have anything in front of the sensor, there would be no way to distinguish that there are two point sources, as opposed to a single point source with a non uniform output.
radarsat1 9y ago
I think the diffuser ensures that there is a random distribution of incidence angle of photons. Then, if I understand, an optimisation is basically used to figure out what pixels correspond to what angles. I think the actual angle calculation is skipped and the reconstruction is done directly, but the "angle" concept is implicitly taken into account by the fact that it is needed for the 3D reconstruction to take place.
- hammock 9y ago
  In essence, the diffuser is the lens (arranging the light in a usable way onto a sensor), just not a lens as we know it.
frumiousirc 9y ago
It's a good question and I think strikes at the heart of the idea. A lens transforms a family of rays to a pixel location. Given knowledge of that pixel's intensity there is a degenerate solution for the original ray (in terms of it's location and direction at some plane). This degeneracy is one thing that leads to blurry photos. The micro lens camera referenced in the article spreads this family over more pixels in a known, analytic way to make the solution more unique. In principle it suffers from the same degeneracy but each micro lens limits the possible location of rays so if any of the pixels under it are hit then the general location is set and the exact pixel in the group determines the direction. The diffuser works similarly but spreads direction and position location over many pixels and in a random way (seeded by the material and it's precise placement). While this spread can not be calculated it can be discovered through calibration with known point sources. In both these latter cases one inverts this analytic or calibrated ray->pixel matrix and applies that to the measured pixels to reconstruct the rays that have likely caused the measurement. In the case of the diffuse "lens", the required matrix inversion can be computationally expensive at best and impossible at worse. However, the methods of compressed sensing (in particular L1 regularization) allow an approximate inversion to be done in a relatively fast manner.

jaclaz 9y ago

For a moment I thought they could manage to have something like the ESPER machine in Blade Runner.

sp332 9y ago
Lytro's light-field cameras can do this. https://vimeo.com/102302646 https://vimeo.com/102302646
- jaclaz 9y ago
  Yes, I know, thanks, but at the end of the day the Lytro nice technology allows for focus and depth of field correction (besides some slight shift in pespective), but once the "wow" effect has faded away (I mean for "plain" photography), that's it. For cinema, VR and CGI it is simply great of course.
  - sp332 9y ago
    Based on the movie, I'm not sure what features you want besides "move slightly to the right".
    - jaclaz 9y ago
      In the movie, from a plain photography, the ESPER machine manages, somehow, to enter into a reflection, and then expand in the reflection, seeing things that are "not there" in the original image, in practice in the fiction the whole 3d space (even what is on the side of the door) is navigable and viewable: http://thelegalgeeks.com/2017/06/28/admissibility-of-zhoras-photo-in-blade-runner/ http://thelegalgeeks.com/2017/06/28/admissibility-of-zhoras-... What I mean can be better appreciated in this reconstruction: https://typesetinthefuture.com/2016/06/19/bladerunner/ https://typesetinthefuture.com/2016/06/19/bladerunner/ https://typesetinthefuture.files.wordpress.com/2016/06/bladerunner_esper_room_layout_full.png https://typesetinthefuture.files.wordpress.com/2016/06/blade... https://vimeo.com/169392777 https://vimeo.com/169392777
      - 9y ago

crusso 9y ago

It depends on your definition of "lensless". You know those kids' books that have the bumpy plastic coating and when you turn the book one way you see one image - look at it from a different angle and you see another image? This is the same concept. They have a bumpy plastic coating that sends the incoming light in different directions. They do some processing on standard images to determine how the scattering works and then use that scattering pattern to reconstruct new images. I would view the bumpy coating as a myriad of lenses that change the character of the incoming light. We have one of those windows in our bathroom with the glass that is warped so that it breaks up the light so much that it gives you privacy. I've often thought that it would be a fun project to create a camera system that you could calibrate to decrypt that scattered image by placing a known image behind the window and pre-determining how the light waves are refracted. It's cool to see that someone implemented something similar.

saycheese 9y ago
On the same note, depends on how you define “scanning” sense they’re clearly scanning. You can do the same thing they’re doing with a bunch of pinhole, video feed with variation in the lighting, etc.
- nwatson 9y ago
  I think "scanning" would imply compositing info gathered at different instances of time to reconstruct 3-D models ... e.g., LIDAR captures a point cloud with points measured at different times and tries to reconstruct objects from that point cloud. Perhaps the time it takes LIDAR to gather the whole point cloud is small, but objects will drift during that time and the reconstruction algorithm might need to account for this. On the other hand, the article's method captures all info in one instantaneous image and reconstructs based on the known, recorded, bias of the irregular lens and the implied 3-D locations of objects.
  - Retric 9y ago
    If you want to be really technical. Each data-point is not collected at the exact same instant.

saycheese 9y ago

Here’s the research paper: https://pdfs.semanticscholar.org/9cff/c0c80b1ae3c1b773b761f37c66e58890639e.pdf https://pdfs.semanticscholar.org/9cff/c0c80b1ae3c1b773b761f3...

bhouston 9y ago

Reminds me of insect eyes.

visarga 9y ago

I especially like the applications in cheap and compact 3D sensing and brain neural interfaces. Very exciting!

solarkraft 9y ago

Article and especially the title are of pretty low quality. Assuming the voxels are surface voxels, how is it even theoretically possible to turn 1 million pixels into 100 million voxels? This means you get 100x the xy resolution AND depth information out of this process. I'm sceptical of that claim. As crusso already mentioned lenses and scanning are essential parts of image capture, a lens being needed to direct the light to the sensor somehow and scanning to actually read out the image. "Using diffuse foils to replace microlens arrays" would probably be a more fitting and still teasing headline. Or "Diffuse foils can replace microlens arrays for 3D imaging", perhaps. Article aide, the research seems very sound and very cool. It demonstrates another case of extracting high quality information from low quality sensors - something I think we'll be seeing a lot more of. Another previously precisely manufactured piece of hardware is being replaced by a software-supported low-quality part through optimizations that in their spirit remind me of the Google Pixel's camera and that drone that can fly (steer) with one rotor.

nsaote3ntean 9y ago
> This means you get 100x the xy resolution AND depth information out of this process. How so? Isn't 1000 x 1000 = 1,000,000 So it's the same x/y resolution with 100x the z resolution. They extract the z by figuring out the different direction that light hitting each sensel is coming from. (And probably doing additional processing.)
rleigh 9y ago
We already have commercial systems which do this. See PhaseFocus for example (http://www.phasefocus.com/technology-virtual-lens/ http://www.phasefocus.com/technology-virtual-lens/). This uses proprietary deconvolution algorithms to reconstruct a 3D volume from diffraction patterns in unfocussed (or partially focussed) 2D planes. See also X-ray crystallography, which samples in Fourier space and uses a computed transform to reconstruct the 3D image. This new microlens technology is very neat, but the processing problem has been solved for several different applications for several years now. Reconstructing a 3D volume is absolutely possible, and while this new development will undoubtedly require new algorithms to work, it's already got a sound basis in existing technology in use today.
9y ago

punnerud 9y ago

The Research article (PDF): https://www.osapublishing.org/DirectPDFAccess/3ADDF00A-E071-C39B-B3DA82E301EA25C6_380297/optica-5-1-1.pdf?da=1&id=380297&seq=0 https://www.osapublishing.org/DirectPDFAccess/3ADDF00A-E071-...

kelvin0 9y ago

This could also be used to analyze some material used as the diffuser. The 'shape' of the caustics captured by the sensor with a given electromagnetic 'lightsource' can probably yield some interesting information regarding the diffuser. Kinda like an spectrograph works.

p1mrx 9y ago

Would it be possible to run this system in reverse, and make a holographic display?

FractalNerve 9y ago

This makes the Holographic_imager a reality, hooray! [1] http://memory-alpha.wikia.com/wiki/Holographic_imager http://memory-alpha.wikia.com/wiki/Holographic_imager To me this is the most major breakthrough I've heard in the recent years, which can and will hopefully affect everything. Using the extra CMOS Chip on your flagship smartphone will allow for taking 3D and soon Holographic pictures! How Amazing! I remember there was an AI trained to turn 2D pictures into 3D [2], combining that with the NPU Chip on smartphones can truly make this happens very soon. [2] http://www.dailymail.co.uk/sciencetech/article-4904298/The-AI-turn-selfie-3D-image.html http://www.dailymail.co.uk/sciencetech/article-4904298/The-A...