6 ms·

Record-breaking camera keeps everything between 3 cm and 1.7 km in focus

108 points by bsenftner 4y ago

Nihilartikel 4y ago
I'm still waiting for the day when the entire back of every cellphone is a flat compound eye surface lightfield camera.
- mateo1 4y ago
  It has proven much cheaper to guess all the extra information rather than capture them.
  - Nihilartikel 4y ago
    Well, sure, _for now_. To assert that that will be true for the future is just a failure of imagination. It's like the assertion that "640k should be enough for anyone". I want my 5gb per snapshot light field information! :)
_justinfunk 4y ago
The actual example images in the article look... bad? Maybe someone who understands optics better could help explain why they look unnatural. Is it an uncanny-valley analogue?
- monkpit 4y ago
  I get what you’re saying about uncanny valley. My first thought when I saw the example image was “maybe it looks weird because my brain expects some parts to be out-of-focus?” I have no idea if that’s true or if it’s just a bad looking picture. The sky looks weird.
  - dsego 4y ago
    It's reconstructed from sub-images, the reconstruction algorithm can probably be improved.
  - lupire 4y ago
    The lack of focal blur prevents depth perception, and the objects don't have "natural scale" clues (like a human or a car) so the photo looks like a flat 2-D collage.
    - loonster 4y ago
      We need eye tracking that would blur the objects that would normally be out of focus for what the person is looking at.
- chucksta 4y ago
  I believe what you are referring to is aberrations? https://expertphotography.com/lens-aberrations-photography/ https://expertphotography.com/lens-aberrations-photography/ Uncanny valley is more your mind not understanding what is being perceived, this is more that the image is not as sharp due to imperfections in the lens. If the lens were perfect it would end up looking like any other picture you see with everything in focus. This is achieved today by stacking say, 3 pictures on top of each other where each picture has a different part of the subject in focus
  - actionfromafar 4y ago
    That link took me to a description of how women should be posed with an "S-shaped figure".
- dv_dt 4y ago
  My opinion is that finished images from professional photographers are cleaned up interpretations. And increasingly so on an automated basis from smart phone image processing stacks. Raw images often don't look as good to the eye, even if they contain fairly complete information that a professional or automation can work with to provide a number of different finishing approaches that are pleasing to human perceptions. This techs raw capture is significantly different from current image sensor data, and their processed image is mostly making it close to a mature "raw" image is sort of how I take the sample photos in the article.
  - hedora 4y ago
    I agree with you about modern phone cameras, but I have an old digital with minimal postprocessing, and the images come out clear (at a whopping 3 megapixels), as do the raws on my Sony A7. Similarly, film cameras worked fine without piles of neural networks, etc during development. The images in the article are all blurry, but seem decent for a research prototype. Presumably, dumping lots of commercial development resources into the technology will improve it.
  - naet 4y ago
    I think I disagree with your opinion. Raw images often look fantastic if they are captured properly (in focus, exposed correctly). I still work regularly with old film cameras, and the photos develop very very nicely without any digital alterations. When I shoot with my digital I usually save both RAW and JPEG formats so I have the extra raw data if I need it but if I've taken a good shot I minimize my digital alterations. There are some additional tweaks that can be done to a "raw" format camera image that contains extra image data compared to a processed/jpeg type flatter image, but there's no reason they should look bad coming out of the camera. Maybe unbalanced white color values at most.
    - NoSorryCannot 4y ago
      Raw camera data requires an interpretation to be displayed on a monitor at all and usually needs a lot of processing to produce a natural-looking photo on the screen. Demosaicing at the very least but gamma curve correction, color response correction, etc. as well.
- Sohcahtoa82 4y ago
  We rely a LOT on focus to get depth cues from 2D images. When everything is made to be in focus, we lose depth cues and everything looks wrong. That said, they certainly could have created better example images. Like, I would have placed an object like a figurine on an outdoor table with a city skyline in the background and taken a picture of the object from about a foot away. Normally, this would put the object in focus with the city being blurred, as well as part of the table. It would have made it very clear what they were going for.
MBCook 4y ago
The seems sort of like what Lytro was doing, just taken one step further. Instead of focusing the pictures after the fact, they take the light field and combine the different focus levels to produce one fully in-focus picture.
- lupire 4y ago
  > combine the different focus levels to produce one fully in-focus picture. That's not the novel. part. Everyone's phone and DSLR with focus stacking does this. The novel part is achieving a deeper range of depth of field.
- dTal 4y ago
  Lytro could also produce a fully in-focus picture. And a light field isn't made of "different focus levels", but rather the reverse - you can generate different virtual focal planes using the lightfield (Lytro's gimmick).
moffkalast 4y ago
Focus stacking program: "Look at what they need to mimic a fraction of our power."
- Etheryte 4y ago
  While handy, focus stacking requires taking multiple images which means you're limited to static subjects only. If I'm not mistaken, the proposed new approach can just as well be used to take pictures of moving subjects.
  - droopyEyelids 4y ago
    I wonder if instantaneous focus stacking could be possible with a module of cameras.
    - moffkalast 4y ago
      Yeah if you had multiple sensor outputs with different known fixed focuses it could likely be done on the fly.
    - Etheryte 4y ago
      The limiting factor here is that you need your lens to shift focus. So if you wanted to take a-la five simultaneous shots for focus stacking you would need five separate lenses at different focal distances. Since five lenses can't occupy the same physical space you're going to end up with five slightly different perspectives. It might be possible to computationally correct the perspective between them to a certain extent, but I'm not sure how good the end result could theoretically be.
      - droopyEyelids 4y ago
        It could also be done through partial mirrors and prisms, but would of course require a brightly lit field or suffer ISO tradeoffs
- ipsum2 4y ago
sudobash1 4y ago
It seems to me, based off of their sample picture that the advantage would not be so much in getting one single frame where everything is in-focus. Instead I would be more excited about being able to "change" the focus after the photo is taken. I assume that would be equally possible with this camera (depending on how the raw data is stored).
- latexr 4y ago
  > Instead I would be more excited about being able to "change" the focus after the photo is taken. Lytro was doing that in 2012: https://en.wikipedia.org/wiki/Lytro https://en.wikipedia.org/wiki/Lytro
- ur-whale 4y ago
  For artistic purposes, changing the focus may make sense indeed. For most other (e.g. sensing) applications ... why would you want to ruin your data by applying a convolution to it and losing information?
- shagie 4y ago
  About a decade ago, I had one of those rectangular Lytro cameras. https://en.wikipedia.org/wiki/Lytro https://en.wikipedia.org/wiki/Lytro And yes, you could change the focus on a picture that was hosted on the web. Unfortunately, those photos, as they required the Lytro site to work, no longer work and I can't find any working archives of them. ( Lytro is winding down https://news.ycombinator.com/item?id=16705391 https://news.ycombinator.com/item?id=16705391 ) But yes, you could - the Lytro's first camera section of https://youtu.be/m8c1__yzX2Q https://youtu.be/m8c1__yzX2Q showed some of the refocusing.
  - tecleandor 4y ago
    I got a Lytro ILLUM to test before they began selling it in Spain years ago. Seemed like an interesting toy, specially the depth map that was unusual for that time (now there are cellphones capturing depth maps with different techniques). If I check my NAS there's the dump of the SD card I used that day, but I wouldn't know how to open those files or process them, as I guess the software is not available anymore and I don't know if anybody has hacked the file format.
Overtonwindow 4y ago
How is this different from the Lytro?
- lupire 4y ago
  This camera has larger depth of field. The article explains the tech, but I don't know how much of Lytro's tech is similar. Both are "light field".
  - tecleandor 4y ago
    From a quick glance they look quite similar in its basic concept. I'd say the main difference is in the metalens that splits the image, that seems quite more advanced in this prototype.
sandworm101 4y ago
How is this any different than taking multiple pictures at different focal lengths? The fancy optics split near and far objects towards different areas of the sensor. Why bother? Why not just use two or more lens paths each aimed at different parts of the sensor? Or use multiple sensors?
- ghastmaster 4y ago
  Your proposals increases physical complexity(think manufacturing price increases) and therefore more points of failure.
smm11 4y ago
Killer robots get yet another major advantage.
hedora 4y ago
The gallery somehow defeats Firefox's "disable disabling pinch to zoom" setting, and it's not clear if the New Atlas article has the full size images anyway. Here's the primary source, with links to full size images: https://www.nature.com/articles/s41467-022-29568-y#Fig6 https://www.nature.com/articles/s41467-022-29568-y#Fig6 The university picture, in particular has some strange artifacts in the tree lines. It looks like the neural network is inventing scene details. Also, the aberration figure for the dolls suggests it's doing some sort of object boundary detection to decide what distance each pixel should be at. This seems to have the same problems focus stacking would have. I wonder if traditional signal processing could fix it.
- jcims 4y ago
  >The gallery somehow defeats Firefox's "disable disabling pinch to zoom" setting Happens with Chrome and Safari on iOS as well, but only when you're in the 'view images' mode after clicking on an image. For me I honestly can't think of anything more hilariously disproportionate than my physiological response to disabled pinch-zoom. It's a whole body experience, starting with this weird cramping/tension in the back of my left hand, which is holding the phone, then a wave of cortisol/adrenaline cocktail washes over me...followed quickly by a stream of expletives and an attempt to crush the phone with my grip or some other destructive outlet. It's wholly irrational and feels entirely limbic in nature, probably a sign of how deep the pinch-zoom gesture is engrained in the psyche.
alberth 4y ago
I guess the CSI meme about "Enhance, Enhance, Enhance" will no longer be a meme but reality.
jhgb 4y ago
"Trilobyte", heh. That's a funny typo.
steeve 4y ago
Isn't it what Litro was doing back in the day? A friend had one, it was very impressive. The form factor was weird though.

108 points by bsenftner 4y ago

Nihilartikel 4y ago

I'm still waiting for the day when the entire back of every cellphone is a flat compound eye surface lightfield camera.

mateo1 4y ago
It has proven much cheaper to guess all the extra information rather than capture them.
- Nihilartikel 4y ago
  Well, sure, _for now_. To assert that that will be true for the future is just a failure of imagination. It's like the assertion that "640k should be enough for anyone". I want my 5gb per snapshot light field information! :)

_justinfunk 4y ago

The actual example images in the article look... bad? Maybe someone who understands optics better could help explain why they look unnatural. Is it an uncanny-valley analogue?

monkpit 4y ago
I get what you’re saying about uncanny valley. My first thought when I saw the example image was “maybe it looks weird because my brain expects some parts to be out-of-focus?” I have no idea if that’s true or if it’s just a bad looking picture. The sky looks weird.
- dsego 4y ago
  It's reconstructed from sub-images, the reconstruction algorithm can probably be improved.
- lupire 4y ago
  The lack of focal blur prevents depth perception, and the objects don't have "natural scale" clues (like a human or a car) so the photo looks like a flat 2-D collage.
  - loonster 4y ago
    We need eye tracking that would blur the objects that would normally be out of focus for what the person is looking at.
chucksta 4y ago
I believe what you are referring to is aberrations? https://expertphotography.com/lens-aberrations-photography/ https://expertphotography.com/lens-aberrations-photography/ Uncanny valley is more your mind not understanding what is being perceived, this is more that the image is not as sharp due to imperfections in the lens. If the lens were perfect it would end up looking like any other picture you see with everything in focus. This is achieved today by stacking say, 3 pictures on top of each other where each picture has a different part of the subject in focus
- actionfromafar 4y ago
  That link took me to a description of how women should be posed with an "S-shaped figure".
dv_dt 4y ago
My opinion is that finished images from professional photographers are cleaned up interpretations. And increasingly so on an automated basis from smart phone image processing stacks. Raw images often don't look as good to the eye, even if they contain fairly complete information that a professional or automation can work with to provide a number of different finishing approaches that are pleasing to human perceptions. This techs raw capture is significantly different from current image sensor data, and their processed image is mostly making it close to a mature "raw" image is sort of how I take the sample photos in the article.
- hedora 4y ago
  I agree with you about modern phone cameras, but I have an old digital with minimal postprocessing, and the images come out clear (at a whopping 3 megapixels), as do the raws on my Sony A7. Similarly, film cameras worked fine without piles of neural networks, etc during development. The images in the article are all blurry, but seem decent for a research prototype. Presumably, dumping lots of commercial development resources into the technology will improve it.
- naet 4y ago
  I think I disagree with your opinion. Raw images often look fantastic if they are captured properly (in focus, exposed correctly). I still work regularly with old film cameras, and the photos develop very very nicely without any digital alterations. When I shoot with my digital I usually save both RAW and JPEG formats so I have the extra raw data if I need it but if I've taken a good shot I minimize my digital alterations. There are some additional tweaks that can be done to a "raw" format camera image that contains extra image data compared to a processed/jpeg type flatter image, but there's no reason they should look bad coming out of the camera. Maybe unbalanced white color values at most.
  - NoSorryCannot 4y ago
    Raw camera data requires an interpretation to be displayed on a monitor at all and usually needs a lot of processing to produce a natural-looking photo on the screen. Demosaicing at the very least but gamma curve correction, color response correction, etc. as well.
Sohcahtoa82 4y ago
We rely a LOT on focus to get depth cues from 2D images. When everything is made to be in focus, we lose depth cues and everything looks wrong. That said, they certainly could have created better example images. Like, I would have placed an object like a figurine on an outdoor table with a city skyline in the background and taken a picture of the object from about a foot away. Normally, this would put the object in focus with the city being blurred, as well as part of the table. It would have made it very clear what they were going for.

MBCook 4y ago

The seems sort of like what Lytro was doing, just taken one step further. Instead of focusing the pictures after the fact, they take the light field and combine the different focus levels to produce one fully in-focus picture.

lupire 4y ago
> combine the different focus levels to produce one fully in-focus picture. That's not the novel. part. Everyone's phone and DSLR with focus stacking does this. The novel part is achieving a deeper range of depth of field.
dTal 4y ago
Lytro could also produce a fully in-focus picture. And a light field isn't made of "different focus levels", but rather the reverse - you can generate different virtual focal planes using the lightfield (Lytro's gimmick).

moffkalast 4y ago

Focus stacking program: "Look at what they need to mimic a fraction of our power."

Etheryte 4y ago
While handy, focus stacking requires taking multiple images which means you're limited to static subjects only. If I'm not mistaken, the proposed new approach can just as well be used to take pictures of moving subjects.
- droopyEyelids 4y ago
  I wonder if instantaneous focus stacking could be possible with a module of cameras.
  - moffkalast 4y ago
    Yeah if you had multiple sensor outputs with different known fixed focuses it could likely be done on the fly.
  - Etheryte 4y ago
    The limiting factor here is that you need your lens to shift focus. So if you wanted to take a-la five simultaneous shots for focus stacking you would need five separate lenses at different focal distances. Since five lenses can't occupy the same physical space you're going to end up with five slightly different perspectives. It might be possible to computationally correct the perspective between them to a certain extent, but I'm not sure how good the end result could theoretically be.
    - droopyEyelids 4y ago
      It could also be done through partial mirrors and prisms, but would of course require a brightly lit field or suffer ISO tradeoffs
ipsum2 4y ago

sudobash1 4y ago

It seems to me, based off of their sample picture that the advantage would not be so much in getting one single frame where everything is in-focus. Instead I would be more excited about being able to "change" the focus after the photo is taken. I assume that would be equally possible with this camera (depending on how the raw data is stored).

latexr 4y ago
> Instead I would be more excited about being able to "change" the focus after the photo is taken. Lytro was doing that in 2012: https://en.wikipedia.org/wiki/Lytro https://en.wikipedia.org/wiki/Lytro
ur-whale 4y ago
For artistic purposes, changing the focus may make sense indeed. For most other (e.g. sensing) applications ... why would you want to ruin your data by applying a convolution to it and losing information?
shagie 4y ago
About a decade ago, I had one of those rectangular Lytro cameras. https://en.wikipedia.org/wiki/Lytro https://en.wikipedia.org/wiki/Lytro And yes, you could change the focus on a picture that was hosted on the web. Unfortunately, those photos, as they required the Lytro site to work, no longer work and I can't find any working archives of them. ( Lytro is winding down https://news.ycombinator.com/item?id=16705391 https://news.ycombinator.com/item?id=16705391 ) But yes, you could - the Lytro's first camera section of https://youtu.be/m8c1__yzX2Q https://youtu.be/m8c1__yzX2Q showed some of the refocusing.
- tecleandor 4y ago
  I got a Lytro ILLUM to test before they began selling it in Spain years ago. Seemed like an interesting toy, specially the depth map that was unusual for that time (now there are cellphones capturing depth maps with different techniques). If I check my NAS there's the dump of the SD card I used that day, but I wouldn't know how to open those files or process them, as I guess the software is not available anymore and I don't know if anybody has hacked the file format.

Overtonwindow 4y ago

How is this different from the Lytro?

lupire 4y ago
This camera has larger depth of field. The article explains the tech, but I don't know how much of Lytro's tech is similar. Both are "light field".
- tecleandor 4y ago
  From a quick glance they look quite similar in its basic concept. I'd say the main difference is in the metalens that splits the image, that seems quite more advanced in this prototype.

sandworm101 4y ago

How is this any different than taking multiple pictures at different focal lengths? The fancy optics split near and far objects towards different areas of the sensor. Why bother? Why not just use two or more lens paths each aimed at different parts of the sensor? Or use multiple sensors?

ghastmaster 4y ago
Your proposals increases physical complexity(think manufacturing price increases) and therefore more points of failure.

smm11 4y ago

Killer robots get yet another major advantage.

hedora 4y ago

The gallery somehow defeats Firefox's "disable disabling pinch to zoom" setting, and it's not clear if the New Atlas article has the full size images anyway. Here's the primary source, with links to full size images: https://www.nature.com/articles/s41467-022-29568-y#Fig6 https://www.nature.com/articles/s41467-022-29568-y#Fig6 The university picture, in particular has some strange artifacts in the tree lines. It looks like the neural network is inventing scene details. Also, the aberration figure for the dolls suggests it's doing some sort of object boundary detection to decide what distance each pixel should be at. This seems to have the same problems focus stacking would have. I wonder if traditional signal processing could fix it.

jcims 4y ago
>The gallery somehow defeats Firefox's "disable disabling pinch to zoom" setting Happens with Chrome and Safari on iOS as well, but only when you're in the 'view images' mode after clicking on an image. For me I honestly can't think of anything more hilariously disproportionate than my physiological response to disabled pinch-zoom. It's a whole body experience, starting with this weird cramping/tension in the back of my left hand, which is holding the phone, then a wave of cortisol/adrenaline cocktail washes over me...followed quickly by a stream of expletives and an attempt to crush the phone with my grip or some other destructive outlet. It's wholly irrational and feels entirely limbic in nature, probably a sign of how deep the pinch-zoom gesture is engrained in the psyche.

alberth 4y ago

I guess the CSI meme about "Enhance, Enhance, Enhance" will no longer be a meme but reality.

jhgb 4y ago

"Trilobyte", heh. That's a funny typo.

steeve 4y ago

Isn't it what Litro was doing back in the day? A friend had one, it was very impressive. The form factor was weird though.