5 ms·

We have had this for years. The term is Light Field Camera. There was an attempt several years back to do this with the Lytro camera. It worked but had a few ma

• GloriousKoji 4y ago
  Resolution will always be an issue since it's a trade off of dedicating a fixed area to single 2D plane or a fraction of it to multiple planes in 3D.
  • jbay808 4y ago
    That's true, but it might be mainly a cost issue. You can make up for the resolution loss by using a much larger sensor; since the microlenses are very thin, it won't increase the weight or bulk of the device very much. Just the cost.
    • aaroninsf 4y ago
      This. The overview of the 48 megapixel camera on the iPhone 14 talked chipperly about the fact that the user-land resolution is already decimated (well, quadrimated?) 4 to 1 to 12mp images, with the subtext "but those are highly determined optimized pixels!"
      • wyager 4y ago
        The 48mp sensor on the iPhone seems like a gimmick - even with purely diffraction-limited optics, I doubt you could seriously increase optical bandwidth at that sensor size beyond, say, 10-20Mp, to say nothing of aberration limiting and shot noise constraints. Just now we're starting to see 40Mp APS-C sensors, and people are debating if it's worth it to do that on a sensor that's like 10x larger than the biggest iphone sensor.
        • foldr 4y ago
          Some simple calculations suggest otherwise. The iPhone 14 Pro 48MP camera has a pixel size of 1220 nanometers. The diameter of the Airy disc at f1.78 for 500nm light is 2170 nanometers. Seems like Apple has designed things so that they're just on the right side of the Rayleigh criterion. The Rayleigh criterion is admittedly pretty lax from the point of view of imaging. People often talk about digital cameras being "diffraction limited" if the diameter of the Airy disc is larger than the dimension of a pixel (which is clearly the case for the iPhone). However, AFAIK, there is no strict limit here. It's something of an open question how good smart sharpening algorithms can get – especially when they're allowed to 'cheat' by using knowledge of the statistical properties of typical real world scenes. The pixel size for a 48MP APS-C sensor is about 2900 nanometers, so it's not 10x bigger (even if you compare area).
          • wyager 4y ago
            > It's something of an open question how good smart sharpening algorithms can get True, but you're strictly limited by what I assume are microscopic gate capacitances on pixels that small. The maximum number of electrons in an iphone pixel are probably less than 10k, leading to less than 7 bits of maximum theoretical entropy per px from shot noise limits. This is going to sharply curtail (no pun intended) your ability to "back out" airy disk overlap.
            • deleted 4y ago
              [deleted]
            • foldr 4y ago
              Stacking multiple exposures should help with that.
  • nico 4y ago
    I wonder if something like DAIN[1] could be used to take fewer planes and then interpolate. Maybe that way the tradeoff wouldn’t be as bad? 1: https://twitter.com/karenxcheng/status/1564636065410953217 https://twitter.com/karenxcheng/status/1564636065410953217
• tgflynn 4y ago
  What is the basic principle behind this approach. A lens essentially allows you to map light rays (which have an origin and a direction vector) to position on the sensor (a pinhole does the same but lets through much less light). Did Lytro have a sensor that could actually measure both the position and direction of an incident light ray ?
  • NovemberWhiskey 4y ago
    IIRC there was a micro-lens array between the objective lens and the sensor; each of the micro lenses added a separate, overlapping contribution on the sensor and with a bunch of signal processing magic you somehow back out the position/direction. Reading the article, I don't at all see what the difference is between the Lytro camera and what's described there - although I'm not an optics expert.
    • tgflynn 4y ago
      One difference may be that, as I recall, Lytro was focused on (2D) photography, while this seems to be targeting 3D, which may be a much more compelling application.
      • NovemberWhiskey 4y ago
        I don't know - it seems pretty much the same - the whole point of the Lytro camera was that you had sufficient information in the recorded image to select your point of focus and depth in post-processing, which is essentially the same as having depth data.
        • tgflynn 4y ago
          The same information is used but there's a big difference between using it to make prettier photos versus reconstructing 3D scenes. The second application domain seems to have much more potential for revolutionary change than the first (to me at least). The average person, or probably even the average VC, isn't going to read a pitch about being able to take photos without having to adjust lens focus and say "gee, with that technology you could build a 3D model as easily as taking a photo".
• sp332 4y ago
  The article says this is an advance in the speed of processing light field images. Also the bit about seeing around an opaque object, which I'm not sure what that means but could be an interesting first?
  • klodolph 4y ago
    > Also the bit about seeing around an opaque object, which I'm not sure what that means but could be an interesting first? It's just a property of light field cameras. Technically, other cameras can do it to some extent too. If there's some opaque object with another object behind it, 1. If you focus on the front object, you'll see the object behind it--blurred. 2. If you focus on the back object, you'll be able to see the back object behind the blur of the front object. With a light field camera, you can essentially "cut away" the blurry parts and get them in focus. You're not able to see anything new, you're just able to see things in the original image more clearly.
    • sp332 4y ago
      Sure, but that's not new. I was wondering if they did something extra.
  • grumbel 4y ago
    Applied Science has a good video[1] showing how you can look around objects. He is using a regular lens there, but with a lightfield camera you can emulate basically any lens you want in software as long as the captured lightfield is big enough. [1] https://www.youtube.com/watch?v=iJ4yL6kaV1A https://www.youtube.com/watch?v=iJ4yL6kaV1A
• fxtentacle 4y ago
  Agree, this appears to be a pure PR piece. So they built a camera with a microlens array and refocussing... Like this one, which has been commercially available for 10+ years? https://raytrix.de/products/ https://raytrix.de/products/
  • riedel 4y ago
    I think the catch is different on this one. They seem to co-optimize the optics hardware and the processing software. They say that new manufacturing allows this kind of stuff. Which I think is cool.Still I wonder how much difference this will realistically make. In my experience (from printed electronics) new customizable manufacturing processes often also come at some cost and expose a lot of new effects, so is difficult to break even with something that keeps one side stable. Although particularly in optics new smartphone lenses I guess have proven how good you can get with such approaches.
• frozenport 4y ago
  Biggest problem was motivation, you could instead capture the whole image and apply a digital blurring filter.
• nradov 4y ago
  The Lytro camera allowed you to choose how to focus the image after taking the exposure. So naturally the only way to do that was with their proprietary software because there was no suitable industry standard. It did then allow for exporting in standard formats like JPEG. They tried to market to sports and nature photographers since it eliminated the need to focus before taking a picture. Thus they would have a better chance of capturing the perfect image at exactly the right instant. Unfortunately, after doing the software focus post-processing the images always looked a little soft. And the autofocus features on DSLRs got faster and more accurate. So there was no market.
  • nomel 4y ago
    > The Lytro camera allowed you to choose how to focus the image after taking the exposure. This isn't technically correct. Everything was in focus, because the aperture was so small. The software would add an artificial blur, using the 3d models generated. So, you chose how to artificially blur the image, after taking the exposure.
• kjeetgill 4y ago
  The title is misleading about the work. The real novelty here is the particular deep-learning processing for the 3D reconstruction.
• cryptonector 4y ago
  > There was also a fair few issues around the fact that the only way to share the photos was a cumbersome vendor lock in site iirc. That's... not how you make a new product popular and make tons of money. That's how you make a new product a niche product. That's how you kill a product. Vendor lock-in is for when you have already acquired the mindshare. Users hate vendor lock-in. To start out having vendor lock-in baked in you'll have to have such an incredibly amazing and useful new product -one that's in its own category- that users will disregard the lock-in for having no choice. The Lytro camera is not such an amazing product. Mindshare is extremely valuable. Vendor lock-in is a tool for when your competition is rising, but it's very risky, as it may close your product to new customers while making your existing customers want to jump ship as soon as practicable. Vendor lock-in is very risky not just for the customer but also for the vendor.
  • jonas21 4y ago
    It's not like they wanted to do this. They had no choice because standard photo viewers can't refocus photos. And a Lytro photo without refocusing is... just a photo, and a low-resolution one at that. So they had to build a UI for refocusing and host the shared photos themselves.
    • tene 4y ago
      They had no choice? Really? Check out this archived post from when they took down their photo sharing site: https://web.archive.org/web/20180327235711/https://lytro.wufoo.com/forms/xfa384s1etqo4m/ https://web.archive.org/web/20180327235711/https://lytro.wuf... If they actually wanted to build a popular product with significant longevity, they could have done FAR more to enable people to build on their file format etc. This "they didn't want to pursue full vendor lock-in at every step of their company; they had no choice" apologism just doesn't stand up to any scrutiny. More likely, they made a bid to capture a greater share of the value, at the cost of limiting the size of their market, and it predictably failed. Or maybe they were prioritizing some kind of acquisition, over building a company with real longevity. I don't really know anything about the company besides a short skim through Google, but it just doesn't seem remotely plausible to me that they really truly wanted to make this accessible to more people and cultivate a larger market and ecosystem, but somehow had literally no choice in pursuing their vendor lock-in strategy. They chose to bet on control over mindshare, and this had predictable consequences.
• somethingsome 4y ago
  Yes, this is a plenoptic camera, the main difference from current methods (e.g., [0]) is that they 'learn' how to reconstruct the images instead on extracting the 3d using optics. Now the resolution is less a problem with plenoptic 2.0 cameras, in [0] they extract full HD images [0] https://doi.org/10.1109/IC3D53758.2021.9687243 https://doi.org/10.1109/IC3D53758.2021.9687243