6 ms·
DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model
- valine 2y agoI get the feeling that lens free cameras are the future. Obviously the results here are no where near good enough, but given the rapid improvement of diffusion models lately the trajectory seems clear. Would love to lose the camera bump on the back of my phone.
- xandrius 2y agoSo you just get something based off GPS, time of day and rotation? Or no photos anymore?
- smallerize 2y agohttps://www.standard.co.uk/news/tech/ai-camera-images-paragraphica-b1084929.html https://www.standard.co.uk/news/tech/ai-camera-images-paragr...
- valine 2y agoNot sure what this has to do with anything. The paper I was commenting on is using diffusion models to parse raw light hitting the sensor as an alternative to a glass lens. No one wants an image generation model hooked up to weather data— that’s kind of ridiculous.
- a1o 2y agoOh god, we are going to make lens a premium feature now aren't we?
- chpatrick 2y agoIt would be pretty great if cheap phones can get good cameras with this technology.
- 082349872349872 2y agoIf/when cheap enough, even non-phone devices (POS terminals, vending machines, etc) will have cameras; will living in a camera-free environment become the premium feature?
- chpatrick 2y agoCameras are already cheap enough to put in everything.
- fpoling 2y agoThere is no camera. It is just a diffusion model trained on a big set that tries to reconstruct the picture. Essentially this is not much different from what Samsung did with their AI-enhanced camera that detected a moon and replaced that with high-resolution picture.
- holoduke 2y agoAnd yet, we will probably get all camera software on our phones with unlimited zoom and details. Turning grain in crispy clear pictures. Inpainting, outpainting etc. In 5 years from now everybody uses it. Everything becomes fake.
- chpatrick 2y agoThe pictures in the paper are pretty damn close, and this is just a prototype. Plus, as you said, phones already have AI filters.
- teruakohatu 2y agoThis is quite amazing that using a diffuser rather than a lens, then using a diffusion model can reconstruct an image so well. The downside of this is that is heavily relies on the model to construct the image. Much like those colorisation models applied to old monochrome photos, the results will probably always look a little off based on the training data. I could imagine taking a photo of some weird art installation and the camera getting confused. You can see examples of this when the model invented fabric texture on the fabric examples and converted solar panels to walls.
- nine_k 2y agoThe model basically guesses and reinvents what these diffuse pixels might be. It's more like a painter producing a picture from memory. It inevitably means that the "camera" visually parses the scene and then synthesizes its picture. The intermediate step is a great moment to semantically edit the scene. Recolor and retexture things. Remove some elements of the scene, or even add some. Use different rendering styles. Imagine pointing such a "camera" at person standing next to a wall, and getting a picture of the person with their skin lacking any wrinkles, clothes looking more lustrous as if it were silk, not cotton, and the graffiti removed from the wall behind. Or making a "photo" that turns a group of kids into a group of cartoon superheroes, while retaining their recognizable faces and postures. (ICBM course, photo evidence made with digital cameras should long have been inadmissible in courts, but this would hasten the transition.)
- neom 2y agoKinda reminds me of this a bit: https://arstechnica.com/gadgets/2020/11/nvidia-used-neural-networks-to-improve-video-calling-bandwidth-by-10x/ https://arstechnica.com/gadgets/2020/11/nvidia-used-neural-n...
- wizzwizz4 2y ago> photo evidence made with digital cameras should long have been inadmissible in courts Sworn testimony is admissible in courts. I think the "you can just make evidence up" threshold was passed a few thousand years ago. The courts still, mostly, work.
- mjburgess 2y agoDoes a camera without a lens make any physics sense? I cannot see how the scene geometry could be recoverable. Rays of light travelling from the scene arrive in all directions. Intuitively, imagine moving your eye at every point along some square inch. Each position of the eye is a different image. Now all those images overlap on the sensor. If you look at the images in the paper, everything except their most macro geometry and colour pallet is clearly generated -- since it changes depending on the prompt. So at a guess, the lensless sensor gets this massive overlap of all possible photos at that location and so is able, at least, to capture minimal macro geometry and colour. This isn't going to be a useful amount of information for almost any application.
- orbital-decay 2y agoYes, compressed sensing cameras do exactly that. They reconstruct a photometrically correct image without the need for focusing optics or pixel arrays. They have limitations (not fundamental ones though) but they're useful for special use cases like X-ray or LWIR single-pixel imaging where focusing optics and pixel arrays are impossible or expensive. It was first used on X-ray telescopes in 1970's in the form of coded aperture, before the grazing incidence mirrors.
- nialv7 2y agoIt's lensless, but it's not just a naked sensor. It still has an optical element - read the paper.
- ziofill 2y ago+1 for the Thor labs candy box
- deleted 2y ago[deleted]
- karmakaze 2y agoThis is not a 'camera' per se. It's more like a human vision system that samples light and hallucinates an appropriate image based on context. The image is constructed from the data more than it is reconstructed. And like human vision, it can be correct more often than not to be useful.
- cactusplant7374 2y agoThanks for the summary. I was looking for this.
- deleted 2y ago[deleted]
- numpad0 2y agoIt's also like some of space telescopes. Computational photography stuff.
- Snoozus 2y agoIt's not as crazy as it seems, a pinhole camera doesn't have any lenses either and works just fine. The hole size is a tradeoff between brightness and detail. This one has many holes and uses software to puzzle their images back together.
- dekhn 2y agoFor those interested in various approaches to lens-free imaging, Laura Waller at Berkeley has been pursuing this area for some time. https://waller-lab.github.io/DiffuserCam/ https://waller-lab.github.io/DiffuserCam/ https://waller-lab.github.io/DiffuserCam/tutorial.html https://waller-lab.github.io/DiffuserCam/tutorial.html includes instructions and code to build your own https://ieeexplore.ieee.org/abstract/document/8747341 https://ieeexplore.ieee.org/abstract/document/8747341 https://ieeexplore.ieee.org/document/7492880 https://ieeexplore.ieee.org/document/7492880
- astrange 2y agoNote the difference between a "diffuser" and a "diffusion model" here.
- momoschili 2y agoThe article mentions the use of a diffractive mask. This can be very generally associated with an optical diffuser in its function. And yes, Laura Waller at UC Berkeley has been one of the pioneers in this research for a few decades.
- 6gvONxR4sf7o 2y agoRe: is this a camera or not, I recently realized that my fancy mirrorless camera is closer to this than i’d previously thought. The sensor has a zillion pixels but each one only measures one color. for example, the pixel at index (145, 2832) might only measure green, while its neighbor at (145, 2833) only measures red. So we use models to fill in the blanks. We didn’t measure redness at (145, 2832) so we guess based on the redness nearby. This kind of guessing is exactly what modern CV is so good at. So the line of what is a camera and what isn’t is a bit blurry to begin with.
- klysm 2y agoThe structure you are referring to is a Bayer Array. Algorithms that do the guessing are called debayering algorithms
- 6gvONxR4sf7o 2y agoI think that’s just a particular (very common) case. In general it’s called demosaicing, right?
- klysm 2y agoNot sure - I’ve seen both used and assumed they were interchangeable. Is there a more general case of a bayer array?
- 6gvONxR4sf7o 2y agoIn general I think you just call it a color filter array.
- astrange 2y agoThey don't use models (although you certainly could.) They usually use plain old closed-form solutions.
- 6gvONxR4sf7o 2y ago
- Dibby053 2y agoThis would be impressive if the examples weren't taken from the same dataset (Laion-5B) that was used to train the Stable Diffusion model it's using.
- Legend2440 2y agoThey also show actual images they took of real scenes. See figure 6.
- Dibby053 2y agoYou're right, I should have read everything before commenting... sorry.
- Thomashuet 2y agoI don't understand the use of a textual description. In which scenario do you not have enough space for a lens and yet have a textual description of the scene?
- pvillano 2y agoI wonder how it "reacts" to optical illusions? The ones we're familiar with are optimized for probing the limits of the human visual system, but there might be some overlap
- albert_e 2y agoso this is like use of (in a different species) light sensitive patches of skin instead of the eye balls (lenses) that most animals on earth evolved ? interesting. even if this does not immediately replace traditional cameras and lenses... I am wondering if this can add a complementary set of capabilities to a traditional camera say next to a phone's camera bump/island/cluster...so that we can drive some enhanced use cases maybe store the wider context in raw format alongside the EXIF data ...so that future photo manipulation models can use that ambient data to do more realistic edits / in painting / out painting etc? I am thinking this will benefit 3D photography and video graphics a lot if you can capture more of the ambient data, not strictly channeled through the lenses
- xg15 2y agoOh great, waiting for the first media piece where pictures from this "camera" are presented as evidence. (Or the inverse, where actual photographic evidence is disputed because who knows if the camera didn't have AI stuff built in)