4 ms·

It would be really cool if you could feed the network a photo with flash that it could use for gathering more information, but then recreated a photo without fl

It would be really cool if you could feed the network a photo with flash that it could use for gathering more information, but then recreated a photo without flash from the non-flash raw. Often flash is not the look people are going for, but would be okay with the flash firing in order to improve the non-flash photo.

y7r4m 7y ago
Absolutely! We recently rebranded our AI solutions from ALLIS (Advanced Low Light Imagine Solution) to ALIIS (All Light Intelligent Imaging Solution) specifically because we are beginning to branch out to handle use cases such as this! As a proof of concept that this task can be tackled directly, a quick search brought up "DeepFlash: Turning a Flash Selfie into a Studio Portrait"[0] Beyond denoising, we are already running experiments with very promising results on haze, lens flare, and reflection removal; super resolution; region adaptive white balancing; single exposure HDR; and a fair bit more. One of the other cooler things we are doing is putting together a unified SDK where our algorithms and neural nets will be able to run pretty much anywhere, on any hardware, using transparent backend switching. (e.g. CPU, GPU, TPU, NPU, DSP, other accelerator ASICs, etc..) [0] https://arxiv.org/abs/1901.04252 https://arxiv.org/abs/1901.04252
- exikyut 7y ago
  Before reading your reply to OP's comment I got to thinking about how the super-resolution process and flash photography might interact (https://news.ycombinator.com/item?id=22905317 https://news.ycombinator.com/item?id=22905317). I get the impression you left the point I got to a long time ago :)
- dmos62 7y ago
  DeepFlash: Turning a Flash Selfie into a Studio Portrait https://www.youtube.com/watch?v=enLmReROhc8 https://www.youtube.com/watch?v=enLmReROhc8
exikyut 7y ago
The way I mistakenly initially parsed this comment gave rise to a potentially-dumb idea/question: What would happen if you - begin capturing video (unsure of fps) on a phone-quality sensor in a near-dark environment - pulse the phone's flash LED(s) like you're taking a photo - do super-resolution on the resulting video to extract a photo... - ...while factoring in the decay in brightness/saturation in consecutive video frames produced by the flash pulse? I vaguely recall reading somewhere that oversaturated photos have more signal in them and are easier to fix than undersaturated. Hmm. IIRC super-resolution worked with 30fps source video for better quality; I wonder if 60fps or 120fps source video would produce better brightness decay data, or whether super-resolution could actually help extract more signal out of the decay sequence too. On the other hand, I'm not sure if super-resolution fundamentally requires largely consistent brightness in order to work as well as it does. :/ Perhaps individual networks could be trained/tuned to specific slices/windows of the brightness gradient. I also wonder if it would be useful to factor the superresolution process into each of the brightness-specific stages or just to do it at the end.
- y7r4m 7y ago
  For the most part, our effort has been focused on single exposure image enhancement, however we are beginning to use recurrent models to improve quality when video information is available. Nonetheless, it's kinda a neat idea, so I tried testing the feasibility of it. I set up a recent flagship phone that claims to have 960fps super-slow-motion video capture next to another phone with a strobe app at 12Hz with a short delay in between pulses. https://www.dropbox.com/s/ha51ntucl3klkcb/cell_flash_960fps.mp4 https://www.dropbox.com/s/ha51ntucl3klkcb/cell_flash_960fps.... There are definitely a few frames where the LED is at an intermediate brightness, however teasing out the exact timings between the flash and the camera may prove to be difficult to correctly synchronize. As for over-saturated images having more signal... although the PSNR calculation may give you a better number, in practice, a region that is over-saturated is just a blob of 1s on the image (assuming float64 pixel values of 0-1) and there is no information there to extract. With a black level near but not at 0, we've found there is often more information hidden in the 'dark noise' than can be discerned by the human eye alone.
  - 7y ago

y7r4m 7y ago

Absolutely! We recently rebranded our AI solutions from ALLIS (Advanced Low Light Imagine Solution) to ALIIS (All Light Intelligent Imaging Solution) specifically because we are beginning to branch out to handle use cases such as this! As a proof of concept that this task can be tackled directly, a quick search brought up "DeepFlash: Turning a Flash Selfie into a Studio Portrait"[0] Beyond denoising, we are already running experiments with very promising results on haze, lens flare, and reflection removal; super resolution; region adaptive white balancing; single exposure HDR; and a fair bit more. One of the other cooler things we are doing is putting together a unified SDK where our algorithms and neural nets will be able to run pretty much anywhere, on any hardware, using transparent backend switching. (e.g. CPU, GPU, TPU, NPU, DSP, other accelerator ASICs, etc..) [0] https://arxiv.org/abs/1901.04252 https://arxiv.org/abs/1901.04252

exikyut 7y ago
Before reading your reply to OP's comment I got to thinking about how the super-resolution process and flash photography might interact (https://news.ycombinator.com/item?id=22905317 https://news.ycombinator.com/item?id=22905317). I get the impression you left the point I got to a long time ago :)
dmos62 7y ago
DeepFlash: Turning a Flash Selfie into a Studio Portrait https://www.youtube.com/watch?v=enLmReROhc8 https://www.youtube.com/watch?v=enLmReROhc8

exikyut 7y ago

The way I mistakenly initially parsed this comment gave rise to a potentially-dumb idea/question: What would happen if you - begin capturing video (unsure of fps) on a phone-quality sensor in a near-dark environment - pulse the phone's flash LED(s) like you're taking a photo - do super-resolution on the resulting video to extract a photo... - ...while factoring in the decay in brightness/saturation in consecutive video frames produced by the flash pulse? I vaguely recall reading somewhere that oversaturated photos have more signal in them and are easier to fix than undersaturated. Hmm. IIRC super-resolution worked with 30fps source video for better quality; I wonder if 60fps or 120fps source video would produce better brightness decay data, or whether super-resolution could actually help extract more signal out of the decay sequence too. On the other hand, I'm not sure if super-resolution fundamentally requires largely consistent brightness in order to work as well as it does. :/ Perhaps individual networks could be trained/tuned to specific slices/windows of the brightness gradient. I also wonder if it would be useful to factor the superresolution process into each of the brightness-specific stages or just to do it at the end.

y7r4m 7y ago
For the most part, our effort has been focused on single exposure image enhancement, however we are beginning to use recurrent models to improve quality when video information is available. Nonetheless, it's kinda a neat idea, so I tried testing the feasibility of it. I set up a recent flagship phone that claims to have 960fps super-slow-motion video capture next to another phone with a strobe app at 12Hz with a short delay in between pulses. https://www.dropbox.com/s/ha51ntucl3klkcb/cell_flash_960fps.mp4 https://www.dropbox.com/s/ha51ntucl3klkcb/cell_flash_960fps.... There are definitely a few frames where the LED is at an intermediate brightness, however teasing out the exact timings between the flash and the camera may prove to be difficult to correctly synchronize. As for over-saturated images having more signal... although the PSNR calculation may give you a better number, in practice, a region that is over-saturated is just a blob of 1s on the image (assuming float64 pixel values of 0-1) and there is no information there to extract. With a black level near but not at 0, we've found there is often more information hidden in the 'dark noise' than can be discerned by the human eye alone.
- 7y ago