9 ms·
Show HN: Wave-based non-line-of-sight computational imaging in Julia
- gugagore 7y ago"Honestly, the quality is probably on par with a 40 year old digital camera. On the other hand, it hopefully means that in about 40 years we will have GoPro’s with this capability strapped to our helmets showing us what’s around the corner before we even get there!" I would have guessed there are some resolution limits. For a traditional [line of sight?] camera, the resolution is limited by the imager and https://en.wikipedia.org/wiki/Angular_resolution https://en.wikipedia.org/wiki/Angular_resolution . What limits are at play for non-line-of-sight imaging?
- krrutkow 7y agoAt least surfaces that are anisotropic scattering, highly absorbing surfaces, and temporal resolution of the laser and shutter will all affect the image's spatial resolution.
- mohn 7y agoThe quantization of light is going to be a serious limit. The detector elements of a camera could be made pretty sensitive, but they can't detect fewer than one photon. In NLOS (and traditional photography) the scene to be imaged is illuminated with N photons but the detector only receives M photons worth of signal back, where M<<N. So future people are going to need: 1) helmet-mounted lasers capable of sustained very high power output, to increase the signal enough to get over the quantum detection threshold 2) to slow their roll, so a lot of imaging can occur before they round the corner or both. There are already practical limitations on laser power output in air because the air will turn into plasma along the beam path. Similarly, for NLOS you need to not burn up and destroy the surface that you're using to bounce the light around the corner.
- uasm 7y ago> "In NLOS (and traditional photography) the scene to be imaged is illuminated with N photons but the detector only receives M photons worth of signal back, where M<<N." How about "averaging" several attempts to compose a single shot? Would it be possible to shine lasers of multiple wavelengths to achieve better results? maybe varying angles as-well?
- gugagore 7y agoAveraging N images should (roughly?) make the noise go down by sqrt(N), so long as you don't introduce more sources of noise during collection. So the camera would have to be still, or the images registered to a common perspective.
- londons_explore 7y agoThe camera or subject can move, and as long as the movements are known then the reconstruction is still perfect. High end mobile phone cameras take hundreds of shots over a second or more to reduce noise, and rely on the fact they can estimate movement of things in the frame with optical flow and a gyroscope.
- gugagore 7y agoThis is what I meant by "registration", for the case when the subject isn't changing.
- etatoby 7y ago> The detector elements of a camera could be made pretty sensitive, but they can't detect fewer than one photon. Or can they? If the detector elements were part of a quantum computer (or a quantum computing "chip", whatever that will turn out to be) they would be able to analyze all the photon paths (Feynman paths) bouncing back from the subject, even those that would decohere / collapse away in a traditional detector. IANAP, but wouldn't a quantum chip be able to perform some amount of NLOS by analyzing the paths of even a single photon?
- darkmighty 7y agoAs soon as your photon interacts with a realistic scene (thermal), I believe it loses coherence with any apparatus you might have prepared in the detector. In short, no, I don't think this is possible, unless you consider idealized environments.
- krrutkow 7y agoDirect link to the Julia project repository: https://github.com/analytech-solutions/fkMigration.jl https://github.com/analytech-solutions/fkMigration.jl
- billconan 7y agosuper cool! why julia?
- phonebucket 7y agoI cannot answer for the author, but for projects which build a lot of expensive numerical routines from scratch, Julia is a good fit: The JIT overhead is a small price compared to the high-level feel of the code and the resulting post-compiled speed.
- elcritch 7y agoThis seems to be one of those “poster child” projects for Julia. As in there are likely lots of custom primitives or numerical routine that’d be impossible to have perform well in Python/Numpy but still saving development effort over coding a custom C++ case (much less C++ expertise needed to do so).
- krrutkow 7y agoFrom my experience it is one of the most productive programming languages to develop in, and the code can be iterated on to reach performance that is as good as C or Fortran code.
- StefanKarpinski 7y agoNote that @krrutkow is the author so this answer is fairly definitive.
- billconan 7y agowait, why can it see color? if this algorithm is based on the speed of light traveling (similar to ultrasonic), I would think it won't show color? the resulting image shows the printed 1,2,3 on that paper. is that because light reflects differently across the same paper due to different colors (black vs. white)?
- samstave 7y agoCan you apply these theories to using a wifi AP to read the 3d layout of the room/space based on RSSI and its various interferences? Could you make a 3D scanner app in conjunction with UBNT and see what could be done....
- krrutkow 7y agoI don't think the wavelengths involved would be the best choice for it. The question is how much reflection is exhibited at wifi frequencies with the surfaces in a room, probably not enough to be able to observe it.
- mbarronj 7y agoNot only wave length, but also beam width. The laser is spatially very compact. Radio waves need larger antenna apertures to be that focused. It's hard to get tight spatial resolution with a beam that looks like a balloon animal. Also, multipath and time-of-flight is difficult with radio waves, and the EM reflectance is weirder. Light is just much more convenient for this purpose. But - it's not outside of the realm of possible.
- jjoonathan 7y ago> Light is just much more convenient for this purpose With RADAR you get phase information, which helps a lot, and now that 60GHz wifi is a thing the balloon animal has a manageable size :)
- orbital-decay 7y ago>It's hard to get tight spatial resolution with a beam that looks like a balloon animal. Would compressed sensing help? It doesn't need the focus to be precise to reconstruct the scene from a single pixel detector.
- woeirua 7y agoNo, compressed sensing only allows you to attempt to reconstruct the original signal from a limited subset of the data sampled in a specific way, as long as you can assume some kind of sparsity on the data in another domain. Fundamentally, you cannot get something from compressed sensing that you wouldn't be able to otherwise normally.
- User23 7y agoThis reminds me of Feynman’s QED, which is a must read for laymen who want to know more about things like why shadows have fuzzy edges and other ways light doesn’t exactly travel in a straight line.
- whatshisface 7y agoYou don't need QED for that, only Maxwell's equations. QED is necessary if you want to explain how those effects jive with the fact that you can count individual photons.
- Judgmentality 7y agoIf I am wrong, please correct me: but I don't believe traditional electromagnetism can explain the fuzziness (which I think is better described as noisiness) of shadows.
- yorwba 7y agoThat's just diffraction. Any kind of wave spreads slightly into the shadow behind an obstacle; the waves described by Maxwell's equations are no exception.
- Groxx 7y agoI suspect "fuzziness" here is something specific? E.g. fuzzy-edged shadows from the sun seems pretty obvious - it's not a point light source. So this is referring to... nano-scale edge fuzziness or something? Or the shenanigans needed to do sub-wavelength features (like we do for silicon lithography)?
- thisrod 7y agoIt depends what you mean by "traditional" and "fuzziness". In ray optics, shadows from point sources are black and white. (If half of the sun is obscured, the surface it illuminates is half as bright, but that's pretty obvious.) That's very old school. Newton could get it right, even though he got particles and waves entirely wrong. In wave/electromagnetic optics, shadows are shades of grey. Just like water waves can be ripples at one end of a beach, crashing surf at the other, and vary continuously in between. In quantum optics, shadows are a superposition of different shades of grey, due to shot noise from the photons. That matters if you're doing very precise interference measurements, but not when you're taking photographs, no matter how short the exposure.
- gurumeditations 7y agoDoes anyone have a more ELI5 explanation?
- AceJohnny2 7y agoMy guess: echolocation, but with electromagnetic waves (light) rather than sound waves
- airstrike 7y agoMakes me wonder if the results of this test could be improved by layering in sound waves for additional resolution
- cjhanks 7y agoSoundwave SnR deteriorates much faster than lightwave. That said, there are companies uses sound for generating maps.
- nixpulvis 7y agoWhy not use other frequencies of the EM spectrum, some that say penetrate walls? I've heard of WiFi being used to 3D map buildings for example.
- adamnemecek 7y agoMy guess is that it doesn't create the echo.
- 3JPLW 7y agoThis site — which appears to be the source data for OP as well — has some really nice visualizations. It looks like it's more a scan than a flash, combined with extraordinarily fast frame rates to precisely measure the light's travel time and allow for reconstruction. http://www.computationalimaging.org/publications/nlos-fk/ http://www.computationalimaging.org/publications/nlos-fk/
- 7y ago
- SubiculumCode 7y agoI can't help but wonder to what extent biological systems could theoretically implement an analogous algorithm. Knowing a tiger is behind the tree would sure be advantageous.
- hirundo 7y agoFirst, evolve a high power laser.
- SubiculumCode 7y agohttps://www.nature.com/articles/nphoton.2011.99 https://www.nature.com/articles/nphoton.2011.99
- lopmotr 7y agoThere are passive NLOS imaging technologies based on shadows: https://arxiv.org/pdf/1807.02444.pdf https://arxiv.org/pdf/1807.02444.pdf https://www.quantamagazine.org/the-new-science-of-seeing-around-corners-20180830/ https://www.quantamagazine.org/the-new-science-of-seeing-aro... But animals can already "see" round corners using sound and smell. Dogs don't need any sort of line of sight to identify who's nearby and cats have night vision.
- amingilani 7y agoWait, what is the hardware required for capturing this kind of data? Can I do this with a point-and-shoot camera?
- snazz 7y agoThe picosecond shutter time scale part and high speed control of a powerful laser would make that awfully difficult.
- amingilani 7y agoI agree, sorry, I went back to a video I a few years ago on something similar and realized that the defining feature of their camera was Femto-photography.
- amingilani 7y agoThere was a similar technique discussed in a Ted Talk using femto-photography, which had an animation showing how it was done for an ELI5[0]. The talk contains references future technology to see around corners. This article may have been inspired by the talk, but it certainly makes it much more available for the rest of us. [0]: https://youtu.be/Y_9vd4HWlVA?t=354 https://youtu.be/Y_9vd4HWlVA?t=354
- eximius 7y agoIt is interesting that the reproduced image seems rotated from the color image. Is that change an artifact of the algorithm or being NLOS, or is it the angle of the perspective (for lack of a better word since I'm not really sure how you would define your viewpoint) / a difference in where the color image was taken from the special sensor (i.e., you could place the normal camera in front of a mirror to have a similar perspective from the special sensor)...
- DrJosiah 7y agoThey used some sort of wall to reflect off of. I'm sure that when they moved / removed the wall to take the regular picture, it wasn't in the same alignment.
- GistNoesis 7y agoThe captured data is a 3D voxel grid (not a 2.5D image like kinect). In theory it can view behind the objects themselves. The reproduced image is an orthogonal projection of this 3D grid of pixel. Any projection could have been chosen. For example the camera projection could have been chosen to obtain exactly the same image.
- eximius 7y agoDo they have a lower effective resolution at different projections due to light reflected being somehow less direct (more reflections?)?
- GistNoesis 7y agoYes. The best views will be from projections whose camera position is at a point scanned by the laser. With the f-k migration algorithm, the output is in fact the aligned sum of 2.5D images that would have produced by cameras located at the laser scanned locations. The algorithm is based on a simple model which doesn't take into account obstructions or secondary ray reflections. In theory for slower algorithms in O(N^5) like inverting the rendering engine you can take obstructions, and ray bouncing into account, but the effective resolution will depend on the dynamic range of your measurements, that you can improve a little by increasing exposure time. In practice reflections don't carry enough information. You can also incorporate phase information (like with holograms) instead of just intensity, if you are able to measure it to get sharper image thanks to coherent imaging. Once you use the phase information, you can improve the spatial resolution, using multiple wavelength.
- sbhn 7y agoPerhaps this can be done with multi frequency audio pulses. That would be easier to build than a NLOS camera
- woeirua 7y agoIt definitely can be done. A process similar to this is used for ultrasonic imaging.
- MrSpiffy 7y agoThis is quite analogous to exploration seismic imaging, including the Stolt migration. Thank you for sharing this.
- woeirua 7y agoYeah, it's strange to me that the people working in this subfield are proposing more exotic approaches to basic time migration which is ideal for this problem and has been known for more than 50 years now.
- deleted 7y ago[deleted]
- lokimedes 7y agoFascinating. I have been using the same approach to generate synthetic aperture radar images, but having a non-radar platform to test the processing algorithms with is quite useful.
- nurettin 7y agoIt appears we are escaping plato's cave allegory. What would be really interesting to reconstruct what is behind walls using run-of-the-mill hardware.
- xyzal 7y agoI wonder wether there are already any military applications based on wave-based imaging. I assume one would gain significant advantage having such capability, especially in urban warfare.