3 ms·
This is great! Computational photography is a really neat field. I've been slowly learning computer vision for a different project, but I must have a lot in com
by TaylorAlexander 6y ago
This is great! Computational photography is a really neat field. I've been slowly learning computer vision for a different project, but I must have a lot in common with the author of this post. Many late nights doing video conversion with the nvidia hardware decoders, photo registration with COLMAP, and fiddling with cameras.
I recently got a spherical camera and I am trying to use it for photogrammetry. I also have an array of four 4k cameras with hardware synchronized shutters hooked up to an NVIDIA Jetson Xavier[1]. That system can record four 4k streams at once to the SSD. I wonder how many 4k streams a Jetson Nano could record, because then you could use 16 of these [2] and four to eight Jetson Nanos to make a camera system with all hardware synchronized shutters that could easily record the data and export it via the network. It would cost around $2500 though. These projects get expensive and I keep thinking I want a sponsor, but the slow pace using what hardware I can buy is probably fine for now.
I'm trying to do a complete photorealistic photogrammetry capture of hiking trails, so I can run my robot in simulation on virtual hiking trails and train real computer vision networks. Lately I've been wondering if there is a GAN in my future...
The frustrating thing about my project is the sheer amount of computation required. I really don't need a direct photogrammetry capture of a trail, an approximation would be fine to some degree. But I take like ten gigabytes of video data and then process each frame to find keypoints, run correlation on all these points, and all this (using COLMAP). This stuff can take days to process on my desktop.
Meanwhile there are neural networks that can compute depth from video in real time, and I wonder what it would take to stitch sequential depth estimations in to one 3D model with RGB textures in one continuous calculation. There's so much research to do!
By the way I found the work in this paper pretty fascinating. [3] Facebook is working on 6dof video recording and playback, which is quite the challenge on many levels!
[1] https://reboot.love/t/new-cameras-on-rover/ https://reboot.love/t/new-cameras-on-rover/
[2] https://www.e-consystems.com/4k-usb-camera.asp https://www.e-consystems.com/4k-usb-camera.asp
[3] https://research.fb.com/wp-content/uploads/2019/09/An-Integrated-6DoF-Video-Camera-and-System-Design.pdf https://research.fb.com/wp-content/uploads/2019/09/An-Integr...
- fxtentacle 6y agoSounds amazing :) Which cameras are you using? How's the encoding quality of the Jetson? Is lossless possible?
- TaylorAlexander 6y agoI am using these cameras: https://www.e-consystems.com/nvidia-cameras/jetson-agx-xavier-cameras/four-synchronized-4k-cameras.asp https://www.e-consystems.com/nvidia-cameras/jetson-agx-xavie... The encoding quality is good, but I have not tried lossless. I am encoding at 20,000 kb/s, which I am realizing is not super high. I may bump them all to 50,000 kb/s and see if it can still record. I am writing to an SSD over ESATA so I assume it can sustain four x 50mbps recording streams. I don't have much concern about the hardware encoder as those are pretty powerful. I don't have an easy link to a raw image sample but I do have a stitched image from two cameras. This picture is extracted from Rover's two front video streams: https://reboot.love/uploads/default/original/1X/2e90f9e9e308f715838e4734bfd6740eccda9e8e.jpeg https://reboot.love/uploads/default/original/1X/2e90f9e9e308...
- fxtentacle 6y agoThat is a very usable quality. Thank you for sharing the details of your setup :) I've been researching hardware-synchronized cameras for quite a while now, but with limited success. And now you show me a pre-assembled affordable solution :)
- pretty_dumm_guy 6y ago> The frustrating thing about my project is the sheer amount of computation required. I really don't need a direct photogrammetry capture of a trail, an approximation would be fine to some degree. But I take like ten gigabytes of video data and then process each frame to find keypoints, run correlation on all these points, and all this (using COLMAP). This stuff can take days to process on my desktop. If the hiking trails are accessible enough, you should have a look at SLAM technique. SLAM allows you to create smooth and rough approximate map of the environment through which you navigate your camera. Colorization of this map could be done by a GAN(might be an interesting side project). I am adding some pointers below : 1. https://www.doc.ic.ac.uk/~ajd/ https://www.doc.ic.ac.uk/~ajd/ - Prof. Davison and his group's work is impressive in this area. 2. https://vision.in.tum.de/research/vslam https://vision.in.tum.de/research/vslam - Prof. Cremers group have some SOTA algorithms in this area. P.S: You don't need a heavy setup for this. A single or a stereo camera should do the job.
- TaylorAlexander 6y agoThank you very much! It has been a long time since I've looked at visual slam actually. That Omnidirectional LSD-SLAM looks really nice. Their code repo has been untouched for six years, but this still makes me realize I need to use vslam! I just found this recent work which seems really useful. https://github.com/ivalab/gf_orb_slam2 https://github.com/ivalab/gf_orb_slam2 I feel like vslam could be the the first step in a post-processing pipeline that would reduce a lot of the computational complexity of solving large maps. Once I can easily make large maps I can build simulated environments and use those for training an agent. Thanks again for the tips!