4 ms·

I've spent a lot of time trying to get 3d models of rural roads with a truck-mounted go-pro and photogrammetry. At the end of the day, I couldn't get it to wor

I've spent a lot of time trying to get 3d models of rural roads with a truck-mounted go-pro and photogrammetry. At the end of the day, I couldn't get it to work despite having some great data sets. I talked to someone who did manage to get this work, proved down to centimeter accuracy but they were only able to get it to work with great difficulty and a rig with 5 different cameras. They said by the time you do all that, you'll have invested enough that lidar would make as much sense cost-wise. I believe the problem is at the MVS step, but it seems like none of the libraries handle this use case very well for some reason, despite having great overlapping features. When taken on the ground, doing orbits looking in toward a common object works reasonably well, and colmap handled this the best. Especially with exhaustive matching, but it takes DAYS to process. (Or, 4-6 hours if you REALLY optimize the process.) But moving in a line, no matter how much overlap of the features you get, just doesn't work that well, and I do not understand why. I think it fails in the feature matching step and it makes no sense since it has really similar images to try and match if you capture at any reasonable frequency. I think THIS worked better than I would have expected because of the 3d aspect where the lens gives you multiple perspectives on objects along the sides.

jakecopp 4y ago
> But moving in a line, no matter how much overlap of the features you get, just doesn't work that well, and I do not understand why. How far apart were the photos taken? I think OpenDroneMap needs lots of overlap with drone photos, and even more with spherical photos to extract features well. I haven't experimented with a non-360 degree camera - the benefit of a 360 degree camera is you effectively double the photos you get (front/back). Cycling or walking also ensures huge overlap.
- Enginerrrd 4y ago
  I can confidently say that sufficient overlap was not the issues. I experimented a lot, but I definitely had well over 99% overlap on some runs without really any improvement. I suspect it has to do with the training data used for feature matching.
- dylan604 4y ago
  When you said 360 camera, I had forgotten about the el cheapo versions with only 2 lenses. I'm used to 8-16 lens setups doing live action 360 VR video. It's been years now since I was involved in that, but the 16 camera system was the one from Google, and because of the extreme overlap, every object was visible in at least 4 lenses at any time. Because of this, it was some of the most compelling 3D available at the time. Funny, with 16 cameras each with their own microSD card that had to be data managed (puke), we felt it wasn't enough as it only produced a cylindrical output and not a full sphere. So, like the glutton for punishment fools we were, we mounted an additional 6 cameras to the rig to capture the zenith/nadir and would stitch/composite into the final. All of that to say, generating 3D photogrammetry images was something we never even considered as a purpose of the footage. We were soley focused on creating VR content. Never mind that some of us involved absolutely would acquire footage for the sole purpose of creating 3D models to be used in SFX for post work. I guess sometimes, you're just too close to something to see it?!?!
jeffreygoesto 4y ago
The reason is that the underlying matrices are rank deficient and the calculation gets numerically unstable to impossible, if you don't vary all kinds of degrees of freedom in the motion. Just changing one axis is not enough. If you look at 3D reconstruction image sequences from around 2000, you get motion sick from looking at them, because they made sure to always also vary rotation and translation in more than one direction. One video with some explanations I found: https://youtu.be/AQfRdr_gZ8g https://youtu.be/AQfRdr_gZ8g
- Enginerrrd 4y ago
  This sounds correct to me and aligns with a lot of other results from my experiments. I thank you for the insight. The only think I'll add is that it can't be an impossible task, because I can look at the same sequence of images and get a pretty good understanding of the 3D topography. That said, some things DON'T come through very well, like slopes so I think starting with priors that contain the gimbal position in addition to the GPS-derived location would really help pick up slopes, if that information were actually included in the pipeline. As advanced as it all is right now, computer vision still has a long way to go I think, with some very fruitful ideas yet to be implemented.
  - jeffreygoesto 4y ago
    I think that is the strength of the deep learning methods that have been thrown at the problem in the meantime. They learn to pick a reasonable choice out of the null space by looking at the scene. Only good if the assumptions encoded into the network actually hold though. Kind of applies to us deep human networks as well. That's why optical illusions are fun to see.

jakecopp 4y ago

> But moving in a line, no matter how much overlap of the features you get, just doesn't work that well, and I do not understand why. How far apart were the photos taken? I think OpenDroneMap needs lots of overlap with drone photos, and even more with spherical photos to extract features well. I haven't experimented with a non-360 degree camera - the benefit of a 360 degree camera is you effectively double the photos you get (front/back). Cycling or walking also ensures huge overlap.

Enginerrrd 4y ago
I can confidently say that sufficient overlap was not the issues. I experimented a lot, but I definitely had well over 99% overlap on some runs without really any improvement. I suspect it has to do with the training data used for feature matching.
dylan604 4y ago
When you said 360 camera, I had forgotten about the el cheapo versions with only 2 lenses. I'm used to 8-16 lens setups doing live action 360 VR video. It's been years now since I was involved in that, but the 16 camera system was the one from Google, and because of the extreme overlap, every object was visible in at least 4 lenses at any time. Because of this, it was some of the most compelling 3D available at the time. Funny, with 16 cameras each with their own microSD card that had to be data managed (puke), we felt it wasn't enough as it only produced a cylindrical output and not a full sphere. So, like the glutton for punishment fools we were, we mounted an additional 6 cameras to the rig to capture the zenith/nadir and would stitch/composite into the final. All of that to say, generating 3D photogrammetry images was something we never even considered as a purpose of the footage. We were soley focused on creating VR content. Never mind that some of us involved absolutely would acquire footage for the sole purpose of creating 3D models to be used in SFX for post work. I guess sometimes, you're just too close to something to see it?!?!

jeffreygoesto 4y ago

The reason is that the underlying matrices are rank deficient and the calculation gets numerically unstable to impossible, if you don't vary all kinds of degrees of freedom in the motion. Just changing one axis is not enough. If you look at 3D reconstruction image sequences from around 2000, you get motion sick from looking at them, because they made sure to always also vary rotation and translation in more than one direction. One video with some explanations I found: https://youtu.be/AQfRdr_gZ8g https://youtu.be/AQfRdr_gZ8g

Enginerrrd 4y ago
This sounds correct to me and aligns with a lot of other results from my experiments. I thank you for the insight. The only think I'll add is that it can't be an impossible task, because I can look at the same sequence of images and get a pretty good understanding of the 3D topography. That said, some things DON'T come through very well, like slopes so I think starting with priors that contain the gimbal position in addition to the GPS-derived location would really help pick up slopes, if that information were actually included in the pipeline. As advanced as it all is right now, computer vision still has a long way to go I think, with some very fruitful ideas yet to be implemented.
- jeffreygoesto 4y ago
  I think that is the strength of the deep learning methods that have been thrown at the problem in the meantime. They learn to pick a reasonable choice out of the null space by looking at the scene. Only good if the assumptions encoded into the network actually hold though. Kind of applies to us deep human networks as well. That's why optical illusions are fun to see.