Disclaimer: I have a degree in robotics and worked for what is now Lyft Level 5, dealing with autonomous driving. What it does is basically a big optimization

Disclaimer: I have a degree in robotics and worked for what is now Lyft Level 5, dealing with autonomous driving. What it does is basically a big optimization problem. You have 6 parameters for each picture (x, y, z, roll, pitch, yaw, although in practice there are better parametrizations based on what is called Quaternions). Than three prameters for the position of each keypoint. Those are simply the same physical spots on multiple image that are matched based on their visual appearance. Last component is a metric that you try to optimize. That is mostly just reprojection error. Given all your current estimates where yould the keypoints be projected on this synthetic image. Then you compare it with the pixel locations of where it actually in and try to minimize this. It is actually a very versatile and robust pipeline which gives you what you’ve seen on the images. Last step is to produce the dense reconstruction which is commonly done using patch match algorithm. You can try it for yourself with the OpenMVG library. Very hackable and versatile.