3 ms·

Is there a good resource I can read about the last point in the article? > We don't know where the camera will be. For example, we ask players to use the camer

Is there a good resource I can read about the last point in the article? > We don't know where the camera will be. For example, we ask players to use the cameras of their mobile devices to explore the size and proportions of a room — to work with the augmented reality helmet. We don't know anything about the rooms, and we can't ask users to use a pattern. Are there automatic techniques to tackle this problem?

midjji 2y ago
Yeah, you can do live simultaneous calibrations and reconstructions and localizations. Its a difficult problem, and a slightly different one for each phone, but there are plenty of variants which work for single phone models. The imu is key to making it possible though. What makes it harder is that older phones have insufficient processing power, whereas newer phones have too many cameras. The phones also do post processing in undocumented ways that are impossible to disable. There is also rolling shutter. This in turn can be simplified down again by just subsampling the images to 480p and estimating camera parameters per image.
midjji 2y ago
I do not think there is an off the shelf solution aside from e.g. arkit. In general a good starting point for such a problem would be to implement https://grail.cs.washington.edu/rome/ https://grail.cs.washington.edu/rome/ on your own. Its not particularily hard, and there is no difference between reconstructing from uncalibrated cameras and reconstructing from many images from one camera with unknown calibration. Then you can check if the calibration is unchanged over time, and if it is, add that as a constraint. So far it will still take half an hour to run, but now you can add in tracking exploiting motion prediction and imu, which in turn will give you the real time speed.
ryandamm 2y ago
This is just describing SLAM (https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping); https://en.wikipedia.org/wiki/Simultaneous_localization_and_... it's a deep field and there are many techniques and tools. Extrinsic calibration (where is my camera in 3D space, and how is it oriented?) being different from intrinsic calibration (how do the pixels in my image map to ray directions relative to the front nodal point of the lens?).

midjji 2y ago

Yeah, you can do live simultaneous calibrations and reconstructions and localizations. Its a difficult problem, and a slightly different one for each phone, but there are plenty of variants which work for single phone models. The imu is key to making it possible though. What makes it harder is that older phones have insufficient processing power, whereas newer phones have too many cameras. The phones also do post processing in undocumented ways that are impossible to disable. There is also rolling shutter. This in turn can be simplified down again by just subsampling the images to 480p and estimating camera parameters per image.

midjji 2y ago

I do not think there is an off the shelf solution aside from e.g. arkit. In general a good starting point for such a problem would be to implement https://grail.cs.washington.edu/rome/ https://grail.cs.washington.edu/rome/ on your own. Its not particularily hard, and there is no difference between reconstructing from uncalibrated cameras and reconstructing from many images from one camera with unknown calibration. Then you can check if the calibration is unchanged over time, and if it is, add that as a constraint. So far it will still take half an hour to run, but now you can add in tracking exploiting motion prediction and imu, which in turn will give you the real time speed.

ryandamm 2y ago

This is just describing SLAM (https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping); https://en.wikipedia.org/wiki/Simultaneous_localization_and_... it's a deep field and there are many techniques and tools. Extrinsic calibration (where is my camera in 3D space, and how is it oriented?) being different from intrinsic calibration (how do the pixels in my image map to ray directions relative to the front nodal point of the lens?).