3 ms·

I've thought about this a bit, but how can you achieve 360° stereo? If you have two 360° cameras side by side, as soon as you look 90° L/R, you no longer have d

I've thought about this a bit, but how can you achieve 360° stereo? If you have two 360° cameras side by side, as soon as you look 90° L/R, you no longer have depth and one eye would just see the side of the other camera. If you just have one 360° camera, you don't have any depth.

deleted 12y ago
[deleted]
- raimondious 12y ago
  No because I'm turning my head. The reason to use 360° cameras is that they capture the full range of the image in a single frame. This allows things like taking one stream of 360° video, and allow you to look around while watching the video, without requiring a mechanical panorama to be taken for each frame. However this is a problem with stereography since you need a particular distance between the two views to achieve depth — the cameras have to be positioned similarly to the eyes in your head to achieve realistic depth. In software this is trivial, but on a robot, it's more difficult.
mattnewport 12y ago
Current approaches are a bit of a hack. You have multiple stereo camera pairs and to handle intermediate directions you do some software interpolation between them. Despite being a hack it works well enough to be reasonably immersive. The Samsung Gear VR comes with a collection of 360 degree panoramic photos and some 3D videos using the same approach. John Carmack talks about it a bit in his Oculus Connect keynote. This approach is obviously wrong if you tilt your head but people don't generally do that enough to notice.
electrograv 12y ago
You could build a 360 degree 3D camera that works for any viewing configuration, but it would involve nontrivial image processing and wouldn't be totally free from small visual glitches/artifacts in some cases. 1. Build an array of a few (let's say 8 or so) cameras pointing outward from the center. 2. Use a stereo matching algorithm to extract a depth map from the perspective of each camera. Keeping track of the position and orientation of each camera in 3D space, these depths become a point cloud associated with each camera. 3. Determine the 3D location and orientation of each "eye" you want to render, then render all point clouds in 3D space to reconstruct a "reprojected" version of the scene from any desired viewpoint. Of course, the farther the eyes deviate from actual camera locations the more stretched/warped the image will appear, but that won't matter much as long as you keep the eye coordinates within the physical space occupied by the camera. Honestly I'd be kind of disappointed if CMU doesn't actually try this. It's disappointing to think that perhaps all this buzz about "hackathons" (as the article mentions) is encouraging -- even at major research universities -- quickly slapping together components to make something kind of work, as opposed to fundamental algorithm development and proper engineering solutions.
- btown 12y ago
  If you pair it with a laser scanner (which won't have any atmospheric interference to deal with on the moon!) you can get a near-perfect depth map without needing to do stereo correspondence (whose accuracy can be thrown off if there are identical features in nearby points - insert joke about all moon rocks looking the same). You could even throw a still DSLR with a telephoto lens on a perfectly calibrated, positionable mount, aim it at rocks in sequence, and get high-resolution textures. For bonus accuracy, have it so that the DSLR mount and the laser-scanner mount can swap positions, so you're getting the textures from the exact same position as the scanner. Send the depth map and textures back to Earth in non-real-time. Program the robot to autonomously explore regions where the depth map hasn't covered. Then on the client side, just feed those textures into a gaming engine or render farm. Bam. That would be a scalable Moon Explorer.
- acgourley 12y ago
  That's what I was hoping to see as well. Doing it on the moon is oddly enough probably easier than many earth locations due to the lack of complex foreground objects.

deleted 12y ago

[deleted]

raimondious 12y ago
No because I'm turning my head. The reason to use 360° cameras is that they capture the full range of the image in a single frame. This allows things like taking one stream of 360° video, and allow you to look around while watching the video, without requiring a mechanical panorama to be taken for each frame. However this is a problem with stereography since you need a particular distance between the two views to achieve depth — the cameras have to be positioned similarly to the eyes in your head to achieve realistic depth. In software this is trivial, but on a robot, it's more difficult.

mattnewport 12y ago

Current approaches are a bit of a hack. You have multiple stereo camera pairs and to handle intermediate directions you do some software interpolation between them. Despite being a hack it works well enough to be reasonably immersive. The Samsung Gear VR comes with a collection of 360 degree panoramic photos and some 3D videos using the same approach. John Carmack talks about it a bit in his Oculus Connect keynote. This approach is obviously wrong if you tilt your head but people don't generally do that enough to notice.

electrograv 12y ago

You could build a 360 degree 3D camera that works for any viewing configuration, but it would involve nontrivial image processing and wouldn't be totally free from small visual glitches/artifacts in some cases. 1. Build an array of a few (let's say 8 or so) cameras pointing outward from the center. 2. Use a stereo matching algorithm to extract a depth map from the perspective of each camera. Keeping track of the position and orientation of each camera in 3D space, these depths become a point cloud associated with each camera. 3. Determine the 3D location and orientation of each "eye" you want to render, then render all point clouds in 3D space to reconstruct a "reprojected" version of the scene from any desired viewpoint. Of course, the farther the eyes deviate from actual camera locations the more stretched/warped the image will appear, but that won't matter much as long as you keep the eye coordinates within the physical space occupied by the camera. Honestly I'd be kind of disappointed if CMU doesn't actually try this. It's disappointing to think that perhaps all this buzz about "hackathons" (as the article mentions) is encouraging -- even at major research universities -- quickly slapping together components to make something kind of work, as opposed to fundamental algorithm development and proper engineering solutions.

btown 12y ago
If you pair it with a laser scanner (which won't have any atmospheric interference to deal with on the moon!) you can get a near-perfect depth map without needing to do stereo correspondence (whose accuracy can be thrown off if there are identical features in nearby points - insert joke about all moon rocks looking the same). You could even throw a still DSLR with a telephoto lens on a perfectly calibrated, positionable mount, aim it at rocks in sequence, and get high-resolution textures. For bonus accuracy, have it so that the DSLR mount and the laser-scanner mount can swap positions, so you're getting the textures from the exact same position as the scanner. Send the depth map and textures back to Earth in non-real-time. Program the robot to autonomously explore regions where the depth map hasn't covered. Then on the client side, just feed those textures into a gaming engine or render farm. Bam. That would be a scalable Moon Explorer.
acgourley 12y ago
That's what I was hoping to see as well. Doing it on the moon is oddly enough probably easier than many earth locations due to the lack of complex foreground objects.