4 ms·

Honestly this doesn't look any better than what we were doing back in 2016-2017. I'm not sure what's novel here. This is the only video I could find, but we we

Honestly this doesn't look any better than what we were doing back in 2016-2017. I'm not sure what's novel here. This is the only video I could find, but we were doing monocular reconstruction from a limited number of RGB (not depth) images AND doing voxel segmentation on the processing side. https://www.youtube.com/watch?v=nqy44VSWh3g https://www.youtube.com/watch?v=nqy44VSWh3g Even as far back as 2010 people were doing reasonable monocular reconstruction including software like meshroom etc...the whole of TU Munich also under Matthias Niessner has been doing this for a while. What's novel here?

tintor 5y ago
Fast enough to be used for mobile robots?
closetnerd 5y ago
Says it’s real-time
- AndrewKemendo 5y ago
  2016 from Matthias Niessner's group https://www.youtube.com/watch?v=keIirXrRb1k https://www.youtube.com/watch?v=keIirXrRb1k http://graphics.stanford.edu/projects/bundlefusion/ http://graphics.stanford.edu/projects/bundlefusion/
  - jonas21 5y ago
    That requires depth input
    - AndrewKemendo 5y ago
      Good point, I don't recall offhand the paper that was the mono-RT one. At a minimum though 6D.ai and a few others had companies that were selling this as a service at least as far back as 2017.
      - nobbis 5y ago
        6D.ai wasn’t released until 2018. You may be thinking of Abound, which you wanted to license in March 2017. I don’t know of anyone else that had real-time meshing in 2017.
- fxtentacle 5y ago
  I always found ORB-SLAM2 pretty impressive, which can map 3D neighborhoods in realtime while you drive around in a car: https://www.youtube.com/watch?v=ufvPS5wJAx0 https://www.youtube.com/watch?v=ufvPS5wJAx0 https://www.youtube.com/watch?v=3BrXWH6zRHg https://www.youtube.com/watch?v=3BrXWH6zRHg
  - nobbis 5y ago
    ORB-SLAM2 is SLAM only (similar to ARKit w/o VIO) so only reconstructs feature points - no surfaces. This research is about surface reconstruction.
pj_mukh 5y ago
Looks like a much better response to white walls/texture less surfaces.
nobbis 5y ago
Their research doesn't just integrate depth maps into a TSDF - it uses NN's to incorporate surface priors. I don't recall you having similar real-time meshing functionality in 2016-2017, Andrew. Can you show what you had? As far as I'm aware, Abound was the first to demo real-time monocular mobile meshing: on Android in early 2017 (e.g. https://www.youtube.com/watch?v=K9CpT-sy7HE https://www.youtube.com/watch?v=K9CpT-sy7HE), and iOS in early 2018 (e.g. https://twitter.com/nobbis/status/972298968574013440 https://twitter.com/nobbis/status/972298968574013440).
fxtentacle 5y ago
This is a paper about a new way of storing/merging 3D data. The actual 3D reconstruction is so-so, I agree. And they kinda cheat by using ARKit (which uses LIDAR internally) to get good camera poses even if there is little texture. So the novel part here is that they can immediately merge all the images into a coherent representation of the 3D space, as opposed to first doing bundle adjustment, then doing pairwise depth matching, then doing streak-based depth matching, and then merging the resulting point clouds. Also, they can use learned 3D shape priors to improve their results. Basically that means "if there is no visible gap, assume the surface is flat". But AFAIK, that's not new. EDIT: My main criticism of this paper after looking at the source code a bit would be that due to the TSDF, which is like a 3D voxel grid, they need insane amounts of GPU memory, or else the scenes either need to be very small or low resolution. That is most likely also the reason why they reconstruction looks so cartoon-like and is smooth on all corners: They lack the memory to store more high-frequency details. EDIT2: Mainly, it looks like they managed to reduce the GPU memory consumption of Atlas [1] which is why they can reconstruct larger areas and/or higher resolution. But it's still far less detail than Colmap [2]. [1] https://github.com/magicleap/Atlas https://github.com/magicleap/Atlas [2] https://colmap.github.io/ https://colmap.github.io/
- nobbis 5y ago
  ARKit doesn’t use LiDAR for camera pose tracking. TSDF memory isn’t an issue since Niessner et al. (2013).
  - onedognight 5y ago
    > ARKit doesn’t use LiDAR for camera pose tracking. It doesn't by default, for power reasons, but it will in a pinch.
    - nobbis 5y ago
      Interesting - I’ve never observed it kicking in. Would love to know in which circumstances it’s used. I assume you work for Apple to know this so understand if you can’t share more.
jayd16 5y ago
Its not a fair fight but your example is much much worse. The mesh reconstruction of the chairs are not suitable for occlusion or physics in your example.
dimatura 5y ago
I haven't read this paper in detail. At a glance, it seems interesting, but not necessarily a huge breakthrough compared to SOTA in monocular 3D reconstruction (which is not a criticism; most papers are incremental and that's fine). That said, what I find with a lot of work on neural techniques applied to 3D reconstruction/VSLAM (a pretty active area in the past few years) is that they're often not significantly more accurate than prior, non-ML techniques, and in the beginnings, often less accurate. On the other hand, they're often more robust. Classical geometry- and optimization- based approaches are fairly brittle, and tend to fail catastrophically or converge to totally implausible solutions in tricky conditions. Learning-based approaches tend to be better at degrading gracefully.

tintor 5y ago

Fast enough to be used for mobile robots?

closetnerd 5y ago

Says it’s real-time

AndrewKemendo 5y ago
2016 from Matthias Niessner's group https://www.youtube.com/watch?v=keIirXrRb1k https://www.youtube.com/watch?v=keIirXrRb1k http://graphics.stanford.edu/projects/bundlefusion/ http://graphics.stanford.edu/projects/bundlefusion/
- jonas21 5y ago
  That requires depth input
  - AndrewKemendo 5y ago
    Good point, I don't recall offhand the paper that was the mono-RT one. At a minimum though 6D.ai and a few others had companies that were selling this as a service at least as far back as 2017.
    - nobbis 5y ago
      6D.ai wasn’t released until 2018. You may be thinking of Abound, which you wanted to license in March 2017. I don’t know of anyone else that had real-time meshing in 2017.
fxtentacle 5y ago
I always found ORB-SLAM2 pretty impressive, which can map 3D neighborhoods in realtime while you drive around in a car: https://www.youtube.com/watch?v=ufvPS5wJAx0 https://www.youtube.com/watch?v=ufvPS5wJAx0 https://www.youtube.com/watch?v=3BrXWH6zRHg https://www.youtube.com/watch?v=3BrXWH6zRHg
- nobbis 5y ago
  ORB-SLAM2 is SLAM only (similar to ARKit w/o VIO) so only reconstructs feature points - no surfaces. This research is about surface reconstruction.

pj_mukh 5y ago

Looks like a much better response to white walls/texture less surfaces.

nobbis 5y ago

Their research doesn't just integrate depth maps into a TSDF - it uses NN's to incorporate surface priors. I don't recall you having similar real-time meshing functionality in 2016-2017, Andrew. Can you show what you had? As far as I'm aware, Abound was the first to demo real-time monocular mobile meshing: on Android in early 2017 (e.g. https://www.youtube.com/watch?v=K9CpT-sy7HE https://www.youtube.com/watch?v=K9CpT-sy7HE), and iOS in early 2018 (e.g. https://twitter.com/nobbis/status/972298968574013440 https://twitter.com/nobbis/status/972298968574013440).

fxtentacle 5y ago

This is a paper about a new way of storing/merging 3D data. The actual 3D reconstruction is so-so, I agree. And they kinda cheat by using ARKit (which uses LIDAR internally) to get good camera poses even if there is little texture. So the novel part here is that they can immediately merge all the images into a coherent representation of the 3D space, as opposed to first doing bundle adjustment, then doing pairwise depth matching, then doing streak-based depth matching, and then merging the resulting point clouds. Also, they can use learned 3D shape priors to improve their results. Basically that means "if there is no visible gap, assume the surface is flat". But AFAIK, that's not new. EDIT: My main criticism of this paper after looking at the source code a bit would be that due to the TSDF, which is like a 3D voxel grid, they need insane amounts of GPU memory, or else the scenes either need to be very small or low resolution. That is most likely also the reason why they reconstruction looks so cartoon-like and is smooth on all corners: They lack the memory to store more high-frequency details. EDIT2: Mainly, it looks like they managed to reduce the GPU memory consumption of Atlas [1] which is why they can reconstruct larger areas and/or higher resolution. But it's still far less detail than Colmap [2]. [1] https://github.com/magicleap/Atlas https://github.com/magicleap/Atlas [2] https://colmap.github.io/ https://colmap.github.io/

nobbis 5y ago
ARKit doesn’t use LiDAR for camera pose tracking. TSDF memory isn’t an issue since Niessner et al. (2013).
- onedognight 5y ago
  > ARKit doesn’t use LiDAR for camera pose tracking. It doesn't by default, for power reasons, but it will in a pinch.
  - nobbis 5y ago
    Interesting - I’ve never observed it kicking in. Would love to know in which circumstances it’s used. I assume you work for Apple to know this so understand if you can’t share more.

jayd16 5y ago

Its not a fair fight but your example is much much worse. The mesh reconstruction of the chairs are not suitable for occlusion or physics in your example.

dimatura 5y ago

I haven't read this paper in detail. At a glance, it seems interesting, but not necessarily a huge breakthrough compared to SOTA in monocular 3D reconstruction (which is not a criticism; most papers are incremental and that's fine). That said, what I find with a lot of work on neural techniques applied to 3D reconstruction/VSLAM (a pretty active area in the past few years) is that they're often not significantly more accurate than prior, non-ML techniques, and in the beginnings, often less accurate. On the other hand, they're often more robust. Classical geometry- and optimization- based approaches are fairly brittle, and tend to fail catastrophically or converge to totally implausible solutions in tricky conditions. Learning-based approaches tend to be better at degrading gracefully.