12 ms·

Real-Time Coherent 3D Reconstruction from Monocular Video

178 points by samber 5y ago

alhirzel 5y ago
Does anyone know what the state of the art is for doing this type of reconstruction as a streaming input to detection and recognition algorithms? For instance, this could be used for object detection and identification on a recycling conveyor line.
- beambot 5y ago
  I don't believe that either does reconstruction... but for the recycling application, there are a handful of companies tackling this problem -- e.g. Everest Labs & Amp Robotics.
  - Animats 5y ago
    The big name in this is Bulk Handling Systems.[1] They build and sell large complete recycling plants. They do about 90% of the sorting with a series of mechanical processes. The robots do only the "quality control" step, pulling out items that got past the choppers, shakers, screens, air blasts, and magnets. That gets the outputs up to 99%+ of the desired product, so it can go into recycling processes that take in cardboard, plastics sorted by type, etc. It makes the robotics cost effective. San Francisco's recycling operation, at Pier 95, uses a system built by them. It's not a glamorous technology, but it gets the job done. [1] https://www.max-ai.com/video-max-ai-autonomous-qc/ https://www.max-ai.com/video-max-ai-autonomous-qc/
billconan 5y ago
can ARkit return accurate camera position?
- upbeat_general 5y ago
  I haven’t looked at any metrics but based on using ARKit applications (and various VIO SLAM implementations) it can but it depends highly on the scene, camera motion, and whether there is LIDAR/Stereo Depth.
adampk 5y ago
I am surprised that the team didn't choose to add the "Fusion" append at the end. This seems to fit into the genealogy of KinectFusion, ElasticFusion, BundleFusion, etc. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... https://www.imperial.ac.uk/dyson-robotics-lab/downloads/elastic-fusion/ https://www.imperial.ac.uk/dyson-robotics-lab/downloads/elas... https://graphics.stanford.edu/projects/bundlefusion/ https://graphics.stanford.edu/projects/bundlefusion/ Very impressive work. I have not seen any use cases for online 3D reconstruction unfortunately. 6D.ai made terrific progress in this tech but also could not find great use cases for online reconstruction and ended up having to sell to Niantic. Seems like what people want, if they want 3D reconstruction, is extremely high-fidelity scans (a la Matterport) and are willing to wait for the model. Unfortunately TSDF approach create a "slimey" end look which isn't usually what people are after if they want an accurate 3D reconstruction. It SEEMS like online 3D reconstruction would be helpful, but I have yet to see a use case for "online"...
- deleted 5y ago
  [deleted]
- tintor 5y ago
  Use case: Mobile robotics, lidar replacement in self-driving vehicles
- tonyarkles 5y ago
  I'm very curious to see how well this would work for online terrain reconstruction. I've got a drone with a pretty powerful onboard computer and it's always nice to be able to solve and tune problems with software instead of additional (e.g. LIDAR) hardware.
- eternityforest 5y ago
  Cave exploration. Put the helmet on, never get lost. Also youtube videos and such with more complex animated characters jumping around and things. If they could convert that to a floorplan diagram it would be wonderful for UI design, lots of use cases for a map.
AndrewKemendo 5y ago
Honestly this doesn't look any better than what we were doing back in 2016-2017. I'm not sure what's novel here. This is the only video I could find, but we were doing monocular reconstruction from a limited number of RGB (not depth) images AND doing voxel segmentation on the processing side. https://www.youtube.com/watch?v=nqy44VSWh3g https://www.youtube.com/watch?v=nqy44VSWh3g Even as far back as 2010 people were doing reasonable monocular reconstruction including software like meshroom etc...the whole of TU Munich also under Matthias Niessner has been doing this for a while. What's novel here?
- tintor 5y ago
  Fast enough to be used for mobile robots?
- closetnerd 5y ago
  Says it’s real-time
  - AndrewKemendo 5y ago
    2016 from Matthias Niessner's group https://www.youtube.com/watch?v=keIirXrRb1k https://www.youtube.com/watch?v=keIirXrRb1k http://graphics.stanford.edu/projects/bundlefusion/ http://graphics.stanford.edu/projects/bundlefusion/
    - jonas21 5y ago
      That requires depth input
      - AndrewKemendo 5y ago
        Good point, I don't recall offhand the paper that was the mono-RT one. At a minimum though 6D.ai and a few others had companies that were selling this as a service at least as far back as 2017.
        5y ago
polishdude20 5y ago
Shame there's no Android or iPhone app available
spyder 5y ago
It's not exactly the same but Neural Radiance Fields are getting more impressive: First one was this but it was slow: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf Then it's got faster: https://www.youtube.com/watch?v=fvXOjV7EHbk https://www.youtube.com/watch?v=fvXOjV7EHbk Lot of interesting papers: https://github.com/yenchenlin/awesome-NeRF https://github.com/yenchenlin/awesome-NeRF
leobg 5y ago
So much for the folks who think Tesla is on a fool’s errand when they’re using cameras instead of LIDAR.
- kajecounterhack 5y ago
  Companies like Waymo and Cruise use this kind of technology too. Unfortunately there are tons of corner cases of weird things you haven't seen before -- for example, some special vehicles self-occlude and you never get enough coverage to observe them correctly until you're too close. In general, radars and lidars used in _conjunction_ with cameras can handle occluded objects much better. Also, to measure the performance / evaluate observations generated from this tech, you would want to compare it to a pretty sizable 3D ground truth set which Tesla does not currently have. There are pretty big advantages to starting with a maximal set of sensors even if (eventually) breakthroughs turn them into unnecessary crutches.
  - leobg 5y ago
    That was very insightful. Do you work in that space? It is comments like yours that make HN a special place.
  - quantumduck 5y ago
    I agree that there are pretty big advantages to start with maximal set of sensors, but that is true only when you are a company with the sole goal of trying to make autonomy stack with shit ton of money and a product that you can modify because you own it. That's not true in the case of Tesla. They started as a EV company that offered autonomy later and since they were selling the product, they had to decide a minimal set of sensors that would, in theory, still work to keep the product cost reasonable. Tesla does have a sizeable 3D ground truth; they collected tons of data mounting LiDAR on their test vehicles.
    - kajecounterhack 5y ago
      > Tesla does have a sizeable 3D ground truth; they collected tons of data mounting LiDAR on their test vehicles. "Sizable" for evaluating safety has to be big enough to give small confidence intervals on your error -- and with self-driving, you need to cover a robust set of rare scenarios upsampled from general driving as well. I really doubt Tesla has what they need to convince themselves of higher levels of safety.
stefan_ 5y ago
Looked cool, then I read that there is some Apple ARkit magic blackbox in the middle of it all.
- fxtentacle 5y ago
  Yeah, I also consider that odd to use LIDAR-based poses and then call it "monocular".
  - upbeat_general 5y ago
    Agreed, I’m interested to see how sensitive it is to camera pose error, and especially long-term drift.
- cmelbye 5y ago
  I don't think that's true. The paper says that a camera pose estimated by a SLAM system is required. ARKit implements SLAM and can easily provide camera pose for each frame through the ARFrame class. But there are countless other implementations of SLAM, including Android ARCore, Oculus Quest, Roomba, self-driving cars, and a number of GitHub repos (https://github.com/tzutalin/awesome-visual-slam https://github.com/tzutalin/awesome-visual-slam).
Animats 5y ago
That's impressive, in that it handles white walls well. Most of the algorithms for that sort of thing have trouble registering big uniform surfaces. That's why many tracking algorithms want objects with lots of texture detail. Some add texture by projecting a noise pattern on the surface being tracked. That's sometimes called "unstructured light", the opposite being projecting a structured pattern such as a grid. The first generation Kinect did that. Of course, this network may do well on this because it's trained on indoor scenes with walls of uniform height and width. Uniform implies flat is likely to emerge as an assumption. You get to see the wall/floor joints and the wall/wall joints, so you have some references. Remember the Tesla that hit the big white semitrailer because the algorithm couldn't measure depth to a uniform surface? This is a hard problem in unstructured situations.

178 points by samber 5y ago

Does anyone know what the state of the art is for doing this type of reconstruction as a streaming input to detection and recognition algorithms? For instance, this could be used for object detection and identification on a recycling conveyor line.

beambot 5y ago
I don't believe that either does reconstruction... but for the recycling application, there are a handful of companies tackling this problem -- e.g. Everest Labs & Amp Robotics.
- Animats 5y ago
  The big name in this is Bulk Handling Systems.[1] They build and sell large complete recycling plants. They do about 90% of the sorting with a series of mechanical processes. The robots do only the "quality control" step, pulling out items that got past the choppers, shakers, screens, air blasts, and magnets. That gets the outputs up to 99%+ of the desired product, so it can go into recycling processes that take in cardboard, plastics sorted by type, etc. It makes the robotics cost effective. San Francisco's recycling operation, at Pier 95, uses a system built by them. It's not a glamorous technology, but it gets the job done. [1] https://www.max-ai.com/video-max-ai-autonomous-qc/ https://www.max-ai.com/video-max-ai-autonomous-qc/

billconan 5y ago

can ARkit return accurate camera position?

upbeat_general 5y ago
I haven’t looked at any metrics but based on using ARKit applications (and various VIO SLAM implementations) it can but it depends highly on the scene, camera motion, and whether there is LIDAR/Stereo Depth.

adampk 5y ago

I am surprised that the team didn't choose to add the "Fusion" append at the end. This seems to fit into the genealogy of KinectFusion, ElasticFusion, BundleFusion, etc. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... https://www.imperial.ac.uk/dyson-robotics-lab/downloads/elastic-fusion/ https://www.imperial.ac.uk/dyson-robotics-lab/downloads/elas... https://graphics.stanford.edu/projects/bundlefusion/ https://graphics.stanford.edu/projects/bundlefusion/ Very impressive work. I have not seen any use cases for online 3D reconstruction unfortunately. 6D.ai made terrific progress in this tech but also could not find great use cases for online reconstruction and ended up having to sell to Niantic. Seems like what people want, if they want 3D reconstruction, is extremely high-fidelity scans (a la Matterport) and are willing to wait for the model. Unfortunately TSDF approach create a "slimey" end look which isn't usually what people are after if they want an accurate 3D reconstruction. It SEEMS like online 3D reconstruction would be helpful, but I have yet to see a use case for "online"...

deleted 5y ago
[deleted]
tintor 5y ago
Use case: Mobile robotics, lidar replacement in self-driving vehicles
tonyarkles 5y ago
I'm very curious to see how well this would work for online terrain reconstruction. I've got a drone with a pretty powerful onboard computer and it's always nice to be able to solve and tune problems with software instead of additional (e.g. LIDAR) hardware.
eternityforest 5y ago
Cave exploration. Put the helmet on, never get lost. Also youtube videos and such with more complex animated characters jumping around and things. If they could convert that to a floorplan diagram it would be wonderful for UI design, lots of use cases for a map.

AndrewKemendo 5y ago

Honestly this doesn't look any better than what we were doing back in 2016-2017. I'm not sure what's novel here. This is the only video I could find, but we were doing monocular reconstruction from a limited number of RGB (not depth) images AND doing voxel segmentation on the processing side. https://www.youtube.com/watch?v=nqy44VSWh3g https://www.youtube.com/watch?v=nqy44VSWh3g Even as far back as 2010 people were doing reasonable monocular reconstruction including software like meshroom etc...the whole of TU Munich also under Matthias Niessner has been doing this for a while. What's novel here?

tintor 5y ago
Fast enough to be used for mobile robots?
closetnerd 5y ago
Says it’s real-time
- AndrewKemendo 5y ago
  2016 from Matthias Niessner's group https://www.youtube.com/watch?v=keIirXrRb1k https://www.youtube.com/watch?v=keIirXrRb1k http://graphics.stanford.edu/projects/bundlefusion/ http://graphics.stanford.edu/projects/bundlefusion/
  - jonas21 5y ago
    That requires depth input
    - AndrewKemendo 5y ago
      Good point, I don't recall offhand the paper that was the mono-RT one. At a minimum though 6D.ai and a few others had companies that were selling this as a service at least as far back as 2017.
      - 5y ago

polishdude20 5y ago

Shame there's no Android or iPhone app available

spyder 5y ago

It's not exactly the same but Neural Radiance Fields are getting more impressive: First one was this but it was slow: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf Then it's got faster: https://www.youtube.com/watch?v=fvXOjV7EHbk https://www.youtube.com/watch?v=fvXOjV7EHbk Lot of interesting papers: https://github.com/yenchenlin/awesome-NeRF https://github.com/yenchenlin/awesome-NeRF

leobg 5y ago

So much for the folks who think Tesla is on a fool’s errand when they’re using cameras instead of LIDAR.

kajecounterhack 5y ago
Companies like Waymo and Cruise use this kind of technology too. Unfortunately there are tons of corner cases of weird things you haven't seen before -- for example, some special vehicles self-occlude and you never get enough coverage to observe them correctly until you're too close. In general, radars and lidars used in _conjunction_ with cameras can handle occluded objects much better. Also, to measure the performance / evaluate observations generated from this tech, you would want to compare it to a pretty sizable 3D ground truth set which Tesla does not currently have. There are pretty big advantages to starting with a maximal set of sensors even if (eventually) breakthroughs turn them into unnecessary crutches.
- leobg 5y ago
  That was very insightful. Do you work in that space? It is comments like yours that make HN a special place.
- quantumduck 5y ago
  I agree that there are pretty big advantages to start with maximal set of sensors, but that is true only when you are a company with the sole goal of trying to make autonomy stack with shit ton of money and a product that you can modify because you own it. That's not true in the case of Tesla. They started as a EV company that offered autonomy later and since they were selling the product, they had to decide a minimal set of sensors that would, in theory, still work to keep the product cost reasonable. Tesla does have a sizeable 3D ground truth; they collected tons of data mounting LiDAR on their test vehicles.
  - kajecounterhack 5y ago
    > Tesla does have a sizeable 3D ground truth; they collected tons of data mounting LiDAR on their test vehicles. "Sizable" for evaluating safety has to be big enough to give small confidence intervals on your error -- and with self-driving, you need to cover a robust set of rare scenarios upsampled from general driving as well. I really doubt Tesla has what they need to convince themselves of higher levels of safety.

stefan_ 5y ago

Looked cool, then I read that there is some Apple ARkit magic blackbox in the middle of it all.

fxtentacle 5y ago
Yeah, I also consider that odd to use LIDAR-based poses and then call it "monocular".
- upbeat_general 5y ago
  Agreed, I’m interested to see how sensitive it is to camera pose error, and especially long-term drift.
cmelbye 5y ago
I don't think that's true. The paper says that a camera pose estimated by a SLAM system is required. ARKit implements SLAM and can easily provide camera pose for each frame through the ARFrame class. But there are countless other implementations of SLAM, including Android ARCore, Oculus Quest, Roomba, self-driving cars, and a number of GitHub repos (https://github.com/tzutalin/awesome-visual-slam https://github.com/tzutalin/awesome-visual-slam).

Animats 5y ago

That's impressive, in that it handles white walls well. Most of the algorithms for that sort of thing have trouble registering big uniform surfaces. That's why many tracking algorithms want objects with lots of texture detail. Some add texture by projecting a noise pattern on the surface being tracked. That's sometimes called "unstructured light", the opposite being projecting a structured pattern such as a grid. The first generation Kinect did that. Of course, this network may do well on this because it's trained on indoor scenes with walls of uniform height and width. Uniform implies flat is likely to emerge as an assumption. You get to see the wall/floor joints and the wall/wall joints, so you have some references. Remember the Tesla that hit the big white semitrailer because the algorithm couldn't measure depth to a uniform surface? This is a hard problem in unstructured situations.