5 ms·

DUSt3R: Geometric 3D Vision Made Easy

145 points by smusamashah 3y ago

globalnode 3y ago
As someone with relatively little knowledge of machine learning, an overview of mvs systems and a desire to understand what these people have done, does anyone have any suggestions on where to start learning so that I could play with the ideas within one lifetime? Would I be wasting my time on basics of machine learning if I only have stereo vision as an interest?
smusamashah 3y ago
People have been posting some really interesting and useful use cases of this tech Getting 3d view from few pictures of an apartment's listing https://twitter.com/JeromeRevaud/status/1764035510236758096 https://twitter.com/JeromeRevaud/status/1764035510236758096 Two pictures of kitchen https://x.com/janusch_patas/status/1764025964915302400 https://x.com/janusch_patas/status/1764025964915302400 Two pictures of office without any overlap https://x.com/JeromeRevaud/status/1763495315389165963 https://x.com/JeromeRevaud/status/1763495315389165963
reactordev 3y ago
This is awesome. Kudos. You have way more respect in my eyes since, not only did you post your paper, you posted the source. Too many times I’ve read claims without source so no one can reproduce and verify results. Now I can, and have, verified the results. Top notch.
- carbocation 3y ago
  Agreed. And not just the source, but a fully functional local demo! Runs great on my M1 pro using: PYTORCH_ENABLE_MPS_FALLBACK=1 python3.10 demo.py --weights checkpoints/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth --device 'mps'
amelius 3y ago
Does this method also work when different cameras (or different camera zoom etc. settings) are used for every input image?
- carbocation 3y ago
  I played around with DUSt3R last night using an iPhone with different lenses (whether you consider this a different camera or not, I defer to you). It worked well. Note that the camera intrinsics aren't used here, so it makes sense that it would tolerate different lenses or cameras. I did not test wildly divergent lens types (e.g., a fisheye lens).
markisus 3y ago
Pretty impressive. I wonder though why it was necessary to put the point map of the second image into the coordinate frame of the first. Isn’t it all the same from the point of the neural net?
Lichtso 3y ago
Am I imagining things or is there a trend here? Seems like we get more and more generalist approaches which are less specific and combine a lot of what used to be individual steps and techniques. In doing so they don't only become conceptually simpler but surprisingly more accurate as well. Possibly because a unified approach is more integrated and thus better at filling the gaps in one sub-problem with information form other sub-problems.
- bonoboTP 3y ago
  That's the trend since 2012 basically, when deep learning took over from hand-tuned feature extraction for image classification. The fiddly, brittle and multi-step nature of 3D vision endured longer but is going through the same transformation.
- xanderlewis 3y ago
  Somewhat related to The Bitter Lesson (though perhaps you’ve already read it): > One thing that should be learned from the bitter lesson is the great power of general purpose methods http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
monkeydust 3y ago
Can this be used for body measurement, eg 4 shots different poses combined together? What kind of accuracy might you get if so ...just curious?
- carbocation 3y ago
  Different poses? I don't think so, at least not with the current setup. For example, I tried this with a dog (walking around the seated dog, taking photos as I did so). The dog turned her head while I was taking photos. The portion of the head that moved was not represented in the final output.
fxtentacle 3y ago
If I understand things correctly, this relies extremely heavily on learned prior shapes, meaning it'll guess depth from a monocular image and then go from there. In line with that, it uses a Vision Transformer like MiDaS (Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer). That's why it can still reconstruct a scene even if the images do not overlap at all: https://twitter.com/JeromeRevaud/status/1763495315389165963 https://twitter.com/JeromeRevaud/status/1763495315389165963 But what that also means is that this is closer to generative AI than to objective measurements. If the image to depth estimation goes very wrong, it might hallucinate shapes that aren't there.
- krasin 3y ago
  > But what that also means is that this is closer to generative AI than to objective measurements. If the image to depth estimation goes very wrong, it might hallucinate shapes that aren't there. But people do that all the time too. Relying on priors is fine for many practical applications and sometimes there's no way around it.
euroderf 3y ago
Great! Now I need an ELI5 tutorial on taking pix of a property and generating a 3D model for 3D printing. Then the kid can request buildings for the model train layout!
yazan94 3y ago
Could this somehow be used to generate a Zillow-style tour of a physical space?

145 points by smusamashah 3y ago

As someone with relatively little knowledge of machine learning, an overview of mvs systems and a desire to understand what these people have done, does anyone have any suggestions on where to start learning so that I could play with the ideas within one lifetime? Would I be wasting my time on basics of machine learning if I only have stereo vision as an interest?

smusamashah 3y ago

People have been posting some really interesting and useful use cases of this tech Getting 3d view from few pictures of an apartment's listing https://twitter.com/JeromeRevaud/status/1764035510236758096 https://twitter.com/JeromeRevaud/status/1764035510236758096 Two pictures of kitchen https://x.com/janusch_patas/status/1764025964915302400 https://x.com/janusch_patas/status/1764025964915302400 Two pictures of office without any overlap https://x.com/JeromeRevaud/status/1763495315389165963 https://x.com/JeromeRevaud/status/1763495315389165963

reactordev 3y ago

This is awesome. Kudos. You have way more respect in my eyes since, not only did you post your paper, you posted the source. Too many times I’ve read claims without source so no one can reproduce and verify results. Now I can, and have, verified the results. Top notch.

carbocation 3y ago
Agreed. And not just the source, but a fully functional local demo! Runs great on my M1 pro using: PYTORCH_ENABLE_MPS_FALLBACK=1 python3.10 demo.py --weights checkpoints/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth --device 'mps'

amelius 3y ago

Does this method also work when different cameras (or different camera zoom etc. settings) are used for every input image?

carbocation 3y ago
I played around with DUSt3R last night using an iPhone with different lenses (whether you consider this a different camera or not, I defer to you). It worked well. Note that the camera intrinsics aren't used here, so it makes sense that it would tolerate different lenses or cameras. I did not test wildly divergent lens types (e.g., a fisheye lens).

markisus 3y ago

Pretty impressive. I wonder though why it was necessary to put the point map of the second image into the coordinate frame of the first. Isn’t it all the same from the point of the neural net?

Lichtso 3y ago

Am I imagining things or is there a trend here? Seems like we get more and more generalist approaches which are less specific and combine a lot of what used to be individual steps and techniques. In doing so they don't only become conceptually simpler but surprisingly more accurate as well. Possibly because a unified approach is more integrated and thus better at filling the gaps in one sub-problem with information form other sub-problems.

bonoboTP 3y ago
That's the trend since 2012 basically, when deep learning took over from hand-tuned feature extraction for image classification. The fiddly, brittle and multi-step nature of 3D vision endured longer but is going through the same transformation.
xanderlewis 3y ago
Somewhat related to The Bitter Lesson (though perhaps you’ve already read it): > One thing that should be learned from the bitter lesson is the great power of general purpose methods http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

monkeydust 3y ago

Can this be used for body measurement, eg 4 shots different poses combined together? What kind of accuracy might you get if so ...just curious?

carbocation 3y ago
Different poses? I don't think so, at least not with the current setup. For example, I tried this with a dog (walking around the seated dog, taking photos as I did so). The dog turned her head while I was taking photos. The portion of the head that moved was not represented in the final output.

fxtentacle 3y ago

If I understand things correctly, this relies extremely heavily on learned prior shapes, meaning it'll guess depth from a monocular image and then go from there. In line with that, it uses a Vision Transformer like MiDaS (Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer). That's why it can still reconstruct a scene even if the images do not overlap at all: https://twitter.com/JeromeRevaud/status/1763495315389165963 https://twitter.com/JeromeRevaud/status/1763495315389165963 But what that also means is that this is closer to generative AI than to objective measurements. If the image to depth estimation goes very wrong, it might hallucinate shapes that aren't there.

krasin 3y ago
> But what that also means is that this is closer to generative AI than to objective measurements. If the image to depth estimation goes very wrong, it might hallucinate shapes that aren't there. But people do that all the time too. Relying on priors is fine for many practical applications and sometimes there's no way around it.

euroderf 3y ago

Great! Now I need an ELI5 tutorial on taking pix of a property and generating a 3D model for 3D printing. Then the kid can request buildings for the model train layout!

yazan94 3y ago

Could this somehow be used to generate a Zillow-style tour of a physical space?