10 ms·

MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images

131 points by jasondavies 2y ago

axoltl 2y ago
I'm having a hard time finding a reference to the hardware the inference is run on. The paper mentions training was done on a single A100 GPU so I'm going to assume inference was run on that same platform. The 22fps result is somewhat meaningless without that information. It does feel like we're getting closer and closer to being able to synthesize novel views in realtime from a small set of images at a framerate and quality high enough for use in AR, which is an interesting concept. I'd love to be able to 'walk around' in my photo library.
- tomp 2y ago
  Once the Gaussian Splats are computed (whether via ML or classical optimisation), they’re very efficient to render (similar to 3D meshes used in games). High fps isn’t incredible. Having said that (I have yet to read the paper), "efficiency" probably refers to the first part (calculating the gaussians in the first place) not rendering.
  - axoltl 2y ago
    You are correct. I was confusing this technique with Novel View Synthesis through diffusion (recent paper: https://arxiv.org/abs/2408.06157 https://arxiv.org/abs/2408.06157) where inference means generating frames rather than points.
  - dagmx 2y ago
    They’re not “very efficient “. They have a significant amount of overdraw due to their transparency and will be a lot more inefficient if you’re only considering material-less surface representation. They’re more efficient to capture however . They’re also more constant in their render time, but meshes will easily be faster in most scenes cases, but scale worse with complexity. The “efficiency” of splats is more about the material response and capturing complexity there, than it is about geometric representation.
- iforgotpassword 2y ago
  > I'd love to be able to 'walk around' in my photo library. Yes this. I've been dreaming about this since I digitized my childhood photos a few years ago. There should be more than enough photos to reconstruct the entire apartment. Or my grandparents' house. Not sure though what happens if items and furniture moves around between shots. I haven't looked much into this yet and just assumed it will need a bit more time until there is a batteries included solution I can just download and run without reading ten pages of instructions and buying a GPU cluster.
programjames 2y ago
Where would you use 3D Gaussian splatting? Static environments for video games?
- halfbreed 2y ago
  I still wonder this myself, but the most obvious area that comes to mind is real estate virtual tours. Once a splat can render in the browser at high fps, then I see this replacing most all other technologies currently being used.
- 55555 2y ago
  Virtual tours for real estate
  - littlestymaar 2y ago
    Are there businesses doing it already or is the tech too immature to be used IRL right now?
    - apinstein 2y ago
      I started and ran a real estate photography platform from 2004-2018. We started r&d on this in ~2016 when consumer VR first came out. At the time we used photogrammetry and it was “dreadful” to try to capture due to mirrors, glass, etc. So I have been following GS tech for a while. I’ve not yet seen anything (open source / papers) that quite gets there yet. I do think it will. In my opinion, there are two useful ways GS can bring to this industry. The first is ability to use photo capture to re-render as a high production quality video similar to what people do with Luma AI today. While this is a really cool capability, it’s also not really that hard to do anymore with drones and gimbals. So, the experience of creating the same thing via GS has to be better and easier, and it’s not clear when that will likely happen due to how painful the capture side is. You really need good real time capture feedback to make sure you have good coverage. Finding out there’s a hole once you’re off location is a deal breaker. The second is to create VR capable experiences. I think the first real useful thing for consumers will be so you can walk around in a small three or 4 foot area and get a stereo sense of what it’s like to be there. This is an amazing consumer experience. But the practicality of scaling this depends on VR hardware and adoption, and that hasn’t yet become commonplace enough to make consumer use “adjacent possible” for broad deployment. I could see it being used on super high end to start out.
rebuilder 2y ago
The indoor example with the staircase and railing was really surprising - there's only one view of much of what's behind the doorframe and it still seems to reconstruct a pretty good 3d scene there.
petargyurov 2y ago
Someone help me understand inference here. Every gaussian splat repo I have looked at doesn't mention how to use the pre-trained models to "simply" take MY images as input and output a GS. They all talk about evaluation, but the CMD interface requires the eval datasets as input. Is training/fine-tuning on my data the only way to get the output?
- littlestymaar 2y ago
  Is there really such thing as a pre-trained model when it comes to Gaussian splatting? I'm not familiar at all with the topic (nor have I read this particular paper) but I remember that the original 3DGS paper took pride in the fact that this was not “IA” or “deep learning”. There's still a gradient descent process to get the Gaussian splats from the data, but as I understood it, there is no “training on a large dataset then inference”, building the GS from your data is the “training phase” and then rendering it is the equivalent of inference. Maybe I understood it all wrong though, or maybe new variants of Gaussian splatting use a deep learning network in addition to what was done in the original work, so I'll be happy to be corrected/clarified by someone with actual knowledge here.
- jorgemf 2y ago
  Basically you train a model per each set of images. The model is a neural network able to render the final image. Different images will require different trained models. Initial gaussian splatting models took hours to train, last year models took minutes to train. I am not sure how much this one takes, but it should be between minutes and hours (and probably more close to minutes than hours).
  - petargyurov 2y ago
    Thank you, that explains it.
  - tomp 2y ago
    No, what you're describing is NeRF, the predecessor technology. The output of Gaussian Splat "training" is a set of 3d gaussians, which can be rendered very quickly. No ML involved at all (only optimisation)! They usually require running COLMAP first (to get the relative location of camera between different images), but NVIDIA's InstantSplat doesn't (it however does use a ML model instead!)
vessenes 2y ago
The tech stack in the splat world is still really young. For instance, I was thinking to myself: “Cool, MVSplat is pretty fast. Maybe I’ll use it to get some renderings of a field by my house.” As far as I can tell, I will need to offer a bunch of photographs with camera pose data added — okay, fair enough, the splat architecture exists to generate splats. Now, what’s the best way to get camera pose data from arbitrary outdoor photos? … Cue a long wrangle through multiple papers. Maybe, as of today… FAR? (https://crockwell.github.io/far/ https://crockwell.github.io/far/). That claims up to 80% pose accuracy depending on source data. I have no idea how MVSplat will deal with 80% accurate camera pose data… And I also don’t understand if I should use a pre-trained model from them or train my own or fine tune one of their models on my photos… This is sounding like a long project. I don’t say this to complain, only to note where the edges are right now, and think about the commercialization gap. There are iPhone apps that will get (shitty) splats together for you right now, and there are higher end commercial projects like Skydio that will work with a drone to fill in a three dimensional representation of an object (or maybe some land, not sure about the outdoor support), but those are like multiple thousand-dollar per month subscriptions + hardware as far as I can tell. Anyway, interesting. I expect that over the next few years we’ll have push button stacks based on ‘good enough’ open models, and those will iterate and go through cycles of being upsold / improved / etc. We are still a ways away from a trawl through an iPhone/gphoto library and a “hey, I made some environments for you!” Type of feature. But not infinitely far away.
- ryandamm 2y ago
  I think the barrier to commercialization is the lack of demonstrated economic value to having push button splats. There's no shortage of small teams wiring together open source splats / NeRF / whatever papers; there's a dearth of valuable, repeatable businesses that could make use of what those small teams are building. Would it be cool to just have content in 3D? Undoubtedly. But figuring out a use case, that's where people need to be focusing. I think there are a lot of opportunities, but it's still early days -- and not just for the technology.
  - vessenes 2y ago
    Yes - agreed. There’s a clear use case for indie content, but tooling around editing/modifying/color/lighting has to improve, and rendering engines or converters need to get better. FWIW it doesn’t seem like a dead-end tech to me though; more likely a gateway tech to cost improvements. We’ll see.

131 points by jasondavies 2y ago

I'm having a hard time finding a reference to the hardware the inference is run on. The paper mentions training was done on a single A100 GPU so I'm going to assume inference was run on that same platform. The 22fps result is somewhat meaningless without that information. It does feel like we're getting closer and closer to being able to synthesize novel views in realtime from a small set of images at a framerate and quality high enough for use in AR, which is an interesting concept. I'd love to be able to 'walk around' in my photo library.

tomp 2y ago
Once the Gaussian Splats are computed (whether via ML or classical optimisation), they’re very efficient to render (similar to 3D meshes used in games). High fps isn’t incredible. Having said that (I have yet to read the paper), "efficiency" probably refers to the first part (calculating the gaussians in the first place) not rendering.
- axoltl 2y ago
  You are correct. I was confusing this technique with Novel View Synthesis through diffusion (recent paper: https://arxiv.org/abs/2408.06157 https://arxiv.org/abs/2408.06157) where inference means generating frames rather than points.
- dagmx 2y ago
  They’re not “very efficient “. They have a significant amount of overdraw due to their transparency and will be a lot more inefficient if you’re only considering material-less surface representation. They’re more efficient to capture however . They’re also more constant in their render time, but meshes will easily be faster in most scenes cases, but scale worse with complexity. The “efficiency” of splats is more about the material response and capturing complexity there, than it is about geometric representation.
iforgotpassword 2y ago
> I'd love to be able to 'walk around' in my photo library. Yes this. I've been dreaming about this since I digitized my childhood photos a few years ago. There should be more than enough photos to reconstruct the entire apartment. Or my grandparents' house. Not sure though what happens if items and furniture moves around between shots. I haven't looked much into this yet and just assumed it will need a bit more time until there is a batteries included solution I can just download and run without reading ten pages of instructions and buying a GPU cluster.

programjames 2y ago

Where would you use 3D Gaussian splatting? Static environments for video games?

halfbreed 2y ago
I still wonder this myself, but the most obvious area that comes to mind is real estate virtual tours. Once a splat can render in the browser at high fps, then I see this replacing most all other technologies currently being used.
55555 2y ago
Virtual tours for real estate
- littlestymaar 2y ago
  Are there businesses doing it already or is the tech too immature to be used IRL right now?
  - apinstein 2y ago
    I started and ran a real estate photography platform from 2004-2018. We started r&d on this in ~2016 when consumer VR first came out. At the time we used photogrammetry and it was “dreadful” to try to capture due to mirrors, glass, etc. So I have been following GS tech for a while. I’ve not yet seen anything (open source / papers) that quite gets there yet. I do think it will. In my opinion, there are two useful ways GS can bring to this industry. The first is ability to use photo capture to re-render as a high production quality video similar to what people do with Luma AI today. While this is a really cool capability, it’s also not really that hard to do anymore with drones and gimbals. So, the experience of creating the same thing via GS has to be better and easier, and it’s not clear when that will likely happen due to how painful the capture side is. You really need good real time capture feedback to make sure you have good coverage. Finding out there’s a hole once you’re off location is a deal breaker. The second is to create VR capable experiences. I think the first real useful thing for consumers will be so you can walk around in a small three or 4 foot area and get a stereo sense of what it’s like to be there. This is an amazing consumer experience. But the practicality of scaling this depends on VR hardware and adoption, and that hasn’t yet become commonplace enough to make consumer use “adjacent possible” for broad deployment. I could see it being used on super high end to start out.

rebuilder 2y ago

The indoor example with the staircase and railing was really surprising - there's only one view of much of what's behind the doorframe and it still seems to reconstruct a pretty good 3d scene there.

petargyurov 2y ago

Someone help me understand inference here. Every gaussian splat repo I have looked at doesn't mention how to use the pre-trained models to "simply" take MY images as input and output a GS. They all talk about evaluation, but the CMD interface requires the eval datasets as input. Is training/fine-tuning on my data the only way to get the output?

littlestymaar 2y ago
Is there really such thing as a pre-trained model when it comes to Gaussian splatting? I'm not familiar at all with the topic (nor have I read this particular paper) but I remember that the original 3DGS paper took pride in the fact that this was not “IA” or “deep learning”. There's still a gradient descent process to get the Gaussian splats from the data, but as I understood it, there is no “training on a large dataset then inference”, building the GS from your data is the “training phase” and then rendering it is the equivalent of inference. Maybe I understood it all wrong though, or maybe new variants of Gaussian splatting use a deep learning network in addition to what was done in the original work, so I'll be happy to be corrected/clarified by someone with actual knowledge here.
jorgemf 2y ago
Basically you train a model per each set of images. The model is a neural network able to render the final image. Different images will require different trained models. Initial gaussian splatting models took hours to train, last year models took minutes to train. I am not sure how much this one takes, but it should be between minutes and hours (and probably more close to minutes than hours).
- petargyurov 2y ago
  Thank you, that explains it.
- tomp 2y ago
  No, what you're describing is NeRF, the predecessor technology. The output of Gaussian Splat "training" is a set of 3d gaussians, which can be rendered very quickly. No ML involved at all (only optimisation)! They usually require running COLMAP first (to get the relative location of camera between different images), but NVIDIA's InstantSplat doesn't (it however does use a ML model instead!)

vessenes 2y ago

The tech stack in the splat world is still really young. For instance, I was thinking to myself: “Cool, MVSplat is pretty fast. Maybe I’ll use it to get some renderings of a field by my house.” As far as I can tell, I will need to offer a bunch of photographs with camera pose data added — okay, fair enough, the splat architecture exists to generate splats. Now, what’s the best way to get camera pose data from arbitrary outdoor photos? … Cue a long wrangle through multiple papers. Maybe, as of today… FAR? (https://crockwell.github.io/far/ https://crockwell.github.io/far/). That claims up to 80% pose accuracy depending on source data. I have no idea how MVSplat will deal with 80% accurate camera pose data… And I also don’t understand if I should use a pre-trained model from them or train my own or fine tune one of their models on my photos… This is sounding like a long project. I don’t say this to complain, only to note where the edges are right now, and think about the commercialization gap. There are iPhone apps that will get (shitty) splats together for you right now, and there are higher end commercial projects like Skydio that will work with a drone to fill in a three dimensional representation of an object (or maybe some land, not sure about the outdoor support), but those are like multiple thousand-dollar per month subscriptions + hardware as far as I can tell. Anyway, interesting. I expect that over the next few years we’ll have push button stacks based on ‘good enough’ open models, and those will iterate and go through cycles of being upsold / improved / etc. We are still a ways away from a trawl through an iPhone/gphoto library and a “hey, I made some environments for you!” Type of feature. But not infinitely far away.

ryandamm 2y ago
I think the barrier to commercialization is the lack of demonstrated economic value to having push button splats. There's no shortage of small teams wiring together open source splats / NeRF / whatever papers; there's a dearth of valuable, repeatable businesses that could make use of what those small teams are building. Would it be cool to just have content in 3D? Undoubtedly. But figuring out a use case, that's where people need to be focusing. I think there are a lot of opportunities, but it's still early days -- and not just for the technology.
- vessenes 2y ago
  Yes - agreed. There’s a clear use case for indie content, but tooling around editing/modifying/color/lighting has to improve, and rendering engines or converters need to get better. FWIW it doesn’t seem like a dead-end tech to me though; more likely a gateway tech to cost improvements. We’ll see.