4 ms·

N-Dimensional Gaussians for Fitting of High Dimensional Functions

67 points by jasondavies 2y ago

3abiton 2y ago
This problem specifically (3D reconstruction with representation fitting) is really an overfitting nightmare, they just adapted to it not really overcame it. Nonetheless interesting work.
- blovescoffee 2y ago
  The point of a Nerf is effectively overfitting a neural network to a scene anyways (I think they say this in the original NeRF paper - or somewhere similar).
- barfbagginus 2y ago
  I get the feeling that some abstract geometric/topological approaches will let us get under fitted general pretrained NERFs that rapidly specialize to a given context, let you combine separate contexts into a composite scene, and maybe let's you give behaviors to entities in the scene. At that point you're kind of trying to generically train over all possible scene graphs. But maybe some kind of geometric delocalization scheme lets you train on small fragments of larger scenes, transfer learning between scenes, and quickly represent and render generic scenes.
vessenes 2y ago
When I think of turning sequences of images into gaussians, I think of the difficulty of getting generalizable information that can be re-rendered out of the pipeline; textures and lighting, basically. From the description at the top of the paper, where they mention adding dimensions for things like albedo, I got excited. But the demos don't do any re-rendering / change of lighting / etc, so I can't tell if this paper is just a 'super high render quality at same training time' paper, which is of course great to have, or if it has a shot at being extended to get us scenes that can be adjusted as to lighting and texture in-engine. Any experts care to chime in?
- WithinReason 2y ago
  The penultimate video has a lighting change
- nurettin 2y ago
  12d gaussians video has a moving light source and z rotation.
- barfbagginus 2y ago
  You should be able to add those features as extra dimensions. I wonder how well this works for scenes with multiple characters, posed skeletons for the characters, etc How would you avoid degenerate parts of the configuration space, where characters clip through each other, clip through level features, etc? Could it theoretically scale to render a rave crowd with 1000 random people dancing under wild lighting effects?
wild_g00sian 2y ago
[dead]
hackandthink 2y ago
I expected Gaussian processes. Can someone relate this to Gaussian processes?
- blovescoffee 2y ago
  If you understand what a Gaussian Process is, you understand what a Gaussian Distribution is. This work estimates the parameters of many Gaussian Distribution in order to fit the approximate geometry of a GD to a geometry in a scene.
- siddboots 2y ago
  A Gaussian process fits a single high dimensional Gaussian, for example, by treating n observations along a single dimension as a n dimensional space. Gaussian mixture models fit a large number of low dimensional Gaussians for example you might imagine 2D data generated by several 2D Gaussian superimposed. This approach is just an example of the latter. It uses higher dimensional Gaussians to capture extra information from a scene, but not in the emulation of an infinite dimensional space in the way that defines Gaussian processes.
  - abhgh 2y ago
    To add to a sibling comment, if you're interested in learning a bit about the both the Gaussian (as in a density estimator like Gassian Mixture Models, aka GMMs) vs Gaussian Processes (GP), I have some write-ups here: [1] and [2]. [1] Fun with GMMs https://blog.quipu-strands.com/fun_with_GMMs https://blog.quipu-strands.com/fun_with_GMMs [2] This is a larger article on BayesOpt, but I've a section dedicated to GPs: https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs#gaussian_processes https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs#gaus...

67 points by jasondavies 2y ago

This problem specifically (3D reconstruction with representation fitting) is really an overfitting nightmare, they just adapted to it not really overcame it. Nonetheless interesting work.

blovescoffee 2y ago
The point of a Nerf is effectively overfitting a neural network to a scene anyways (I think they say this in the original NeRF paper - or somewhere similar).
barfbagginus 2y ago
I get the feeling that some abstract geometric/topological approaches will let us get under fitted general pretrained NERFs that rapidly specialize to a given context, let you combine separate contexts into a composite scene, and maybe let's you give behaviors to entities in the scene. At that point you're kind of trying to generically train over all possible scene graphs. But maybe some kind of geometric delocalization scheme lets you train on small fragments of larger scenes, transfer learning between scenes, and quickly represent and render generic scenes.

vessenes 2y ago

When I think of turning sequences of images into gaussians, I think of the difficulty of getting generalizable information that can be re-rendered out of the pipeline; textures and lighting, basically. From the description at the top of the paper, where they mention adding dimensions for things like albedo, I got excited. But the demos don't do any re-rendering / change of lighting / etc, so I can't tell if this paper is just a 'super high render quality at same training time' paper, which is of course great to have, or if it has a shot at being extended to get us scenes that can be adjusted as to lighting and texture in-engine. Any experts care to chime in?

WithinReason 2y ago
The penultimate video has a lighting change
nurettin 2y ago
12d gaussians video has a moving light source and z rotation.
barfbagginus 2y ago
You should be able to add those features as extra dimensions. I wonder how well this works for scenes with multiple characters, posed skeletons for the characters, etc How would you avoid degenerate parts of the configuration space, where characters clip through each other, clip through level features, etc? Could it theoretically scale to render a rave crowd with 1000 random people dancing under wild lighting effects?

wild_g00sian 2y ago

[dead]

hackandthink 2y ago

I expected Gaussian processes. Can someone relate this to Gaussian processes?

blovescoffee 2y ago
If you understand what a Gaussian Process is, you understand what a Gaussian Distribution is. This work estimates the parameters of many Gaussian Distribution in order to fit the approximate geometry of a GD to a geometry in a scene.
siddboots 2y ago
A Gaussian process fits a single high dimensional Gaussian, for example, by treating n observations along a single dimension as a n dimensional space. Gaussian mixture models fit a large number of low dimensional Gaussians for example you might imagine 2D data generated by several 2D Gaussian superimposed. This approach is just an example of the latter. It uses higher dimensional Gaussians to capture extra information from a scene, but not in the emulation of an infinite dimensional space in the way that defines Gaussian processes.
- abhgh 2y ago
  To add to a sibling comment, if you're interested in learning a bit about the both the Gaussian (as in a density estimator like Gassian Mixture Models, aka GMMs) vs Gaussian Processes (GP), I have some write-ups here: [1] and [2]. [1] Fun with GMMs https://blog.quipu-strands.com/fun_with_GMMs https://blog.quipu-strands.com/fun_with_GMMs [2] This is a larger article on BayesOpt, but I've a section dedicated to GPs: https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs#gaussian_processes https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs#gaus...