5 ms·

Making Gaussian Splats more smaller

91 points by ibobev 3y ago

Scene_Cast2 3y ago
Oh, so that's how Gaussian Splats achieve view-dependence. Strongly reminds me of work on realtime relighting from a decade ago. IIRC some version of Halo used a couple of SH coefficients (either per pixel or per vertex) for some dynamic shadow / lighting thing. Then, it devolved into figuring out a better set of decomposition functions (SH is Fourier Transform over a spherical coordinate basis). IIRC Haar wavelets worked well too. I'd guess that with modern approaches, a learned function would do even better.
- bhouston 3y ago
  Halo and SH: https://pdfs.semanticscholar.org/566f/c019459435ac3ad25d9a4941d5da02d7ba59.pdf https://pdfs.semanticscholar.org/566f/c019459435ac3ad25d9a49...
imbusy111 3y ago
There's probably some locality you can exploit as well. Given the same surface and close proximity, I imagine the coefficients are nearly identical. Or store only the rotation of certain typical small set of SH coefficients in two values + index. For an ideal, single material sphere, all the coefficients could be lined up exactly, right?
- ReactiveJelly 3y ago
  I think the SH of a splat also depends on geometry around it, for shadows and reflections, so if the sphere is inside a scene, the sphere's SHs may not all be rotations of each other. Cause I've seen SH used for baked-in shadows.
  - imbusy111 3y ago
    Very true. But the splats cannot, at the moment, account for moving lights, nor dynamic scenes, so it seems like only the 0-th harmonic would be affected by shadowing.
- aras_p 3y ago
  I looked at that in the previous post (https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-smaller/ https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-s...) and while there's some nearby correllations in positions & colors (for colors I actually (ab)use that by doing BC7 compression on that data), the rotations & scales are "pretty much random". Which makes sense; if they were not random the GS optimizer would probably have represented them as smaller amount of larger splats.
- BenoitP 3y ago
  Feels like the field is testing an changing lots of parameters right now. Hell, we were doing volumetric NeRF not long ago. It's going to be very interesting to where it converges. I would not be surprised to see volumetric wavelets at some point.
samspenc 3y ago
As someone dabbling in the 3D space as a hobby, Gaussian Splats have been the hot trend these past few weeks and everyone in the 3D space is looking into it for getting realistic photo scans into 3D, as an alternative / supplement to NeRF and photogrammetry.
theschwa 3y ago
"How does “reflection” of the vase on the metal part of the table work, you might ask? The gaussians have “learned” the ages-old trick of duplicating and mirroring the geometry for reflection!" Thank you! I'd seen people remove some of the SH coefficients while still getting reflections, and I couldn't understand how that worked. The old tricks sometimes really are the best tricks.
- contravariant 3y ago
  Does make you wonder if you'd see a shadowy vase if you look at the table from below. Although now that I think about it, there's no real reason to not just make the Gaussians invisible from behind.
  - ReactiveJelly 3y ago
    I think I've seen so in videos. Like someone dipping a camera underwater and the reflection is real splats. But since the splats aren't a surface, they don't exactly have a "behind". They might render from every angle on purpose.
    - BenoitP 3y ago
      > But since the splats aren't a surface, they don't exactly have a "behind". Spherical Harmonics apply to the whole sphere, splats can learn a SH color for their "behind". But by definition there is no data for these (no camera to tell what the color is). Nothing is preventing current pipelines to define that the opposite direction (splat looking at the camera) has a special color (black, zero-alpha, blend of blurred splats between camera and splat, etc); or to regularize all splats so that there is some amount of an "undefined" transparent component to be applied where camera won't define the SH.
      - vanderZwan 3y ago
        > Nothing is preventing current pipelines to define that the opposite direction (splat looking at the camera) has a special color (black, zero-alpha, blend of blurred splats between camera and splat, etc) Wouldn't you need to know that there is no other camera view from the opposite direction? (sorry if dumb question, haven't actually looked at how GS are generated from input data)
firtoz 3y ago
Impressive article! > Of course gaussian splatting at this point is useful for “rotate around a scanned object” use case; it is not useful for “in games” or many other cases. We don’t know how to re-light them, or how to animate them well, etc. etc. Yet. I know almost for certain that animating them would be a serious challenge, but for relighting, I can more easily imagine some functions that filter some splats based on some kind of spatial and directional querying and applying multipliers for some of their values, however I'm practically daydreaming at this point.
- ReactiveJelly 3y ago
  I guess the hard part is reliably figuring out the albedo / color of a darkly shadowed area. It's kind of like dividing by zero. For materials, you could look at the SH and try to fit it into metallic / plastic / matte. But they already got photogrammetry into games, and this is just photogrammetry with splats instead of triangles, right? There's got to be a way.
- vanderZwan 3y ago
  Alternatively just do a Myst style game. Actually, I think that could be a very nice aesthetic to explore in a hypothetical Myst remake

91 points by ibobev 3y ago

Oh, so that's how Gaussian Splats achieve view-dependence. Strongly reminds me of work on realtime relighting from a decade ago. IIRC some version of Halo used a couple of SH coefficients (either per pixel or per vertex) for some dynamic shadow / lighting thing. Then, it devolved into figuring out a better set of decomposition functions (SH is Fourier Transform over a spherical coordinate basis). IIRC Haar wavelets worked well too. I'd guess that with modern approaches, a learned function would do even better.

bhouston 3y ago
Halo and SH: https://pdfs.semanticscholar.org/566f/c019459435ac3ad25d9a4941d5da02d7ba59.pdf https://pdfs.semanticscholar.org/566f/c019459435ac3ad25d9a49...

imbusy111 3y ago

There's probably some locality you can exploit as well. Given the same surface and close proximity, I imagine the coefficients are nearly identical. Or store only the rotation of certain typical small set of SH coefficients in two values + index. For an ideal, single material sphere, all the coefficients could be lined up exactly, right?

ReactiveJelly 3y ago
I think the SH of a splat also depends on geometry around it, for shadows and reflections, so if the sphere is inside a scene, the sphere's SHs may not all be rotations of each other. Cause I've seen SH used for baked-in shadows.
- imbusy111 3y ago
  Very true. But the splats cannot, at the moment, account for moving lights, nor dynamic scenes, so it seems like only the 0-th harmonic would be affected by shadowing.
aras_p 3y ago
I looked at that in the previous post (https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-smaller/ https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-s...) and while there's some nearby correllations in positions & colors (for colors I actually (ab)use that by doing BC7 compression on that data), the rotations & scales are "pretty much random". Which makes sense; if they were not random the GS optimizer would probably have represented them as smaller amount of larger splats.
BenoitP 3y ago
Feels like the field is testing an changing lots of parameters right now. Hell, we were doing volumetric NeRF not long ago. It's going to be very interesting to where it converges. I would not be surprised to see volumetric wavelets at some point.

samspenc 3y ago

As someone dabbling in the 3D space as a hobby, Gaussian Splats have been the hot trend these past few weeks and everyone in the 3D space is looking into it for getting realistic photo scans into 3D, as an alternative / supplement to NeRF and photogrammetry.

theschwa 3y ago

"How does “reflection” of the vase on the metal part of the table work, you might ask? The gaussians have “learned” the ages-old trick of duplicating and mirroring the geometry for reflection!" Thank you! I'd seen people remove some of the SH coefficients while still getting reflections, and I couldn't understand how that worked. The old tricks sometimes really are the best tricks.

contravariant 3y ago
Does make you wonder if you'd see a shadowy vase if you look at the table from below. Although now that I think about it, there's no real reason to not just make the Gaussians invisible from behind.
- ReactiveJelly 3y ago
  I think I've seen so in videos. Like someone dipping a camera underwater and the reflection is real splats. But since the splats aren't a surface, they don't exactly have a "behind". They might render from every angle on purpose.
  - BenoitP 3y ago
    > But since the splats aren't a surface, they don't exactly have a "behind". Spherical Harmonics apply to the whole sphere, splats can learn a SH color for their "behind". But by definition there is no data for these (no camera to tell what the color is). Nothing is preventing current pipelines to define that the opposite direction (splat looking at the camera) has a special color (black, zero-alpha, blend of blurred splats between camera and splat, etc); or to regularize all splats so that there is some amount of an "undefined" transparent component to be applied where camera won't define the SH.
    - vanderZwan 3y ago
      > Nothing is preventing current pipelines to define that the opposite direction (splat looking at the camera) has a special color (black, zero-alpha, blend of blurred splats between camera and splat, etc) Wouldn't you need to know that there is no other camera view from the opposite direction? (sorry if dumb question, haven't actually looked at how GS are generated from input data)

firtoz 3y ago

Impressive article! > Of course gaussian splatting at this point is useful for “rotate around a scanned object” use case; it is not useful for “in games” or many other cases. We don’t know how to re-light them, or how to animate them well, etc. etc. Yet. I know almost for certain that animating them would be a serious challenge, but for relighting, I can more easily imagine some functions that filter some splats based on some kind of spatial and directional querying and applying multipliers for some of their values, however I'm practically daydreaming at this point.

ReactiveJelly 3y ago
I guess the hard part is reliably figuring out the albedo / color of a darkly shadowed area. It's kind of like dividing by zero. For materials, you could look at the SH and try to fit it into metallic / plastic / matte. But they already got photogrammetry into games, and this is just photogrammetry with splats instead of triangles, right? There's got to be a way.
vanderZwan 3y ago
Alternatively just do a Myst style game. Actually, I think that could be a very nice aesthetic to explore in a hypothetical Myst remake