3 ms·

Kinda disappointing that the spherical harmonic data is just totally thrown out. It's a pretty big factor in making gaussian splats look so good, and I'm sure t

Kinda disappointing that the spherical harmonic data is just totally thrown out. It's a pretty big factor in making gaussian splats look so good, and I'm sure there are interesting ways of compressing the data!

slimbuck 3y ago
It's definitely possible to compress/palletise the SH bands (see https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-more-smaller/ https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-m... for one approach). We just found that splats encode some view-dependent shininess anyway (for example the guitar body https://playcanvas.com/model-viewer?load=https://code.playcanvas.com/viewer/guitar-cleaned.ply https://playcanvas.com/model-viewer?load=https://code.playca...) and so considered the extra data and rendering cost of SH not worth it for now. It would be quite straightforward to add SH to the PLY file and we'll definitely consider it in future!
- ladberg 3y ago
  Oh nice! I didn't realize those kinds of effects were possible without SH
- jayd16 3y ago
  Are the baked in reflections technically bug or is it fundamental in some way?
  - taneq 3y ago
    They’re fundamental to the scene if you recite light transport and not just geometry. Gaussian splats are just a clever way to sample a lightfield and reflections are part of the lightfield.
    - jayd16 3y ago
      Well it's more that I'm asking if lighting data should live entirely in the spherical harmonics. It's certainly feasible and common in the game world. ...But maybe that doesn't make sense in the GS context.
- crubier 3y ago
  Yeah splats can encode view dependency without SH, but not with a correct geometry. E.g. often, you see reflections encoded in GS as just a 3d scene on the other side of the mirror. Other view dependency effect can make the surfaces "fluffy", etc. This is not great when you also try to get decent geometry. That said, I don't think SH are the best way to represent view dependency. Other approaches using MLP or deep features seem more promising.

slimbuck 3y ago

It's definitely possible to compress/palletise the SH bands (see https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-more-smaller/ https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-m... for one approach). We just found that splats encode some view-dependent shininess anyway (for example the guitar body https://playcanvas.com/model-viewer?load=https://code.playcanvas.com/viewer/guitar-cleaned.ply https://playcanvas.com/model-viewer?load=https://code.playca...) and so considered the extra data and rendering cost of SH not worth it for now. It would be quite straightforward to add SH to the PLY file and we'll definitely consider it in future!

ladberg 3y ago
Oh nice! I didn't realize those kinds of effects were possible without SH
jayd16 3y ago
Are the baked in reflections technically bug or is it fundamental in some way?
- taneq 3y ago
  They’re fundamental to the scene if you recite light transport and not just geometry. Gaussian splats are just a clever way to sample a lightfield and reflections are part of the lightfield.
  - jayd16 3y ago
    Well it's more that I'm asking if lighting data should live entirely in the spherical harmonics. It's certainly feasible and common in the game world. ...But maybe that doesn't make sense in the GS context.
crubier 3y ago
Yeah splats can encode view dependency without SH, but not with a correct geometry. E.g. often, you see reflections encoded in GS as just a 3d scene on the other side of the mirror. Other view dependency effect can make the surfaces "fluffy", etc. This is not great when you also try to get decent geometry. That said, I don't think SH are the best way to represent view dependency. Other approaches using MLP or deep features seem more promising.