5 ms·

Can someone eli5 splatting? How does it differ to normal photogrammetry techniques

dagmx 2y ago
The core solve is similar but instead of generating a mesh, you generate a series of elipsoids that have color, transparency and some spectral radiance information. The idea is that by layering enough of these on top of each other, you can make a high quality visual representation not bounded by geometry. Think of them like brush strokes in mid air versus a sculpture.
- brcmthrowaway 2y ago
  Thats insane, Does hollywood use them? Or still use meshes?
  - corysama 2y ago
    Ideas around point-based rendering have been around for a long time. But, have not been widely used. Hollywood and games might use them for smoke, water, explosions, but rarely for solid objects. There has been a big explosion of interest in the area since this release of https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ which proposed a way to generate the point cloud data from photographs in a way that is procedurally a lot like photogrammetry. Hundreds of research projects have spawned off of that paper.
  - dagmx 2y ago
    Gaussian splats themselves aren’t very useful for Hollywood because they don’t have any issues with higher cost representations that are more accurate. That said, Gaussian splats are based on very old papers that do have use in Hollywood. The closest example is radiance caches for Renderman. This would store points in space with very similar information to a splat, then interpolate between them to provide things like bounce lighting. The first Pixar film to use this throughout was Up.
    - gessha 2y ago
      > higher cost representations that are more accurate What are some examples of those representations?
      - dagmx 2y ago
        Just standard geometry and materials that they’re willing to spend the time properly constructing and path tracing. Nothing fancy. They just have a higher up front cost to make than Gaussian splats but are infinitely more directable. Ultimately the ability to control everything at a micro and macro level is what film wants, not cheap rendering.
zanecodes 2y ago
From my understanding, traditional photogrammetry typically generates 3d point clouds from image pixels by correlating visual features between images with known camera parameters, allowing the camera pose of each image to be estimated in a shared coordinate space. These point clouds are postprocessed to estimate closed surfaces which can then be converted into textured triangle meshes to be rendered using traditional 3d rasterization techniques. Gaussian splatting represents a scene a cloud of 3d gaussian ellipsoids, with direction-dependent color components (usually represented using spherical harmonics) to deal with effects like reflections. The "Gaussian" part is important, because gaussian distributions are easy to differentiate, making it possible (and fast) to optimize the positions, sizes, orientations, and colors of a collection of Gaussian splats to minimize the difference between the input photos and the rendered scene. This optimization is usually done by starting with the same 3d point clouds and camera poses estimated using the same or similar tools as traditional photogrammetry (e.g. COLMAP), and using this point cloud to place and color your initial Gaussian splats. One of the key insights in the original Gaussian splatting paper was the use of some heuristics to determine when to split a splat into smaller ones to provide higher detail over a given area, and when to combine splats into larger ones to cover uniform/low detail areas. The nature of Gaussian splats being essentially fancy point clouds means that they can't currently be easily integrated into existing 3d scene manipulation pipelines, although this is rapidly changing as they gain popularity, and tools to convert them into textured meshes and estimate material properties like albedo, reflectance, and so on do exist.
nick238 2y ago
Corridor Digital has a video where they showed some of the limitations of photomapping vs. splats: https://youtu.be/GaGcLhhhbDs?si=rJVxF8yNwwbfLBw9&t=306 https://youtu.be/GaGcLhhhbDs?si=rJVxF8yNwwbfLBw9&t=306 and there's an earlier one where they broke it down a bit more https://www.youtube.com/watch?v=YX5AoaWrowY https://www.youtube.com/watch?v=YX5AoaWrowY

The core solve is similar but instead of generating a mesh, you generate a series of elipsoids that have color, transparency and some spectral radiance information. The idea is that by layering enough of these on top of each other, you can make a high quality visual representation not bounded by geometry. Think of them like brush strokes in mid air versus a sculpture.

brcmthrowaway 2y ago
Thats insane, Does hollywood use them? Or still use meshes?
- corysama 2y ago
  Ideas around point-based rendering have been around for a long time. But, have not been widely used. Hollywood and games might use them for smoke, water, explosions, but rarely for solid objects. There has been a big explosion of interest in the area since this release of https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ which proposed a way to generate the point cloud data from photographs in a way that is procedurally a lot like photogrammetry. Hundreds of research projects have spawned off of that paper.
- dagmx 2y ago
  Gaussian splats themselves aren’t very useful for Hollywood because they don’t have any issues with higher cost representations that are more accurate. That said, Gaussian splats are based on very old papers that do have use in Hollywood. The closest example is radiance caches for Renderman. This would store points in space with very similar information to a splat, then interpolate between them to provide things like bounce lighting. The first Pixar film to use this throughout was Up.
  - gessha 2y ago
    > higher cost representations that are more accurate What are some examples of those representations?
    - dagmx 2y ago
      Just standard geometry and materials that they’re willing to spend the time properly constructing and path tracing. Nothing fancy. They just have a higher up front cost to make than Gaussian splats but are infinitely more directable. Ultimately the ability to control everything at a micro and macro level is what film wants, not cheap rendering.

zanecodes 2y ago

From my understanding, traditional photogrammetry typically generates 3d point clouds from image pixels by correlating visual features between images with known camera parameters, allowing the camera pose of each image to be estimated in a shared coordinate space. These point clouds are postprocessed to estimate closed surfaces which can then be converted into textured triangle meshes to be rendered using traditional 3d rasterization techniques. Gaussian splatting represents a scene a cloud of 3d gaussian ellipsoids, with direction-dependent color components (usually represented using spherical harmonics) to deal with effects like reflections. The "Gaussian" part is important, because gaussian distributions are easy to differentiate, making it possible (and fast) to optimize the positions, sizes, orientations, and colors of a collection of Gaussian splats to minimize the difference between the input photos and the rendered scene. This optimization is usually done by starting with the same 3d point clouds and camera poses estimated using the same or similar tools as traditional photogrammetry (e.g. COLMAP), and using this point cloud to place and color your initial Gaussian splats. One of the key insights in the original Gaussian splatting paper was the use of some heuristics to determine when to split a splat into smaller ones to provide higher detail over a given area, and when to combine splats into larger ones to cover uniform/low detail areas. The nature of Gaussian splats being essentially fancy point clouds means that they can't currently be easily integrated into existing 3d scene manipulation pipelines, although this is rapidly changing as they gain popularity, and tools to convert them into textured meshes and estimate material properties like albedo, reflectance, and so on do exist.

nick238 2y ago

Corridor Digital has a video where they showed some of the limitations of photomapping vs. splats: https://youtu.be/GaGcLhhhbDs?si=rJVxF8yNwwbfLBw9&t=306 https://youtu.be/GaGcLhhhbDs?si=rJVxF8yNwwbfLBw9&t=306 and there's an earlier one where they broke it down a bit more https://www.youtube.com/watch?v=YX5AoaWrowY https://www.youtube.com/watch?v=YX5AoaWrowY