3 ms·

Excited to see what license this is released under. Would love to see some open source games using this.

jsheard 3y ago
Performance aside, someone needs to figure out a generalizable way to make the scenes dynamic before it will really be usable for games. History is littered with alternatives to triangles meshes that looked promising until we realised there's no efficient way to animate them.
- 999900000999 3y ago
  Can you explain what a dynamic is ? I was more thinking you'd run this tool, and then have an algorithm convert it( bake the mesh).
  - lawlessone 3y ago
    They probably mean animated, changeable etc. Like movement, or changes in lighting.
- CuriouslyC 3y ago
  Even if this doesn't replace triangles everywhere, I'm guessing it's still going to be the easiest way to generate a large volume of static art assets, which means we will see hybrid rendering pipelines.
  - jsheard 3y ago
    AIUI these algorithms currently bake all of the lighting into the surface colors statically, which mostly works if the entire scene is constructed as one giant blob where nothing moves but if you wanted to render an individual NeRF asset inside an otherwise standard triangle-based pipeline then it would need to be more adaptable than that. Even if the asset itself isn't animated it would need to adapt to the local lighting at the bare minimum, which I haven't seen anyone tackle yet, the focus has been on the rendering-one-giant-static-blob problem. For hybrid pipelines to work the splatting algorithm would probably need to output the standard G-Buffer channels (unlit surface color, normal, roughness, etc) which can then go through the same lighting pass as the triangle-based assets, rather than the splatting algorithm trying to infer lighting by itself and inevitably getting a result that's inconsistent with how the triangle-based assets are lit. Think of those old cartoons where you could always tell when part of the scenery was going to move because the animation cel would stick out like a sore thumb against the painted background, that's the kind of illusion break you would get if the lighting isn't consistent.
    - somethingsome 3y ago
      For NeRF this problems exists. However, in the past it was already solved for gaussian splatting. Usually you define a normal field over the (2D) splat, This allows you to have phong shading at least. It is not too difficult to go to a 2D normal field over the 3D gaussians..
KaiserPro 3y ago
Thats super hard as this is basically a very large pointcloud with oddly shaped points. The objects represented in the point cloud have no inherent metadata embedded (ie its a chair, table, person etc) so any kind of interaction is super hard. Its not impossible, but currently not practical. More over its not that optimised for real-time rendering. Yes, a lot of points have been pruned, but its far more optimal to have lower resolution meshes

Performance aside, someone needs to figure out a generalizable way to make the scenes dynamic before it will really be usable for games. History is littered with alternatives to triangles meshes that looked promising until we realised there's no efficient way to animate them.

999900000999 3y ago
Can you explain what a dynamic is ? I was more thinking you'd run this tool, and then have an algorithm convert it( bake the mesh).
- lawlessone 3y ago
  They probably mean animated, changeable etc. Like movement, or changes in lighting.
CuriouslyC 3y ago
Even if this doesn't replace triangles everywhere, I'm guessing it's still going to be the easiest way to generate a large volume of static art assets, which means we will see hybrid rendering pipelines.
- jsheard 3y ago
  AIUI these algorithms currently bake all of the lighting into the surface colors statically, which mostly works if the entire scene is constructed as one giant blob where nothing moves but if you wanted to render an individual NeRF asset inside an otherwise standard triangle-based pipeline then it would need to be more adaptable than that. Even if the asset itself isn't animated it would need to adapt to the local lighting at the bare minimum, which I haven't seen anyone tackle yet, the focus has been on the rendering-one-giant-static-blob problem. For hybrid pipelines to work the splatting algorithm would probably need to output the standard G-Buffer channels (unlit surface color, normal, roughness, etc) which can then go through the same lighting pass as the triangle-based assets, rather than the splatting algorithm trying to infer lighting by itself and inevitably getting a result that's inconsistent with how the triangle-based assets are lit. Think of those old cartoons where you could always tell when part of the scenery was going to move because the animation cel would stick out like a sore thumb against the painted background, that's the kind of illusion break you would get if the lighting isn't consistent.
  - somethingsome 3y ago
    For NeRF this problems exists. However, in the past it was already solved for gaussian splatting. Usually you define a normal field over the (2D) splat, This allows you to have phong shading at least. It is not too difficult to go to a 2D normal field over the 3D gaussians..

KaiserPro 3y ago

Thats super hard as this is basically a very large pointcloud with oddly shaped points. The objects represented in the point cloud have no inherent metadata embedded (ie its a chair, table, person etc) so any kind of interaction is super hard. Its not impossible, but currently not practical. More over its not that optimised for real-time rendering. Yes, a lot of points have been pruned, but its far more optimal to have lower resolution meshes