5 ms·

Hoping for the day this technology makes it into gaming. Looks incredible.

by alphabetting 4y ago

TruthWillHurt 4y ago
My thoughts exactly. This is the future - both realistic and lazy (as a scene can simply be described to a proompt).
- underanalyzer 4y ago
  To clarify for people who don't follow NeRF techniques, this research is not prompt based. The algorithm is capturing the 3d scene from real life images. There is some super promising work in mixing NeRF based techniques with various generative models to create 3d objects from prompts but it doesn't seem close to creating anything of this kind of scale / detail yet. I do agree this is a future possibility though.
  - underanalyzer 4y ago
    I will admit I stand corrected about not being close: https://twitter.com/_akhaliq/status/1648848468234911754 https://twitter.com/_akhaliq/status/1648848468234911754
_0ffh 4y ago
The problem for gaming is, these scenes are completely static.
- alphabetting 4y ago
  True but even for static areas and backgrounds I feel like this detail and fidelity blows away what is used in current AAA games.
  - NoOneNearMe 4y ago
    I think a bigger problem is how much computational resources NERF models need. From the paper, "Our model, Mip-NeRF 360, and our 'mip-NeRF 360 + iNGP' baseline were all trained on 8 NVIDIA Tesla V100-SXM2-16GB GPUs." Even with all that, it takes almost an hour to train. A developer would need to do this for every scene, which may make it unfeasible for indies, those who would benefit the most from NERF in gaming.
    - piyh 4y ago
      The better question is how many person hours would it take to model a scene like that with alternative approaches, and can a NeRF be used to create outputs that could be used in a game engine?
      - deleted 4y ago
        [deleted]
  - mrguyorama 4y ago
    That's basically just FMV games all over again.
- spyder 4y ago
  Lot of game assets are static, and people are working on animated NERFs too. Also another necessary thing will be collisions and physics. example for character animation: https://zju3dv.github.io/animatable_nerf/ https://zju3dv.github.io/animatable_nerf/ example for the static world already working in Unreal engine real-time: https://www.youtube.com/watch?v=GjpzMDur7UY https://www.youtube.com/watch?v=GjpzMDur7UY diffusion models for NERF asset generation: https://sirwyver.github.io/DiffRF/ https://sirwyver.github.io/DiffRF/ https://dreamfusion3d.github.io/ https://dreamfusion3d.github.io/ Physics: https://www.youtube.com/watch?v=Md0PM-wv_Xg https://www.youtube.com/watch?v=Md0PM-wv_Xg https://www.youtube.com/watch?v=Eklh1pIAri0 https://www.youtube.com/watch?v=Eklh1pIAri0 and there are a lot more research being done on this, it's hard to keep up with all the things happening in AI

Hoping for the day this technology makes it into gaming. Looks incredible.

TruthWillHurt 4y ago

My thoughts exactly. This is the future - both realistic and lazy (as a scene can simply be described to a proompt).

underanalyzer 4y ago
To clarify for people who don't follow NeRF techniques, this research is not prompt based. The algorithm is capturing the 3d scene from real life images. There is some super promising work in mixing NeRF based techniques with various generative models to create 3d objects from prompts but it doesn't seem close to creating anything of this kind of scale / detail yet. I do agree this is a future possibility though.
- underanalyzer 4y ago
  I will admit I stand corrected about not being close: https://twitter.com/_akhaliq/status/1648848468234911754 https://twitter.com/_akhaliq/status/1648848468234911754

_0ffh 4y ago

The problem for gaming is, these scenes are completely static.

alphabetting 4y ago
True but even for static areas and backgrounds I feel like this detail and fidelity blows away what is used in current AAA games.
- NoOneNearMe 4y ago
  I think a bigger problem is how much computational resources NERF models need. From the paper, "Our model, Mip-NeRF 360, and our 'mip-NeRF 360 + iNGP' baseline were all trained on 8 NVIDIA Tesla V100-SXM2-16GB GPUs." Even with all that, it takes almost an hour to train. A developer would need to do this for every scene, which may make it unfeasible for indies, those who would benefit the most from NERF in gaming.
  - piyh 4y ago
    The better question is how many person hours would it take to model a scene like that with alternative approaches, and can a NeRF be used to create outputs that could be used in a game engine?
    - deleted 4y ago
      [deleted]
- mrguyorama 4y ago
  That's basically just FMV games all over again.
spyder 4y ago
Lot of game assets are static, and people are working on animated NERFs too. Also another necessary thing will be collisions and physics. example for character animation: https://zju3dv.github.io/animatable_nerf/ https://zju3dv.github.io/animatable_nerf/ example for the static world already working in Unreal engine real-time: https://www.youtube.com/watch?v=GjpzMDur7UY https://www.youtube.com/watch?v=GjpzMDur7UY diffusion models for NERF asset generation: https://sirwyver.github.io/DiffRF/ https://sirwyver.github.io/DiffRF/ https://dreamfusion3d.github.io/ https://dreamfusion3d.github.io/ Physics: https://www.youtube.com/watch?v=Md0PM-wv_Xg https://www.youtube.com/watch?v=Md0PM-wv_Xg https://www.youtube.com/watch?v=Eklh1pIAri0 https://www.youtube.com/watch?v=Eklh1pIAri0 and there are a lot more research being done on this, it's hard to keep up with all the things happening in AI