6 ms·

Instant neural graphics primitives with a multiresolution hash encoding

179 points by ath92 5y ago

ath92 5y ago
For some additional context, when the original NeRF paper (https://arxiv.org/pdf/2003.08934.pdf https://arxiv.org/pdf/2003.08934.pdf) was published 2 years ago, it reportedly took at least 12 hours (depending on hardware used of course) to train on the scene with the bulldozer. This has now been reduced to about 5 seconds (!), with realtime rendering of the result.
- hwers 5y ago
  The gigapixel example could be done with fourier features which takes about a few minutes to train (on colab-like resources). Definitely still a huge improvement though (and based on more clever hashing techniques than optimization).
WithinReason 5y ago
Goodbye polygons, hello neural networks?
- aappleby 5y ago
  More like "run this low-quality polygon and raytracing renderer at 320x240 @20 fps, upscale to 4k120 with acceptable quality".
- aantix 5y ago
  I’ve seen the GTA demo. Are there any commercial games currently doing this?
  - JayStavis 5y ago
    Neural rendering? I doubt it. Check out deep learning super sampling though (DLSS) from NVIDIA, which has to be plumbed into the game itself to enable. https://www.nvidia.com/en-us/geforce/technologies/dlss/ https://www.nvidia.com/en-us/geforce/technologies/dlss/
- ReactiveJelly 5y ago
  Not sure yet. This is probably going to fight virtual geometry tech like Unreal's Nanite, which is still using triangles but using clever automated LoD and GPGPU rasterization so that rendering e.g. 20 million pixel-sized triangles is fast and looks just as good as rendering a trillion triangles. (normally very small or thin triangles are a pathological case for hardware rasterizers)
- EZ-Cheeze 5y ago
  I think meshes and textures can be replaced by intelligently shifting, raytraced billions of platonic solids http://zeroprecedent.com/platonic http://zeroprecedent.com/platonic
  - ReactiveJelly 5y ago
    Why not billions of triangles? Unreal is betting on Nanite because triangles have so many nice properties in addition to having the whole art pipeline already set up. (I could not get the URL to load. Maybe HN hugged it)
    - EZ-Cheeze 5y ago
      Triangles have no volume and no diffraction occurs inside them as it does with Platonic solids. The idea is that real-time raytracing will allow complex variations and interactions of "Platonic dust particles" and the rays bouncing and refracting between and in them. It would be a more expressive "clay" for the AI to tinker with than triangles - the orientation/color/transparency changes of each solid will be able to elicit more visual effects than doing it with flat triangles.I got banned from Eleuther discord today The One#3740
  - blovescoffee 5y ago
    This person just invented a very handwavey GAN
    - EZ-Cheeze 5y ago
      Effective representation changes everything Like hashtables
deleted 5y ago
[deleted]
The_rationalist 5y ago
MauranKilom 5y ago
My summary (from someone who is not in the field but likes backpropagation): The core idea behind this type of approach ("parametric encoding") is that you learn a scene as some spatial data + a (small) neural network. For example, a 128^3 grid of data values and a 10k parameter model. In the forward pass you feed whatever data is at the voxel(s) in question to the network, and the backward pass updates both the network and the same voxel(s). The innovation in this paper is in how the spatial data is represented. Prior work includes dense grids, multi-resolution grids and octrees to name some - but all of them are either GPU-unfriendly or waste parameters on empty space. They figured that they can just hash the coordinates and use them directly as an index into a data array (edit: A multi-resolution stack of data arrays - sorry for not getting this right initially), with hash collisions left to the network to figure out (it's gonna figure out whether there's a collision on fine layer through info from the coarser ones, I guess). (Relatively) few parameters + GPU-friendly data structure = fast training. Tempted to try and implement this myself...
- amelius 5y ago
  Isn't the effect of hashing the same as sampling more coarsely?
  - MauranKilom 5y ago
    I think the key here is that e.g. surface information only grows at O(N²) rate whereas number of grid points scales as O(N³). The hash function approach means your arrays will be filled with detailed information densely, whereas sampling coarsely would still leave most of the array with "nothing here" information. Your comment made me realize that I forgot to mention the multi-resolution aspect of their hash encoding (there are several data arrays corresponding to different resolutions - coarse ones are 1:1 indexed but finer ones have hash collisions for the network to deal with). It's in the title, but I should still include it.
- ReactiveJelly 5y ago
  If it's so fast, I'd like to see it working on a smaller scale on a CPU. Every new deep learning paper that comes out, I'm disappointed that it needs... - A $500-$1,000 GPU - A huge proprietary NVidia driver - Some odd language or language extensions, usually CUDA - Python
tmilard 5y ago
Magic, impressive. I have no word.
michaelgiba 5y ago
Wow this is a game changer / landmark advancement

179 points by ath92 5y ago

For some additional context, when the original NeRF paper (https://arxiv.org/pdf/2003.08934.pdf https://arxiv.org/pdf/2003.08934.pdf) was published 2 years ago, it reportedly took at least 12 hours (depending on hardware used of course) to train on the scene with the bulldozer. This has now been reduced to about 5 seconds (!), with realtime rendering of the result.

hwers 5y ago
The gigapixel example could be done with fourier features which takes about a few minutes to train (on colab-like resources). Definitely still a huge improvement though (and based on more clever hashing techniques than optimization).

WithinReason 5y ago

Goodbye polygons, hello neural networks?

aappleby 5y ago
More like "run this low-quality polygon and raytracing renderer at 320x240 @20 fps, upscale to 4k120 with acceptable quality".
aantix 5y ago
I’ve seen the GTA demo. Are there any commercial games currently doing this?
- JayStavis 5y ago
  Neural rendering? I doubt it. Check out deep learning super sampling though (DLSS) from NVIDIA, which has to be plumbed into the game itself to enable. https://www.nvidia.com/en-us/geforce/technologies/dlss/ https://www.nvidia.com/en-us/geforce/technologies/dlss/
ReactiveJelly 5y ago
Not sure yet. This is probably going to fight virtual geometry tech like Unreal's Nanite, which is still using triangles but using clever automated LoD and GPGPU rasterization so that rendering e.g. 20 million pixel-sized triangles is fast and looks just as good as rendering a trillion triangles. (normally very small or thin triangles are a pathological case for hardware rasterizers)
EZ-Cheeze 5y ago
I think meshes and textures can be replaced by intelligently shifting, raytraced billions of platonic solids http://zeroprecedent.com/platonic http://zeroprecedent.com/platonic
- ReactiveJelly 5y ago
  Why not billions of triangles? Unreal is betting on Nanite because triangles have so many nice properties in addition to having the whole art pipeline already set up. (I could not get the URL to load. Maybe HN hugged it)
  - EZ-Cheeze 5y ago
    Triangles have no volume and no diffraction occurs inside them as it does with Platonic solids. The idea is that real-time raytracing will allow complex variations and interactions of "Platonic dust particles" and the rays bouncing and refracting between and in them. It would be a more expressive "clay" for the AI to tinker with than triangles - the orientation/color/transparency changes of each solid will be able to elicit more visual effects than doing it with flat triangles.I got banned from Eleuther discord today The One#3740
- blovescoffee 5y ago
  This person just invented a very handwavey GAN
  - EZ-Cheeze 5y ago
    Effective representation changes everything Like hashtables

deleted 5y ago

[deleted]

The_rationalist 5y ago

MauranKilom 5y ago

My summary (from someone who is not in the field but likes backpropagation): The core idea behind this type of approach ("parametric encoding") is that you learn a scene as some spatial data + a (small) neural network. For example, a 128^3 grid of data values and a 10k parameter model. In the forward pass you feed whatever data is at the voxel(s) in question to the network, and the backward pass updates both the network and the same voxel(s). The innovation in this paper is in how the spatial data is represented. Prior work includes dense grids, multi-resolution grids and octrees to name some - but all of them are either GPU-unfriendly or waste parameters on empty space. They figured that they can just hash the coordinates and use them directly as an index into a data array (edit: A multi-resolution stack of data arrays - sorry for not getting this right initially), with hash collisions left to the network to figure out (it's gonna figure out whether there's a collision on fine layer through info from the coarser ones, I guess). (Relatively) few parameters + GPU-friendly data structure = fast training. Tempted to try and implement this myself...

amelius 5y ago
Isn't the effect of hashing the same as sampling more coarsely?
- MauranKilom 5y ago
  I think the key here is that e.g. surface information only grows at O(N²) rate whereas number of grid points scales as O(N³). The hash function approach means your arrays will be filled with detailed information densely, whereas sampling coarsely would still leave most of the array with "nothing here" information. Your comment made me realize that I forgot to mention the multi-resolution aspect of their hash encoding (there are several data arrays corresponding to different resolutions - coarse ones are 1:1 indexed but finer ones have hash collisions for the network to deal with). It's in the title, but I should still include it.
ReactiveJelly 5y ago
If it's so fast, I'd like to see it working on a smaller scale on a CPU. Every new deep learning paper that comes out, I'm disappointed that it needs... - A $500-$1,000 GPU - A huge proprietary NVidia driver - Some odd language or language extensions, usually CUDA - Python

tmilard 5y ago

Magic, impressive. I have no word.

michaelgiba 5y ago

Wow this is a game changer / landmark advancement