5 ms·

Except the plenoxel representation is also 2-3 orders of magnitude larger than an MLP-based NeRF. It’s not very surprising that a sparse voxel representation ca

Except the plenoxel representation is also 2-3 orders of magnitude larger than an MLP-based NeRF. It’s not very surprising that a sparse voxel representation can capture a plenoptic function. Representation of volumetric video further presses the size disadvantages of voxel based techniques. The reasons for using deep learning function approximators are manifold, for instance in RL, state and state-action spaces quickly become too large for tabular methods. Using grids or tables also basically closes off the opportunities for exploiting meta-learning and analysis-by-synthesis. Plenoxels also rely on explicit specification of a known grid structure, where the HyperNeRF method can learn latent parametric manifolds and handle dynamic objects with changing topologies.

3dthrow 5y ago
An MLP-based NeRF actually has a comparable number of parameters to plenoxels (it's not 2-3 orders of magnitude smaller). The original NeRF is 8 dense layers with 256 channels and then this one adds another network with 6 dense layers with 64 channels, so roughly speaking 7x256x256 + 5x64x64. And remember that the voxel grid is sparse, though we don't get exact numbers here. We shouldn't be miserly with our megabytes in 2021. What concerns me is how HyperNeRF requires 64 hours of training time on 4 TPU v4s; if you want to use this for communication or entertainment, it's light-years away from interactive. Extending plenoxels to support dynamic objects would be great future work.
- dougabug 5y ago
  You can’t really compare HyperNeRFs to Plenoxels because the former does so much more. Vanilla NeRF has just under half a million weights (if we ignore the coarse network), although more recent work has shown comparable results with only four layers. To compare apples-to-apples, we should probably also apply the (hand-tuned) TVD regularization used in plenoxels. Plenoxels store 28 parameters per grid point, so at a 256^3 grid size this would be just under half a billion parameters. You don’t know which voxels you can prune at the outset (btw, DNNs can be pruned as well), but for real scenes they show sparsity reaching ~5% after 12.8k iterations. For a real 360 deg scene, because the voxelized volume is quite bounded, you also need to consider the 64 layer multi-sphere image required to capture the background. Basically, you need a fairly large GPU card to train plenoxels on any kind of real scene. The option to make this space-compute tradeoff is still useful.

3dthrow 5y ago

An MLP-based NeRF actually has a comparable number of parameters to plenoxels (it's not 2-3 orders of magnitude smaller). The original NeRF is 8 dense layers with 256 channels and then this one adds another network with 6 dense layers with 64 channels, so roughly speaking 7x256x256 + 5x64x64. And remember that the voxel grid is sparse, though we don't get exact numbers here. We shouldn't be miserly with our megabytes in 2021. What concerns me is how HyperNeRF requires 64 hours of training time on 4 TPU v4s; if you want to use this for communication or entertainment, it's light-years away from interactive. Extending plenoxels to support dynamic objects would be great future work.

dougabug 5y ago
You can’t really compare HyperNeRFs to Plenoxels because the former does so much more. Vanilla NeRF has just under half a million weights (if we ignore the coarse network), although more recent work has shown comparable results with only four layers. To compare apples-to-apples, we should probably also apply the (hand-tuned) TVD regularization used in plenoxels. Plenoxels store 28 parameters per grid point, so at a 256^3 grid size this would be just under half a billion parameters. You don’t know which voxels you can prune at the outset (btw, DNNs can be pruned as well), but for real scenes they show sparsity reaching ~5% after 12.8k iterations. For a real 360 deg scene, because the voxelized volume is quite bounded, you also need to consider the 64 layer multi-sphere image required to capture the background. Basically, you need a fairly large GPU card to train plenoxels on any kind of real scene. The option to make this space-compute tradeoff is still useful.