4 ms·

Since neural networks are universal, we can often solve problems we don't know exactly how to solve with neural networks. NeRF is a great example. But once solv

Since neural networks are universal, we can often solve problems we don't know exactly how to solve with neural networks. NeRF is a great example. But once solved, we should try to reverse engineer the solution and optimize. https://arxiv.org/abs/2112.05131 https://arxiv.org/abs/2112.05131 did such work and found you can get NeRF quality without any neural network at all and as a result 100x faster.

echelon 5y ago
> But once solved, we should try to reverse engineer the solution and optimize. This is a great point. Does anybody know of any papers that explore non-neural text to speech / voice synthesis that achieve better than parametric quality?
exikyut 5y ago
Demo videos and overview: https://alexyu.net/plenoxels/ https://alexyu.net/plenoxels/
orbital-decay 5y ago
NeRF triggered the development of a number of improved methods, though. There's DONeRF [1] that is built upon NNs and is currently faster than comparable solutions (the field evolves fast so I may be wrong). [1] https://github.com/facebookresearch/DONERF https://github.com/facebookresearch/DONERF
- blurbleblurble 5y ago
  Yeah there are dozens of spinoffs inspired by NeRF, it's kind of amazing
- deleted 5y ago
  [deleted]
dougabug 5y ago
Except the plenoxel representation is also 2-3 orders of magnitude larger than an MLP-based NeRF. It’s not very surprising that a sparse voxel representation can capture a plenoptic function. Representation of volumetric video further presses the size disadvantages of voxel based techniques. The reasons for using deep learning function approximators are manifold, for instance in RL, state and state-action spaces quickly become too large for tabular methods. Using grids or tables also basically closes off the opportunities for exploiting meta-learning and analysis-by-synthesis. Plenoxels also rely on explicit specification of a known grid structure, where the HyperNeRF method can learn latent parametric manifolds and handle dynamic objects with changing topologies.
- 3dthrow 5y ago
  An MLP-based NeRF actually has a comparable number of parameters to plenoxels (it's not 2-3 orders of magnitude smaller). The original NeRF is 8 dense layers with 256 channels and then this one adds another network with 6 dense layers with 64 channels, so roughly speaking 7x256x256 + 5x64x64. And remember that the voxel grid is sparse, though we don't get exact numbers here. We shouldn't be miserly with our megabytes in 2021. What concerns me is how HyperNeRF requires 64 hours of training time on 4 TPU v4s; if you want to use this for communication or entertainment, it's light-years away from interactive. Extending plenoxels to support dynamic objects would be great future work.
  - dougabug 5y ago
    You can’t really compare HyperNeRFs to Plenoxels because the former does so much more. Vanilla NeRF has just under half a million weights (if we ignore the coarse network), although more recent work has shown comparable results with only four layers. To compare apples-to-apples, we should probably also apply the (hand-tuned) TVD regularization used in plenoxels. Plenoxels store 28 parameters per grid point, so at a 256^3 grid size this would be just under half a billion parameters. You don’t know which voxels you can prune at the outset (btw, DNNs can be pruned as well), but for real scenes they show sparsity reaching ~5% after 12.8k iterations. For a real 360 deg scene, because the voxelized volume is quite bounded, you also need to consider the 64 layer multi-sphere image required to capture the background. Basically, you need a fairly large GPU card to train plenoxels on any kind of real scene. The option to make this space-compute tradeoff is still useful.
blurbleblurble 5y ago
Awesome! I'd love to see someone do this with light field networks: https://www.vincentsitzmann.com/lfns/ https://www.vincentsitzmann.com/lfns/ By the way, in the plenoxel video they say "the key component is the differentiable volumetric rendering, not the neural network". After watching the fractal community struggle to come up with good distance estimators for years, this makes so much sense. In the end, automatic differentiation came out to be one of the most solid methods for coming up with distance estimators for an arbitrary fractal formula.

echelon 5y ago

> But once solved, we should try to reverse engineer the solution and optimize. This is a great point. Does anybody know of any papers that explore non-neural text to speech / voice synthesis that achieve better than parametric quality?

exikyut 5y ago

Demo videos and overview: https://alexyu.net/plenoxels/ https://alexyu.net/plenoxels/

orbital-decay 5y ago

NeRF triggered the development of a number of improved methods, though. There's DONeRF [1] that is built upon NNs and is currently faster than comparable solutions (the field evolves fast so I may be wrong). [1] https://github.com/facebookresearch/DONERF https://github.com/facebookresearch/DONERF

blurbleblurble 5y ago
Yeah there are dozens of spinoffs inspired by NeRF, it's kind of amazing
deleted 5y ago
[deleted]

dougabug 5y ago

Except the plenoxel representation is also 2-3 orders of magnitude larger than an MLP-based NeRF. It’s not very surprising that a sparse voxel representation can capture a plenoptic function. Representation of volumetric video further presses the size disadvantages of voxel based techniques. The reasons for using deep learning function approximators are manifold, for instance in RL, state and state-action spaces quickly become too large for tabular methods. Using grids or tables also basically closes off the opportunities for exploiting meta-learning and analysis-by-synthesis. Plenoxels also rely on explicit specification of a known grid structure, where the HyperNeRF method can learn latent parametric manifolds and handle dynamic objects with changing topologies.

3dthrow 5y ago
An MLP-based NeRF actually has a comparable number of parameters to plenoxels (it's not 2-3 orders of magnitude smaller). The original NeRF is 8 dense layers with 256 channels and then this one adds another network with 6 dense layers with 64 channels, so roughly speaking 7x256x256 + 5x64x64. And remember that the voxel grid is sparse, though we don't get exact numbers here. We shouldn't be miserly with our megabytes in 2021. What concerns me is how HyperNeRF requires 64 hours of training time on 4 TPU v4s; if you want to use this for communication or entertainment, it's light-years away from interactive. Extending plenoxels to support dynamic objects would be great future work.
- dougabug 5y ago
  You can’t really compare HyperNeRFs to Plenoxels because the former does so much more. Vanilla NeRF has just under half a million weights (if we ignore the coarse network), although more recent work has shown comparable results with only four layers. To compare apples-to-apples, we should probably also apply the (hand-tuned) TVD regularization used in plenoxels. Plenoxels store 28 parameters per grid point, so at a 256^3 grid size this would be just under half a billion parameters. You don’t know which voxels you can prune at the outset (btw, DNNs can be pruned as well), but for real scenes they show sparsity reaching ~5% after 12.8k iterations. For a real 360 deg scene, because the voxelized volume is quite bounded, you also need to consider the 64 layer multi-sphere image required to capture the background. Basically, you need a fairly large GPU card to train plenoxels on any kind of real scene. The option to make this space-compute tradeoff is still useful.

blurbleblurble 5y ago

Awesome! I'd love to see someone do this with light field networks: https://www.vincentsitzmann.com/lfns/ https://www.vincentsitzmann.com/lfns/ By the way, in the plenoxel video they say "the key component is the differentiable volumetric rendering, not the neural network". After watching the fractal community struggle to come up with good distance estimators for years, this makes so much sense. In the end, automatic differentiation came out to be one of the most solid methods for coming up with distance estimators for an arbitrary fractal formula.