3 ms·

I mean, this is the parent posts point; the use case for a static photo or a static 3d nerf is pretty limited. With other structured data compositing and anima

I mean, this is the parent posts point; the use case for a static photo or a static 3d nerf is pretty limited. With other structured data compositing and animating is relatively trivial. It turns out that people have approached this problem before and you can composite nerf too (1) by sampling different functions over the volume. …but, let’s not pretend. The complaint is entirely valid. You’re taking a high resolution voxel grid and encoding it into a model. Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Practically speaking, the applications you can use this for are therefore reasonably limited right now. [1] - https://www.unite.ai/st-nerf-compositing-and-editing-for-video-synthesis/ https://www.unite.ai/st-nerf-compositing-and-editing-for-vid...

yorwba 5y ago
> Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Any image transformation you can do on voxels you can straightforwardly transfer to nerfs. Voxel data is just a lookup table from discrete positions to material properties like color and density. When you apply a transform, you change the inputs (e.g. multiplying them with a rotation matrix) or the outputs (e.g. changing the color). If you want to do the same thing with a nerf that maps continuous positions and directions to material properties like color and density, just transform the inputs or the outputs. The major difference is that with voxel data you can easily do output-modifying transformations directly on the stored representation, while for nerfs it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model.
- wokwokwok 5y ago
  > Any image transformation you can do on voxels you can straightforwardly transfer to nerfs No. > it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model. I think it’s a bit more complex than you imagine; it’s not “cheaper/not cheaper”; it’s literally the only way of doing it. If you have a transformation f(x) that takes a pixel array as in input and returns a pixel array as an output, that is a trivial transformation. If you have a transformation that takes a vector input f(x) and returns a pixel output, it’s seriously bloody hard to convert it to a “good” vector again. Consider taking a layered svg and applying a box blur. Now you want an svg again. It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg. Now you add the constraint in 3d; you can never have a full voxel representation in memory even temporarily because of memory constraints. At best you’re looking at applying voxel level transformations on the fly to render specific views, and then retrain those into a new nerf model. I think that counts as … not straightforward. Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time. You have to bake it into nerfs to use it at all.
  - yorwba 5y ago
    > Consider taking a layered svg and applying a box blur. > Now you want an svg again. > It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg. Nerfs are not svgs. Consider taking a nerf and applying a box blur. Easy, a box blur on voxel data takes multiple samples within a box and averages them together, so to do the same thing to a nerf just take multiple samples and average them together. That does get slower the more samples you need, but you never have to materialize a full voxel representation.
    - wokwokwok 5y ago
      You still have to process every voxel. > Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. > It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time.

yorwba 5y ago

> Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf. Any image transformation you can do on voxels you can straightforwardly transfer to nerfs. Voxel data is just a lookup table from discrete positions to material properties like color and density. When you apply a transform, you change the inputs (e.g. multiplying them with a rotation matrix) or the outputs (e.g. changing the color). If you want to do the same thing with a nerf that maps continuous positions and directions to material properties like color and density, just transform the inputs or the outputs. The major difference is that with voxel data you can easily do output-modifying transformations directly on the stored representation, while for nerfs it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model.

wokwokwok 5y ago
> Any image transformation you can do on voxels you can straightforwardly transfer to nerfs No. > it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model. I think it’s a bit more complex than you imagine; it’s not “cheaper/not cheaper”; it’s literally the only way of doing it. If you have a transformation f(x) that takes a pixel array as in input and returns a pixel array as an output, that is a trivial transformation. If you have a transformation that takes a vector input f(x) and returns a pixel output, it’s seriously bloody hard to convert it to a “good” vector again. Consider taking a layered svg and applying a box blur. Now you want an svg again. It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg. Now you add the constraint in 3d; you can never have a full voxel representation in memory even temporarily because of memory constraints. At best you’re looking at applying voxel level transformations on the fly to render specific views, and then retrain those into a new nerf model. I think that counts as … not straightforward. Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time. You have to bake it into nerfs to use it at all.
- yorwba 5y ago
  > Consider taking a layered svg and applying a box blur. > Now you want an svg again. > It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg. Nerfs are not svgs. Consider taking a nerf and applying a box blur. Easy, a box blur on voxel data takes multiple samples within a box and averages them together, so to do the same thing to a nerf just take multiple samples and average them together. That does get slower the more samples you need, but you never have to materialize a full voxel representation.
  - wokwokwok 5y ago
    You still have to process every voxel. > Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. > It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time.