6 ms·

Just yesterday I was literally musing to myself "I wonder if NeRFs would help with 3D object synthesis", and here we are. It's definitely a fun time to be invo

Just yesterday I was literally musing to myself "I wonder if NeRFs would help with 3D object synthesis", and here we are. It's definitely a fun time to be involved.

regegrt 4y ago
It's not based on the NeRF concept though, is it? Its outputs can provide the inputs for NeRF training, which is why they mention NeRFs. But it's not NeRF technology.
- deleted 4y ago
  [deleted]
noduerme 4y ago
it's actually a really fun time to know how to sculpt in ZBrush and print out models.
- nmfisher 4y ago
  If I had any artistic talent whatsoever, I'd probably agree with you!
  - noduerme 4y ago
    I won't lie... ZBrush is brutally hard. I got a subscription for work and only used it for one paid job, ever. But it's super satisfying if you just want to spend Sunday night making a clay elephant or rhinoceros, and drop $20 to have the file printed out and shipped to you by Thursday. I've fed lots of my sculpture renderings to Dali and gotten some pretty cool 2D results... but nothing nearly as cool as the little asymmetrical epoxy sculptures I can line up on the bookshelf...
popinman322 4y ago
NeRFs are a form of inverse renderer; this paper uses Score Jacobian Chaining[0] instead. Model reconstruction from NeRFs is also an active area of research. Check out the "Model Reconstruction" section of Awesome NeRF[1]. From the SJC paper: > We introduce a method that converts a pretrained 2D diffusion generative model on images into a 3D generative model of radiance fields, without requiring access to any 3D data. The key insight is to interpret diffusion models as function f with parameters θ, i.e., x = f (θ). Applying the chain rule through the Jacobian ∂x/∂θ converts a gradient on image x into a gradient on the parameter θ. > Our method uses differentiable rendering to aggregate 2D image gradients over multiple viewpoints into a 3D asset gradient, and lifts a generative model from 2D to 3D. We parameterize a 3D asset θ as a radiance field stored on voxels and choose f to be the volume rendering function. Interpretation: they take multiple input views, then optimize parameters (a voxel grid in this case) to a differentiable renderer (the volume rendering function for voxels) such that they can reproduce the input views. [0]: https://pals.ttic.edu/p/score-jacobian-chaining https://pals.ttic.edu/p/score-jacobian-chaining [1]: https://github.com/awesome-NeRF/awesome-NeRF https://github.com/awesome-NeRF/awesome-NeRF

Just yesterday I was literally musing to myself "I wonder if NeRFs would help with 3D object synthesis", and here we are. It's definitely a fun time to be involved.

regegrt 4y ago

It's not based on the NeRF concept though, is it? Its outputs can provide the inputs for NeRF training, which is why they mention NeRFs. But it's not NeRF technology.

deleted 4y ago
[deleted]

noduerme 4y ago

it's actually a really fun time to know how to sculpt in ZBrush and print out models.

nmfisher 4y ago
If I had any artistic talent whatsoever, I'd probably agree with you!
- noduerme 4y ago
  I won't lie... ZBrush is brutally hard. I got a subscription for work and only used it for one paid job, ever. But it's super satisfying if you just want to spend Sunday night making a clay elephant or rhinoceros, and drop $20 to have the file printed out and shipped to you by Thursday. I've fed lots of my sculpture renderings to Dali and gotten some pretty cool 2D results... but nothing nearly as cool as the little asymmetrical epoxy sculptures I can line up on the bookshelf...

popinman322 4y ago

NeRFs are a form of inverse renderer; this paper uses Score Jacobian Chaining[0] instead. Model reconstruction from NeRFs is also an active area of research. Check out the "Model Reconstruction" section of Awesome NeRF[1]. From the SJC paper: > We introduce a method that converts a pretrained 2D diffusion generative model on images into a 3D generative model of radiance fields, without requiring access to any 3D data. The key insight is to interpret diffusion models as function f with parameters θ, i.e., x = f (θ). Applying the chain rule through the Jacobian ∂x/∂θ converts a gradient on image x into a gradient on the parameter θ. > Our method uses differentiable rendering to aggregate 2D image gradients over multiple viewpoints into a 3D asset gradient, and lifts a generative model from 2D to 3D. We parameterize a 3D asset θ as a radiance field stored on voxels and choose f to be the volume rendering function. Interpretation: they take multiple input views, then optimize parameters (a voxel grid in this case) to a differentiable renderer (the volume rendering function for voxels) such that they can reproduce the input views. [0]: https://pals.ttic.edu/p/score-jacobian-chaining https://pals.ttic.edu/p/score-jacobian-chaining [1]: https://github.com/awesome-NeRF/awesome-NeRF https://github.com/awesome-NeRF/awesome-NeRF