3 ms·

I might be misunderstanding your use case, but are you trying to interpolate movement between frames (ie: are you trying to reconstruct a dynamic 4D scene or is

I might be misunderstanding your use case, but are you trying to interpolate movement between frames (ie: are you trying to reconstruct a dynamic 4D scene or is the capture fully static?) If you are trying to capture dynamics, most of the Nerfstudio methods are geared towards static captures and will give poor results for scenes with movement. There are many dynamic NeRF works out there - for example https://dynamic3dgaussians.github.io/ https://dynamic3dgaussians.github.io/ and https://github.com/andrewsonga/Total-Recon https://github.com/andrewsonga/Total-Recon both provide code if you want to play around. With that being said, robust 4D reconstruction is still very much an open research problem (especially when limited to monocular RGB data / casual phone captures). I'd expect a lot of movement in the space in the months/years to come!

xrd 2y ago
Really appreciate your response. I'm trying to recreate my experiences with claymation when I was a kid. I want to take a picture of an object, like a lego figure, and then move it slightly, take another picture, then move the figure again slightly. Once I have some frames, I want to use a NERFs to interpolate between those frames. When I was young and doing claymation, I would move the figure, shoot two frames, and do that 12 times per second of film. And, the lights I used were often so hot, that my clay would melt and alter the figure. It was a chore. I thought I could capture fewer in-between frames and let the NERF figure out the interpolation, and perhaps get some weird side effects. Especially if it hallucinates. I'm not sure if a NERF is the right approach, but it seems like a good starting point. Thank you.
- turkihaithem 2y ago
  You might also want to take a look at diffusion models: https://vidim-interpolation.github.io/ https://vidim-interpolation.github.io/ In terms of publicly available code, I think Stable Video Diffusion can do frame interpolation (https://stability.ai/news/introducing-stable-video-diffusion-api https://stability.ai/news/introducing-stable-video-diffusion...), but I haven't tried it myself.
  - xrd 2y ago
    That stuff is perfect. Thanks. I will definitely play with this.

xrd 2y ago

Really appreciate your response. I'm trying to recreate my experiences with claymation when I was a kid. I want to take a picture of an object, like a lego figure, and then move it slightly, take another picture, then move the figure again slightly. Once I have some frames, I want to use a NERFs to interpolate between those frames. When I was young and doing claymation, I would move the figure, shoot two frames, and do that 12 times per second of film. And, the lights I used were often so hot, that my clay would melt and alter the figure. It was a chore. I thought I could capture fewer in-between frames and let the NERF figure out the interpolation, and perhaps get some weird side effects. Especially if it hallucinates. I'm not sure if a NERF is the right approach, but it seems like a good starting point. Thank you.

turkihaithem 2y ago
You might also want to take a look at diffusion models: https://vidim-interpolation.github.io/ https://vidim-interpolation.github.io/ In terms of publicly available code, I think Stable Video Diffusion can do frame interpolation (https://stability.ai/news/introducing-stable-video-diffusion-api https://stability.ai/news/introducing-stable-video-diffusion...), but I haven't tried it myself.
- xrd 2y ago
  That stuff is perfect. Thanks. I will definitely play with this.