5 ms·

>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting

by daenz 5y ago

>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting 3D scene within tens of milliseconds. Generating the novel viewpoints is almost fast enough for VR, assuming you're tethered to a desktop computer with whatever GPUs they're using (probably the best setup possible). The holy grail (from my estimation) is getting both the training and the rendering to fit into a VR frame budget. They'll probably achieve it soon with some very clever techniques that only require differential re-training as the scene changes. The result will be a VR experience with live people and objects that feels photorealistic, because it essentially is based on real photos.

simsla 5y ago
> plus data on the camera angles they were taken from Doesn't seem like much of a stretch to determine the angles as well. E.g. a semi brute forced way with GANs
- c4wrd 5y ago
  I've spent a lot of time thinking about this (i.e. taking a video and creating a 3D scene) and I don't think that it is feasible in most cases to have good accuracy. If you need to infer the angle, you need make a lot of biased assumptions about things like velocity, position, etc., of the camera and even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time. Now I'm not saying it's not possible, but I'd believe that if you want an accurate 3D scene, you'd rather be spending your computation budget on things other than determining those angles when it can be simply be provided by hardware.
  - krasin 5y ago
    https://github.com/NVLabs/instant-ngp https://github.com/NVLabs/instant-ngp has a script that converts a video into frames and then uses COLMAP ([1]) to compute camera poses. You can then train a NeRF model within a few seconds. It all works pretty well. Trying it on your own video is pretty straightforward. 1. https://colmap.github.io/ https://colmap.github.io/
  - riotnrrd 5y ago
    You're far too pessimistic (or maybe you don't know the field well). The problem of estimating the relative poses of the cameras responsible for a set of photos is a long standing and essentially "solved" problem in computer vision. I say "solved" because there is still active research (increasing accuracy, faster, more robust, etc.) but there are decades-old, well known techniques that any dedicated programmer could implement in a week. If you're genuinely curious, look into structure from motion, visual odometry, or SLAM.
  - doliveira 5y ago
    > even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time Not really, with SLAM there are various algorithms to keep inaccuracy in check. Basically it works by a feedback loop of guessing an estimate for position and then updating it using landmarks.
- riotnrrd 5y ago
  You don't even need anything that fancy. Traditional structure-from-motion, or visual odometry gives accurate enough position estimations. If you want to experiment, take a bunch (~100) of photos of an object, and use COLMAP to generate the poses. COLMAP implements a global SfM technique, so it will be very accurate but very slow.
dexter89_kp3 5y ago
You could train the initial model in sim and do sim2real finetuning. Someone must be working on this already
hwers 5y ago
You can just make a mesh once you have the NeRF, which is plenty fast for VR. (What's important isn't new perspective generation but scene training. New perspective generation doesn't have to be real time just within a reasonable time to preprocess and then make a mesh.)
- andybak 5y ago
  The meshes I've seen from nerfs are pretty horrible and you lose many of the things that make nerfs interesting in the first place.
  - hwers 5y ago
    That doesn't make much sense. You should be able to get arbitrarily good meshes by generating more and more viewpoints.
    - andybak 5y ago
      OK. So that's using a render of a NERFfield as input into photogrammetry? Yeah - that might be more feasible. I was talking about directly generating a mesh from the NERF density itself (marching cubes). But NERFs can capture and render things that are very difficult subjects for photogrammetry: Fur, vegetation, reflective or transparent surfaces etc.

by daenz 5y ago

simsla 5y ago

> plus data on the camera angles they were taken from Doesn't seem like much of a stretch to determine the angles as well. E.g. a semi brute forced way with GANs

c4wrd 5y ago
I've spent a lot of time thinking about this (i.e. taking a video and creating a 3D scene) and I don't think that it is feasible in most cases to have good accuracy. If you need to infer the angle, you need make a lot of biased assumptions about things like velocity, position, etc., of the camera and even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time. Now I'm not saying it's not possible, but I'd believe that if you want an accurate 3D scene, you'd rather be spending your computation budget on things other than determining those angles when it can be simply be provided by hardware.
- krasin 5y ago
  https://github.com/NVLabs/instant-ngp https://github.com/NVLabs/instant-ngp has a script that converts a video into frames and then uses COLMAP ([1]) to compute camera poses. You can then train a NeRF model within a few seconds. It all works pretty well. Trying it on your own video is pretty straightforward. 1. https://colmap.github.io/ https://colmap.github.io/
- riotnrrd 5y ago
  You're far too pessimistic (or maybe you don't know the field well). The problem of estimating the relative poses of the cameras responsible for a set of photos is a long standing and essentially "solved" problem in computer vision. I say "solved" because there is still active research (increasing accuracy, faster, more robust, etc.) but there are decades-old, well known techniques that any dedicated programmer could implement in a week. If you're genuinely curious, look into structure from motion, visual odometry, or SLAM.
- doliveira 5y ago
  > even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time Not really, with SLAM there are various algorithms to keep inaccuracy in check. Basically it works by a feedback loop of guessing an estimate for position and then updating it using landmarks.
riotnrrd 5y ago
You don't even need anything that fancy. Traditional structure-from-motion, or visual odometry gives accurate enough position estimations. If you want to experiment, take a bunch (~100) of photos of an object, and use COLMAP to generate the poses. COLMAP implements a global SfM technique, so it will be very accurate but very slow.

dexter89_kp3 5y ago

You could train the initial model in sim and do sim2real finetuning. Someone must be working on this already

hwers 5y ago

You can just make a mesh once you have the NeRF, which is plenty fast for VR. (What's important isn't new perspective generation but scene training. New perspective generation doesn't have to be real time just within a reasonable time to preprocess and then make a mesh.)

andybak 5y ago
The meshes I've seen from nerfs are pretty horrible and you lose many of the things that make nerfs interesting in the first place.
- hwers 5y ago
  That doesn't make much sense. You should be able to get arbitrarily good meshes by generating more and more viewpoints.
  - andybak 5y ago
    OK. So that's using a render of a NERFfield as input into photogrammetry? Yeah - that might be more feasible. I was talking about directly generating a mesh from the NERF density itself (marching cubes). But NERFs can capture and render things that are very difficult subjects for photogrammetry: Fur, vegetation, reflective or transparent surfaces etc.