5 ms·

I've been fascinated by NeRFs for a few years. But, there are really no viable models that run on consumer hardware like llama or stable diffusion. Or, am I w

by xrd 3y ago

I've been fascinated by NeRFs for a few years. But, there are really no viable models that run on consumer hardware like llama or stable diffusion. Or, am I wrong? NeRF Studio seems promising but never works on my 6GB nvidia. I would really like to find a way to interpolate between two images using a NeRF (get the hallucination of the "image in between"). Is there such a thing out there?

dheera 3y ago
I don't know about 6GB but if you have a 12GB or 16GB card you should be able to run the vast majority of NeRF work out there, most of it is designed to run on a single GPU.
- xrd 3y ago
  Do you know if there are any NERFs that can be run in command line mode, where you can see an intermediate image by giving it two or more images? I'm less interested in video than in a single image result.
riotnrrd 3y ago
I mostly have experience with Instant NGP, and it should work on older consumer NVIDIA cards. Their github page calls out Pascal cards as working, for example. 6 GB isn't much memory, though, so you may be limited in final resolution of the latent model and thus of the output.
cameronfraser 3y ago
nerfstudio should work on 6gb of vram, i've used the nerfacto model extensively on a laptop with that constraint. Try reducing the num-nerf-samples-per-ray param and downscaling images
vanderZwan 3y ago
It feels like optimizing NeRFs to the point where they are usable by consumer hardware while producing decent results is the main thing everyone is working on, so give it a few more years
dTal 3y ago
6GB of VRAM! Luxury! I managed to reconstruct the lego scene using TensoRF[0] (not strictly NeRF, but a similar approach with similar results) last night on my nvidia-equipped T480 (2gb of VRAM)[1], so it's possible. >But, there are really no viable models that run on consumer hardware like llama or stable diffusion. This isn't a strictly accurate framing, there is no pre-trained "model" that you "run" inference on like with Llama or Stable Diffusion. You are training the model, from scratch, on each new scene. The viability of this on a given GPU depends on the combined size of the input + output, i.e. the resolution and number of input images and the resolution and compactness of the resulting data structure. There's nothing in principle preventing you from training tiny low-res nerfs from tiny low-res images, except that all the researchers in this space are working with standard datasets of a standard size on big beefy machines and their code is full of magic numbers. Also, many of the improvements on the original NeRF achieve their speedup through much hungrier data structures (voxels, multiresolution, etc). TensoRF appears to have a very compact scene representation (like the original NeRF) and very fast training (like instant-ngp) so it seems to be a sweet spot for low-end hardware - at any rate, it's the first thing I managed to get working on this laptop. The main downside seems to be that inference (generating new images) is quite slow, at about 14 seconds. [0] https://github.com/apchenstu/TensoRF/ https://github.com/apchenstu/TensoRF/ - it's apparently included in nerfstudio as well [1] batch_size = 512 in configs/lego.txt (with your 6gb you'd get away with 2048) and compute_extra_metrics=False wherever it appears in train.py
somethingsome 3y ago
I obtain high quality without neural networks or specific hardware, but I don't have the traction of high end labs. It requires a bit more of manual work, but not that much

by xrd 3y ago

dheera 3y ago

I don't know about 6GB but if you have a 12GB or 16GB card you should be able to run the vast majority of NeRF work out there, most of it is designed to run on a single GPU.

xrd 3y ago
Do you know if there are any NERFs that can be run in command line mode, where you can see an intermediate image by giving it two or more images? I'm less interested in video than in a single image result.

riotnrrd 3y ago

I mostly have experience with Instant NGP, and it should work on older consumer NVIDIA cards. Their github page calls out Pascal cards as working, for example. 6 GB isn't much memory, though, so you may be limited in final resolution of the latent model and thus of the output.

cameronfraser 3y ago

nerfstudio should work on 6gb of vram, i've used the nerfacto model extensively on a laptop with that constraint. Try reducing the num-nerf-samples-per-ray param and downscaling images

vanderZwan 3y ago

It feels like optimizing NeRFs to the point where they are usable by consumer hardware while producing decent results is the main thing everyone is working on, so give it a few more years

dTal 3y ago

6GB of VRAM! Luxury! I managed to reconstruct the lego scene using TensoRF[0] (not strictly NeRF, but a similar approach with similar results) last night on my nvidia-equipped T480 (2gb of VRAM)[1], so it's possible. >But, there are really no viable models that run on consumer hardware like llama or stable diffusion. This isn't a strictly accurate framing, there is no pre-trained "model" that you "run" inference on like with Llama or Stable Diffusion. You are training the model, from scratch, on each new scene. The viability of this on a given GPU depends on the combined size of the input + output, i.e. the resolution and number of input images and the resolution and compactness of the resulting data structure. There's nothing in principle preventing you from training tiny low-res nerfs from tiny low-res images, except that all the researchers in this space are working with standard datasets of a standard size on big beefy machines and their code is full of magic numbers. Also, many of the improvements on the original NeRF achieve their speedup through much hungrier data structures (voxels, multiresolution, etc). TensoRF appears to have a very compact scene representation (like the original NeRF) and very fast training (like instant-ngp) so it seems to be a sweet spot for low-end hardware - at any rate, it's the first thing I managed to get working on this laptop. The main downside seems to be that inference (generating new images) is quite slow, at about 14 seconds. [0] https://github.com/apchenstu/TensoRF/ https://github.com/apchenstu/TensoRF/ - it's apparently included in nerfstudio as well [1] batch_size = 512 in configs/lego.txt (with your 6gb you'd get away with 2048) and compute_extra_metrics=False wherever it appears in train.py

somethingsome 3y ago

I obtain high quality without neural networks or specific hardware, but I don't have the traction of high end labs. It requires a bit more of manual work, but not that much