7 ms·
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields
- IshKebab 3y agoVery impressive! How far are we from having this sort of thing in VR? The paper says the model render time was 0.9s which I guess means very far if that is per frame?
- krasin 3y agoinstant-ngp ([1]) from NVIDIA can render NeRF in VR in real-time, assuming a very good desktop video card. Note that instant-ngp is not as photo-realistic as Zip-NeRF. But it's still very good! 1. https://github.com/NVlabs/instant-ngp https://github.com/NVlabs/instant-ngp
- IshKebab 3y agoThat looks amazing. What about video? How much data is one model?
- Y_Y 3y agoWhere's the damn code?
- esafak 3y agohttps://paperswithcode.com/paper/zip-nerf-anti-aliased-grid-based-neural https://paperswithcode.com/paper/zip-nerf-anti-aliased-grid-...
- lairv 3y agoAnother recent cool work in this field is this paper : https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ They manage to get the same quality, with <1hr of training, and running at 60fps 1080p, it uses point cloud instead of volumetric representation
- evilhackerdude 3y agoCode coming in June/July: https://twitter.com/Snosixtytwo/status/1666551263964667904 https://twitter.com/Snosixtytwo/status/1666551263964667904
- porphyra 3y agoI'm anxiously waiting for the code (or for someone to reimplement it open source). Sounds very fun to play with. I've recently been having fun with OpenMVS [1]. Using Gaussian splatting (which is initialized with a point cloud) would bring it to the next level! [1] https://github.com/cdcseacave/openMVS https://github.com/cdcseacave/openMVS
- pezezin 3y agoA few years ago, after my grandparents died, I went to their apartment and took around 2000 photos to run them through OpenMVG and OpenMVS and make a 3D model to remember it forever. This looks way better, I hope one day I have the hardware to be able to run it...
- EZ-Cheeze 3y agothank you SO much for posting this, and to the HN community for putting it at the top i think yours is the only comment with actual value (and i'm including mine)
- yarg 3y agoThis is amazing, but... It feels too damned clean, and I'm not sure its just the weirdly alien camera stability.
- dTal 3y agoDue to Nyquist, it will be missing specular reflections with high angular frequency, which gives everything a dull, "satin" texture. You will never see anything "twinkle", "sparkle", or "glitter" in one of these, nor will mirrors likely be accurate. You can see this right in the beginning of the video, center frame - a mirror has an odd, blurry texture. You can see it again more subtly in the kitchen - it's full of reflective objects, in a room with bright overhead spotlights, and yet we never see sharp highlights. Diffuse highlights, sure, but nothing goes 'ting'.
- nawgz 3y agoIt's just really horrible camera work. Rolling the camera like that only makes sense when it corresponds with acceleration, but here every corner has a lot of camera roll and the amount is not correlated with the movement. That induces motion sickness, which is why it feels bad to look at despite excellent image quality. A piece of commentary I'm less sure of is that there seems to be a total lack of motion blur, which is not what we've come to expect out of video
- londons_explore 3y agoI think the 'bad' camera work is because the camera route is probably hard coded coordinates in a python script, and someone has done trial and error to try and find some set of paths that don't pass through stuff...
- nawgz 3y agoIndeed, but that path code should just keep the perspective upright to avoid causing motion sickness
- jrk 3y agoThis is just a raw result of many frames strung together from a view synthesis technique, with an arbitrary, programmer-designed camera path. Neither of these issues is fundamental to the technique: - The camera path could be anything, and nicer ones could be easily designed by an artist - Motion blur is just a matter of supersampling in time. You actually don't want blur in the base reconstruction of individual views, as that would mean loss of detail when you were sitting still. In short, this video is not meant to show the output you'd actually want for an application (which might be different for a movie vs. VR vs. something else), but just to distill many outputs from a view synthesis algorithm into a form easily digestible by a human reviewer.
- xrd 3y agoI've been fascinated by NeRFs for a few years. But, there are really no viable models that run on consumer hardware like llama or stable diffusion. Or, am I wrong? NeRF Studio seems promising but never works on my 6GB nvidia. I would really like to find a way to interpolate between two images using a NeRF (get the hallucination of the "image in between"). Is there such a thing out there?
- dheera 3y agoI don't know about 6GB but if you have a 12GB or 16GB card you should be able to run the vast majority of NeRF work out there, most of it is designed to run on a single GPU.
- xrd 3y agoDo you know if there are any NERFs that can be run in command line mode, where you can see an intermediate image by giving it two or more images? I'm less interested in video than in a single image result.
- riotnrrd 3y agoI mostly have experience with Instant NGP, and it should work on older consumer NVIDIA cards. Their github page calls out Pascal cards as working, for example. 6 GB isn't much memory, though, so you may be limited in final resolution of the latent model and thus of the output.
- cameronfraser 3y agonerfstudio should work on 6gb of vram, i've used the nerfacto model extensively on a laptop with that constraint. Try reducing the num-nerf-samples-per-ray param and downscaling images
- vanderZwan 3y agoIt feels like optimizing NeRFs to the point where they are usable by consumer hardware while producing decent results is the main thing everyone is working on, so give it a few more years
- 3y ago
- djsavvy 3y agoI’m not sure I understand what this does —- what are the inputs and outputs?
- krasin 3y agoThe inputs are just photos from different positions. Then a neural net is trained so that you can ask to render a photo from any pose (including unseen). This particular work (Zip-NeRF) builds on top of the original NeRF paper: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf (the website has a good explanation what NeRF aka Neural Radiance Fields are)
- charliea0 3y agoDefinitely less aliasing, but it looks blurrier than the baseline.
- swayvil 3y agoA house that cluttered, cramped and crowded, yet they have 2 dining tables. It defies comprehension.
- deleted 3y ago[deleted]
- londons_explore 3y agoI know the feeling... Children can quickly clutter any house.
- next_xibalba 3y agoStreetview about to go ham.
- krasin 3y agoWaymo agrees: https://waymo.com/research/block-nerf/ https://waymo.com/research/block-nerf/
- next_xibalba 3y agoI guess it's no coincidence that these two papers share 3 authors.
- EZ-Cheeze 3y agoIf you understand the implications of this and wanna get rich with me, email me at inventor_man@outlook.com
- petermcneeley 3y agoIs there a live web demo?
- defaultcompany 3y agoI spent about 50 hours manually doing something similar a few years ago [1] but it was literally made by taking hundreds of 360 degree panoramas every two inches inside a room on a fixed path. The end result was awesome but it was so time consuming. It’s crazy what they are doing now using ML with a few input images. [1]: https://forums.tigsource.com/index.php?topic=69545.0 https://forums.tigsource.com/index.php?topic=69545.0
- evilhackerdude 3y agoWow. I'm seriously amazed. Best intro text, best item lore, best in-game computer and it looks absolutely great. Are you on Twitter/Mastodon?
- defaultcompany 3y agoThank you for saying those nice things! I am not really on any social media platforms at the moment. I used to post updates to that blog I linked but it’s been a long time since I could work on that project. Hopefully someday I will have a playable level to release. There is a demo available at the bottom of that page if you are interested. Anyway, thanks!
- coolspot 3y agoThat’s a very nice house!
- FloatArtifact 3y agoIs there any way to get a usable point cloud suitable for 3D reconstruction in blender?
- Demmme 3y agoI knew it will be worth it to make videos and pictures from the flat of my now dead grandpa . I love how this can preserve spaces
- vanderZwan 3y agoI did the same with my grandparents, guess this is a more common way of saying goodbye than I thought! I'm afraid I wasn't systematic enough to create a useful data set though, lots of gaps. I'll make sure I won't repeat that mistake and take a good fly-through of the apartments of my parents.
- smusamashah 3y agoUnofficial implementation https://paperswithcode.com/paper/zip-nerf-anti-aliased-grid-based-neural https://paperswithcode.com/paper/zip-nerf-anti-aliased-grid-...
- xiaolingxiao 3y agoHas anyone tried to turn this into a "product or experience" of sorts, and care to share their experience in doing so?