8 ms·
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
- ArtWomb 6y agoPersonally prefer blocky Voxel Art to the photoreal scene ;) NVidia also released their RTXDI SDK for global illumination at a scale of millions of dynamic lights in real time. Combined with GANCraft, anyone could become a world class environmental artist using only Pixel Art tools. https://developer.nvidia.com/rtxdi https://developer.nvidia.com/rtxdi
- moistbar 6y agoNVidia really likes rendering landscapes, huh? https://blogs.nvidia.com/blog/2019/03/18/gaugan-photorealistic-landscapes-nvidia-research/ https://blogs.nvidia.com/blog/2019/03/18/gaugan-photorealist...
- BugsJustFindMe 6y agoLandscape images are classic exemplars for texture-by-number algorithms because nature's variety means that it's easier to make them look real enough. See the OG transfer algorithm called "Image Analogies" from decades before the GAN boom: https://mrl.cs.nyu.edu/projects/image-analogies/potomac.html https://mrl.cs.nyu.edu/projects/image-analogies/potomac.html https://mrl.cs.nyu.edu/projects/image-analogies/arch.html https://mrl.cs.nyu.edu/projects/image-analogies/arch.html
- Iv 6y agoYes, there are all sort of weirdness in their rendering but that's what you get in a research paper. Put that in the hands of actual game designers and you will have incredible possibilities.
- Vermeulen 6y agoTake this idea, and apply it to Google Earth. Have it procedurally generate the rest of the data as I zoom in. Its incredible I can check out my small home town on Google Earth now entirely in 3d (which wasnt the case just a few months ago). Yet the trees/cars are still these blocky low resolution things sticking out of the ground. Imagine Google Earth procedurally generating a high resolution mesh as I zoomed in. Train it with high resolution photogrammetry of other similar locations for the ground truth - and let me zoom in endless in their VR app
- esclerofilo 6y agoI believe Microsoft Flight Simulator does this (and it has VR support). It gets a bit tiring to see the same trees on every place on earth, though.
- ThalesX 6y agoTheir building gen is also super nice but it really takes away from the experience when flying a known location.
- LennyWhiteJr 6y agoYeah, going and flying over the small rural farm I grew up on I was disappointing since Flight Simulator had procedurally generated a whole compound full of random buildings that made it look like some cult compound.
- kebman 6y agoIt's been a couple of years, but I still like the Outerra project. And also the Bohemia Interactive VBS.
- x86ARMsRace 6y agoWhat would be fun is the inverse. Feed in a section of Google Earth and have it generate a Minecraft world out the other end.
- Bjartr 6y agoIt appears that someone's worked that out. reddit.com/r/Minecraft/comments/x56lg/using_google_earth_terrain_data_for_making_a/
- x86ARMsRace 6y agoThis is fantastic, good find!
- 6y ago
- xixixao 6y agoAt the end of the paper it says that one frame takes 10secs to render. I wonder whether one day this method will be able to render in real time (say 30fps).
- Tenoke 6y agoAlmost definitely. There's many ways to optimize further with software and hardware is only getting better. I wouldn't be surprised if it's doable today with some cheating, a bit more hardware and a lot of work on optimization.
- tachyonbeam 6y agoMaybe, but OTOH we have very efficient 3D rendering technology that we understand very well. If I had more compute, I'd want to raytrace everything in real-time, but I wouldn't feel the need to bring neural networks into the mix. A better use case of machine learning is probably to help procedurally generate the data to be rendered. It would be really neat to be able to turn a few photos of a real-world location into high quality 3D meshes with no gaps, for example.
- willis936 6y agoThis creates a realistic topography from voxels. If you could do this in realtime then you could have a game where you have the flexibility of minecraft yet the appearance of a more photorealistic game. Imagine playing a game that looked like Control except everything is destructible and constructible. It's an exciting idea.
- deleted 6y ago[deleted]
- tediousdemise 6y agoCivil engineers, architects and landscapers are going to have a field day with this.
- rocky1138 6y agoWould having a higher resolution texture pack make for better results?
- willis936 6y agoIt doesn't look like it uses any texture information. I think it only takes in a list of block locations and spits out a scene. I would think you would have to train it with every different combination of textures.
- zelon88 6y agoIt looks impressive, but what exactly is the machine learning doing on the original to produce the result? And wouldn't it be possible to simply take the original minecraft map as a height map and texture map and then regenerate a new world with the original world data and more advanced post processing? You could interpolate and randomize more detail into the scene than you started with.
- erikpukinskis 6y agoIt’s not really adding any meaningful detail per se. where there’s a grass block it’s just rendering grass. All it is doing is projecting a stable image of “grass” (taken from a labeled image database) in that voxel. Not to minimize the awesomeness of that... doing it stably in 3D while moving the camera is the point of this paper, and is amazing. But it’s not really adding detail beyond “these are the kinds of pixels that grass has and the AI figured out we can put them in this arrangement without making things jumpy”
- debacle 6y agoIt also seems like your brain is doing most of the work here: https://nvlabs.github.io/GANcraft/images/vox2img.png https://nvlabs.github.io/GANcraft/images/vox2img.png The renderer seems to be adding some resolution, smoothing, and mipmapping. Shaders can do the same thing, and in real time.
- Bjartr 6y agoModern shaders[1] do a lot, but you'll never mistake them for anything but minecraft. They don't quite get to where this paper is demonstrating. [1] https://www.rockpapershotgun.com/best-minecraft-shaders https://www.rockpapershotgun.com/best-minecraft-shaders
- lainga 6y agoIt doesn't look like it can do structures, either.
- captainclam 6y agoQuestion for the experts: Is it possible that GANs will be used for rendering video game environments in the near-ish future? This has been one of my private predictions with respect to upcoming tech, but I'd love to know if people are already thinking about this, or alternatively, why it won't happen.
- Tenoke 6y agoNon real-time as in generating levels for your game to release it's doable today. Real-time it will probably be doable soon, especially for shared-world games where a server can generate for multiple people at a time rather than single-player. Even real-time today it should be doable if you create your game with that in mind. You really don't need to generate all the textures, just a compact representation of the level which is to be rendered normally after the fact.
- malka 6y agoIt has already started with dlss. I am not sure about Nvidia implementation but super resolution can have some adverserial training. If you mean asset / level generation. Then yes. It is the next step in procedural generation imo.
- Agentlien 6y agoDLSS is just image upscaling using a neural network. It's a very different problem from what is shown here or what I believe GP is talking about.
- aspaviento 6y agoIf the result is also a 3d environment, it could save a lot of time designing scenarios.
- mcbuilder 6y agoArtistically, developers could do some trippy dream sequences with GANs, where the glitchyness and training artifacts add to the immersions. Because one can sample GANs or mix in latent dimensions, the experience can be tailored individually based on the characters decisions for instance.
- harias 6y agoSounds a lot like Google's GAN for fantastical creatures[0]. Labels to photorealism seems to be the core idea behind both [0] https://ai.googleblog.com/2020/11/using-gans-to-create-fantastical.html?m=1 https://ai.googleblog.com/2020/11/using-gans-to-create-fanta...
- debacle 6y agoThe neural net part of this seems somewhat trivial and also misapplied. This is not a realtime renderer, and I would hazard that if you gave someone who knows GLSL the task, they would produce something far and away more compelling than this, that could probably render at <1 FPS. https://nvlabs.github.io/GANcraft/images/vox2img.png https://nvlabs.github.io/GANcraft/images/vox2img.png
- exporectomy 6y agoThey would produce something which won't generalize to other types of environment without another huge load of human labor. Your complaint could be made about just about any new technology. It's usually worse than what came before it at first, but the value is in the potential to eventually become better than what came before it.
- throwaway10110 6y agoHeh i wonder if we ever get Minecraft 2.0, i get a good chuckle how it just barely runs on consoles and ultrapowerful PCs yet looks so "basic" Myself and my son absolutely love it and spend months in this pandemic deep in minecraft worlds
- lwansbrough 6y agoThat’s mostly a product of Minecraft’s technical choices. Modern computers can render axis aligned voxel grids on the order of 1,000,000^3 (think Minecraft scale but the blocks are sub millimeter) with PBR/GI in real time. Interactive would be another story I suppose.
- iguessthislldo 6y agoI like how if you look close enough, the outlines of trees and hills still are block-ish.
- killvung 6y agoBack in the day someone complained whether the Computer Graphic course is related to Machine Learning / AI topic at all. Now here's an answer.
- jchanimal 6y agoAnyone doing GAN with before: old streetview pre bikelanes after: streetview with new bike lanes profit: now you can see what any town would look like with complete streets. I call it Complete Street View. Please do implement. Of course it would be dreamlike, this is a strength as you wouldn’t want the gan to make design recommendations, just a plausible feel.
- Bjartr 6y agoThere's a probably a larger opportunity for an urban redevelopment sketching/brainstorming tool of sorts.
- brundolf 6y agoSuper cool. With that said I bet this would choke on lots of actual Minecraft worlds, because people often build things using blocks where the semantics get thrown completely out the window in favor of aesthetics. Want a big mural on the wall? You're going to be building the wall itself out of differently-colored blocks of wool Maybe they'll solve that part one day :) Edit: That said, it could choke in some really interesting ways...
- waiseristy 6y agoWow! Amazing results, it's like marching cubes on an acid trip!
- 29athrowaway 6y agoThis is what happened in your head when you played Atari games as a kid.
- d23 6y agoMaybe I'm just too dumb, but I wish these papers would cut the nonsense and explain the key elements in layman's terms with simple examples. I'm super curious how you can do something like this in a fully unsupervised fashion, but the "Hybird Voxel-conditional Neural Rendering" doesn't mean much to me. Maybe if I knew what "voxel-bounded neural radiance fields" were...
- chrisseaton 6y ago> I wish these papers would cut the nonsense and explain the key elements in layman's terms with simple examples If papers did that, they'd be a thousand pages long. The target audience is people intimately familiar with the state of the art.
- Mehdi2277 6y agoThe voxel bounded neural radiance field is important as neural radiance field was some prior research paper this builds off. But the very high level is just voxel data to image generation using some form of neural nets. I didn’t look at the paper but I’d hope it summarizes neural radiance fields and if not it’ll at least cite them and then you’d read there and see how this paper extends that work.
- sly010 6y agoThe image translators work for the construct program, but there is way too much information to decode the matrix. You get used to it. I don't even see the code, all I see is blonde, brunette, readhead...
- xupybd 6y agoWow that site and all it's auto play videos crashed my phone. I get that you want to show off the cool tech but please don't put that many videos on autoplay.
- shuringai 6y agoThe clickbait article makes people believe "you can create 3d models of ANY 2d object" but in reality, this would only come down to cars, cats and human faces. We have only so much datasets that are suitable for a GAN.