4 ms·
Doom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7
by godelski 2y ago
Doom system requirements:
- 4 MB RAM
- 12 MB disk space
Stable diffusion v1
> 860M UNet and CLIP ViT-L/14 (540M)
Checkpoint size:
4.27 Gb
7.7 GB (full EMA)
Running on a TPU-v5e
Peak compute per chip (bf16) 197 TFLOPs
Peak compute per chip (Int8) 393 TFLOPs
HBM2 capacity and bandwidth 16 GB, 819 GBps
Interchip Interconnect BW 1600 Gbps
This is quite impressive, especially considering the speed. But there's still a ton of room for improvement. It seems it didn't even memorize the game despite having the capacity to do so hundreds of times over. So we definitely have lots of room for optimization methods. Though who knows how such things would affect existing tech since the goal here is to memorize.
What's also interesting about this work is it's basically saying you can rip a game if you're willing to "play" (automate) it enough times and spend a lot more on storage and compute. I'm curious what the comparison in cost and time would be if you hired an engineer to reverse engineer Doom (how much prior knowledge do they get considering pertained models and visdoom environment. Was doom source code in T5? And which vit checkpoint was used? I can't keep track of Google vit checkpoints).
I would love to see the checkpoint of this model. I think people would find some really interesting stuff taking it apart.
- https://www.reddit.com/r/gaming/comments/a4yi5t/original_doom_system_requirements_from_my_25_year/ https://www.reddit.com/r/gaming/comments/a4yi5t/original_doo...
- https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/tree/main https://huggingface.co/CompVis/stable-diffusion-v-1-4-origin...
- https://cloud.google.com/tpu/docs/v5e https://cloud.google.com/tpu/docs/v5e
- https://github.com/Farama-Foundation/ViZDoom https://github.com/Farama-Foundation/ViZDoom
- https://zdoom.org/index https://zdoom.org/index
- snickmy 2y agoThose are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequence of frames with such contextual awareness without explictly having to encode it, is the real breakthrough here. Both from a pure gaming standpoint, but on simulation in general.
- pickledoyster 2y ago>you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. OR one can hope it will be thrown to the heap of nonviable tech with the rest of spam waste
- tobr 2y agoI suppose it also doesn't really matter what kinds of resources the game originally requires. The diffusion model isn't going to require twice as much memory just because the game does. Presumably you wouldn't even necessarily need to be able to render the original game in real time - I would imagine the basic technique would work even if you used a state of the Hollywood-quality offline renderer to render each input frame, and that the performance of the diffusion model would be similar?
- godelski 2y agoWell the majority of ML systems are compression machines (entropy minimizers), so ideally you'd want to see if you can learn the assets and game mechanics through play alone (what this paper shows). Better would be to do so more efficiently than that devs themselves, finding better compression. Certainly the game is not perfectly optimized. But still, this is a step in that direction. I mean no one has accomplished this before so even with a model with far higher capacity it's progress. (I think people are interpreting my comment as dismissive. I'm critiquing but the key point I was making was about how there's likely better architectures, training methods, and all sorts of stuff to still research. Personally I'm glad there's still more to research. That's the fun part)
- godelski 2y agoI'm not sure what you're saying is irrelevant. 1) the model has enough memory to store not only all game assets and engine but even hundreds of "plays". 2) me mentioning that there's still a lot of room to make these things better (seems you think so too so maybe not this one?) 3) an interesting point I was wondering to compare current state of things (I mean I'll give you this but it's just a random thought and I'm not reviewing this paper in an academic setting. This is HN, not NeurIPS. I'm just curious ¯ \ _ ( ツ ) _ / ¯) 4) the point that you can rip a game I'm really not sure what you're contesting to because I said several things. > it lacks basic things like pre-computing, storing, etc. It does? Last I checked neural nets store information. I guess I need to return my PhD because last I checked there's a UNet in SD 1.4 and that contains a decoder.
- dTal 2y ago>What's also interesting about this work is it's basically saying you can rip a game if you're willing to "play" (automate) it enough times and spend a lot more on storage and compute That's the least of it. It means you can generate a game from real footage. Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year.
- isaacfung 2y agoThe possibility seems far beyond gaming(given enough computation resources). You can feed it with videos of usage of any software or real world footage recorded by a Go Pro mounted on your shoulder(with body motion measured by some sesnors though the action space would be much larger). Such a "game engine" can potentially be used as a simulation gym environment to train RL agents.
- camtarn 2y agoPlus, presumably, either training it on pilot inputs (and being able to map those to joystick inputs and mouse clicks) or having the user have an identical fake cockpit to play in and a camera to pick up their movements. And, unless you wanted a simulator that only allowed perfectly normal flight, you'd have to have those airliners go through every possible situation that you wanted to reproduce: warnings, malfunctions, emergencies, pilots pushing the airliner out of its normal flight envelope, etc.
- phh 2y ago> Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year. I guess that's the occasion to remind that ML is splendid at interpolating, but extrapolating, maybe don't keep your hopes too high. Namely, to have a "perfect flight sim" using GoPros, you'll need to record hundreds of stalls and crashs.
- dvngnt_ 2y agowouldnt make more sense to train using microsoft flight simulator the same way they did DOOM, but im not sure what the point is if the game already exists
- godelski 2y ago