3 ms·

I also thought this, but refer back to the paper, not the abstract: > A is the set of key presses and mouse movements… > …to condition on actions, we simply l

I also thought this, but refer back to the paper, not the abstract: > A is the set of key presses and mouse movements… > …to condition on actions, we simply learn an embedding A_emb for each action So, it’s clear that in this model the diffusion process is conditioned by embedding A that is derived from user actions rather than words. Then a noised start frame is encoded into latents and concatenated on to the noise latents as a second conditioning. So we have a diffusion model which is trained solely on images of doom, and which is conditioned on current doom frames and user actions to produce subsequent frames. So yes, the users are playing it. However, it should be unsurprising that this is possible. This is effectively just a neural recording of the game. But it’s a cool tech demo.

foota 2y ago
I wonder if they could somehow feed in a trained Gaussian splats model to this to get better images? Since the splats are specifically designed for rendering it seems like it would be an efficient way for the image model to learn the geometry without having to encode it on the image model itself.
- Chance-Device 2y ago
  I’m not sure how that would help vs just training the model with the conditionings described in the paper. I’m not very familiar with Gaussian splats models, but aren’t they just a way of constructing images using multiple superimposed parameterized Gaussian distributions, sort of like the Fourier series does with waveforms using sine and cosine waves? I’m not seeing how that would apply here but I’d be interested in hearing how you would do it.
  - foota 2y ago
    I'm not certain where it would fit in, but my thinking is this. There's been a bunch of work on making splats efficient and good at representing geometry. Reading more, perhaps NERFs would be a better fit, since they're an actual neutral network. My thinking is that if you trained a NERF ahead of time to represent the geometry and layout of the levels, and plug that in to the diffusion model (as a part of computing the latents, and then also on the other side so it can be used to improve the rendering) then the diffusion model could focus on learning how actions manipulate the world without having to learn the geometry representation.
    - Chance-Device 2y ago
      I don’t know if that would really help, I have a hard time imagining exactly what that model would be doing in practise. To be honest none of the stuff in the paper is very practical, you almost certainly do not want a diffusion model trying to be an entire game under any circumstances. What you might want to do is use a diffusion model to transform a low poly, low fidelity game world into something photorealistic. So the geometry, player movement and physics etc would all make sense, and then the model paints over it something that looks like reality based on some primitive texture cues in the low fidelity render. I’d bet money that something like that will happen and it is the future of games and video.
      - foota 2y ago
        Yeah, I realize this will never be useful for much in practice (although maybe as some kind of client side prediction for cloud gaming? But likely if you could run this in real time you might as well run whatever game there is in real time as well, unless there's some kind of massive world running on the server that's too large to stream the geometry for effectively), I was mostly just trying to think of a way to avoid the issues with fake looking frames or forgetting what the level looks like when you turn around that someone mentioned. Not exactly that, but Nvidia does something like this already, they call it DLSS. It uses previous frames and motion vector to render a next frame using machine learning.
psb217 2y ago
The agent never interacts with the simulator during training or evaluation. There is no user, there is only an agent which trained to play the real game and which produced the sequences of game frames and actions that were used to train the simulator and to provide ground truth sequences of game experience for evaluation. Their evaluation metrics are all based on running short simulations in the diffusion model which are initiated with some number of conditioning frames taken from the real game engine. Statements in the paper like: "GameNGen shows that an architecture and model weights exist such that a neural model can effectively run a complex game (DOOM) interactively on existing hardware." are wildly misleading.

foota 2y ago

I wonder if they could somehow feed in a trained Gaussian splats model to this to get better images? Since the splats are specifically designed for rendering it seems like it would be an efficient way for the image model to learn the geometry without having to encode it on the image model itself.

Chance-Device 2y ago
I’m not sure how that would help vs just training the model with the conditionings described in the paper. I’m not very familiar with Gaussian splats models, but aren’t they just a way of constructing images using multiple superimposed parameterized Gaussian distributions, sort of like the Fourier series does with waveforms using sine and cosine waves? I’m not seeing how that would apply here but I’d be interested in hearing how you would do it.
- foota 2y ago
  I'm not certain where it would fit in, but my thinking is this. There's been a bunch of work on making splats efficient and good at representing geometry. Reading more, perhaps NERFs would be a better fit, since they're an actual neutral network. My thinking is that if you trained a NERF ahead of time to represent the geometry and layout of the levels, and plug that in to the diffusion model (as a part of computing the latents, and then also on the other side so it can be used to improve the rendering) then the diffusion model could focus on learning how actions manipulate the world without having to learn the geometry representation.
  - Chance-Device 2y ago
    I don’t know if that would really help, I have a hard time imagining exactly what that model would be doing in practise. To be honest none of the stuff in the paper is very practical, you almost certainly do not want a diffusion model trying to be an entire game under any circumstances. What you might want to do is use a diffusion model to transform a low poly, low fidelity game world into something photorealistic. So the geometry, player movement and physics etc would all make sense, and then the model paints over it something that looks like reality based on some primitive texture cues in the low fidelity render. I’d bet money that something like that will happen and it is the future of games and video.
    - foota 2y ago
      Yeah, I realize this will never be useful for much in practice (although maybe as some kind of client side prediction for cloud gaming? But likely if you could run this in real time you might as well run whatever game there is in real time as well, unless there's some kind of massive world running on the server that's too large to stream the geometry for effectively), I was mostly just trying to think of a way to avoid the issues with fake looking frames or forgetting what the level looks like when you turn around that someone mentioned. Not exactly that, but Nvidia does something like this already, they call it DLSS. It uses previous frames and motion vector to render a next frame using machine learning.

psb217 2y ago

The agent never interacts with the simulator during training or evaluation. There is no user, there is only an agent which trained to play the real game and which produced the sequences of game frames and actions that were used to train the simulator and to provide ground truth sequences of game experience for evaluation. Their evaluation metrics are all based on running short simulations in the diffusion model which are initiated with some number of conditioning frames taken from the real game engine. Statements in the paper like: "GameNGen shows that an architecture and model weights exist such that a neural model can effectively run a complex game (DOOM) interactively on existing hardware." are wildly misleading.