3 ms·

Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete

Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete steps). So rather simple.

Sharlin 2y ago
This is not "just" video, however. It's interactive in real time. Sure, you can say that playing is simply video with some extra parameters thrown in to encode player input, but still.
- slashdave 2y ago
  It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.
  - raincole 2y ago
    ? I highly suggest you to read the paper briefly before commenting on the topic. The whole point is that it's not just generating a video.
    - slashdave 2y ago
      I did. It is generating a video, using latent information on player actions during the process (which it also predicts). It is not interactive.
  - SeanAnderson 2y ago
    I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The user is "actually" firing a gun.
    - nopakos 2y ago
      Maybe it's so advanced, it knows the players' next moves, so it is a video!
      - slashdave 2y ago
        I guess you are being sarcastic, except this is precisely what it is doing. And it's not hard: player movement is low information and probably not the hardest part of the model.
    - smusamashah 2y ago
      It's interactive but can it go beyond what it learned from the videos. As in, can the camera break free and roam around the map from different angles? I don't think it will be able to do that at all. There are still a few hallucinations in this rendering, it doesn't look it understands 3d.
      - Sharlin 2y ago
        You might be surprised. Generating views from novel angles based on a single image is not novel, and if anything, this model has more than a single frame as input. I’d wager that it’s quite able to extrapolate DOOM-like corridors and rooms even if it hasn’t seen the exact place during training. And sure, it’s imperfect but on the other hand it works in real time on a single TPU.
    - hypertele-Xii 2y ago
      Then why do monsters become blurry smudgy messes when shot? That looks like a video compression artifact of a neural network attempting to replicate low-structure image (source material contains guts exploding, very un-structured visual).
      - Sharlin 2y ago
        Uh, maybe because monster death animations make up a small part of the training material (ie. gameplay) so the model has not learned to reproduce them very well? There cannot be "video compression artifacts" because it hasn’t even seen any compressed video during training, as far as I can see. Seriously, how is this even a discussion? The article is clear that the novel thing is that this is real-time frame generation conditioned on the previous frame(s) AND player actions. Just generating video would be nothing new.
  - Sharlin 2y ago
    Uff, I guess you’re right. Mea culpa. I misread their diagram to represent inference when it was about training instead. The latter is conditioned on actions, but… how do they generate the actual output frames then? What’s the input? Is it just image-to-image based on the previous frame? The paper doesn’t seem to explain the inference part at all well :(
    - slashdave 2y ago
      It should be possible to generate an initial image from Gaussian noise, including the latent information on player position
InDubioProRubio 2y ago
Video is also higher resolution, as the pixels flip for the high resolution world by moving through it. Swivelling your head without glasses, even the blurry world contains more information in the curve of pixelchange.
- slashdave 2y ago
  Correct, for the sprites. However, the walls in Doom are texture mapped, and so have the same issue as videos. Interesting, though, because I assume the antialiasing is something approximate, given the extreme demands on CPUs of the era.

Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete steps). So rather simple.

Sharlin 2y ago

This is not "just" video, however. It's interactive in real time. Sure, you can say that playing is simply video with some extra parameters thrown in to encode player input, but still.

slashdave 2y ago
It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.
- raincole 2y ago
  ? I highly suggest you to read the paper briefly before commenting on the topic. The whole point is that it's not just generating a video.
  - slashdave 2y ago
    I did. It is generating a video, using latent information on player actions during the process (which it also predicts). It is not interactive.
- SeanAnderson 2y ago
  I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The user is "actually" firing a gun.
  - nopakos 2y ago
    Maybe it's so advanced, it knows the players' next moves, so it is a video!
    - slashdave 2y ago
      I guess you are being sarcastic, except this is precisely what it is doing. And it's not hard: player movement is low information and probably not the hardest part of the model.
  - smusamashah 2y ago
    It's interactive but can it go beyond what it learned from the videos. As in, can the camera break free and roam around the map from different angles? I don't think it will be able to do that at all. There are still a few hallucinations in this rendering, it doesn't look it understands 3d.
    - Sharlin 2y ago
      You might be surprised. Generating views from novel angles based on a single image is not novel, and if anything, this model has more than a single frame as input. I’d wager that it’s quite able to extrapolate DOOM-like corridors and rooms even if it hasn’t seen the exact place during training. And sure, it’s imperfect but on the other hand it works in real time on a single TPU.
  - hypertele-Xii 2y ago
    Then why do monsters become blurry smudgy messes when shot? That looks like a video compression artifact of a neural network attempting to replicate low-structure image (source material contains guts exploding, very un-structured visual).
    - Sharlin 2y ago
      Uh, maybe because monster death animations make up a small part of the training material (ie. gameplay) so the model has not learned to reproduce them very well? There cannot be "video compression artifacts" because it hasn’t even seen any compressed video during training, as far as I can see. Seriously, how is this even a discussion? The article is clear that the novel thing is that this is real-time frame generation conditioned on the previous frame(s) AND player actions. Just generating video would be nothing new.
- Sharlin 2y ago
  Uff, I guess you’re right. Mea culpa. I misread their diagram to represent inference when it was about training instead. The latter is conditioned on actions, but… how do they generate the actual output frames then? What’s the input? Is it just image-to-image based on the previous frame? The paper doesn’t seem to explain the inference part at all well :(
  - slashdave 2y ago
    It should be possible to generate an initial image from Gaussian noise, including the latent information on player position

InDubioProRubio 2y ago

Video is also higher resolution, as the pixels flip for the high resolution world by moving through it. Swivelling your head without glasses, even the blurry world contains more information in the curve of pixelchange.

slashdave 2y ago
Correct, for the sprites. However, the walls in Doom are texture mapped, and so have the same issue as videos. Interesting, though, because I assume the antialiasing is something approximate, given the extreme demands on CPUs of the era.