4 ms·

It's telling IMO that they only want people opinions based on our notoriously faulty memories rather than sitting comparable situations next to one another in t

It's telling IMO that they only want people opinions based on our notoriously faulty memories rather than sitting comparable situations next to one another in the game and simulation then analyzing them. Several things jump out watching the example video.

GaggiX 2y ago
>rather than sitting comparable situations next to one another in the game and simulation then analyzing them. That's literally how the human rating was setup if you read the paper.
- meheleventyone 2y ago
  I think you misunderstand me. I don't mean a snap evaluation and deciding between two very-short competing videos which is what the participants were doing. I mean doing an actual analysis of how well the simulation matches the ground truth of the game. What I'd posit is that it's not actually a very good replication of the game but very good a replicating short clips that almost look like the game and the short time horizons are deliberately chosen because the authors know the model lacks coherence beyond that.
  - GaggiX 2y ago
    >I mean doing an actual analysis of how well the simulation matches the ground truth of the game. Do you mean the PSNR and LPIPS metrics used in paper?
    - meheleventyone 2y ago
      No, I think I've been pretty clear that I'm interested in how mechanically sound the simulation is. Also those measures are over an even shorter duration so even less relevant to how coherent it is at real game scales.
      - GaggiX 2y ago
        How should this be concretely evaluated and measured? A vibe check?
        meheleventyone 2y ago
        I think the studies evaluation using very short video and humans is much more of a vibe check than what I’ve suggested. Off the top of my head DOOM is open source so it should be reasonable to setup repeatable scenarios and use some frames from the game to create a starting scenario for the simulation that is the same. Then the input from the player of the game could be used to drive the simulated version. You could go further and instrument events occurring in the game for direct comparison to the simulation. I’d be interested in setting a baseline for playtime of the level in question and using sessions of around that length as an ultimate test. There are some on obvious mechanical deficiencies seen in the videos they’ve published. One that really stood out to me was the damage taken when in the radioactive slime. So I don’t think the analysis would need to particularly deep to find differences.

GaggiX 2y ago

>rather than sitting comparable situations next to one another in the game and simulation then analyzing them. That's literally how the human rating was setup if you read the paper.

meheleventyone 2y ago
I think you misunderstand me. I don't mean a snap evaluation and deciding between two very-short competing videos which is what the participants were doing. I mean doing an actual analysis of how well the simulation matches the ground truth of the game. What I'd posit is that it's not actually a very good replication of the game but very good a replicating short clips that almost look like the game and the short time horizons are deliberately chosen because the authors know the model lacks coherence beyond that.
- GaggiX 2y ago
  >I mean doing an actual analysis of how well the simulation matches the ground truth of the game. Do you mean the PSNR and LPIPS metrics used in paper?
  - meheleventyone 2y ago
    No, I think I've been pretty clear that I'm interested in how mechanically sound the simulation is. Also those measures are over an even shorter duration so even less relevant to how coherent it is at real game scales.
    - GaggiX 2y ago
      How should this be concretely evaluated and measured? A vibe check?
      - meheleventyone 2y ago
        I think the studies evaluation using very short video and humans is much more of a vibe check than what I’ve suggested. Off the top of my head DOOM is open source so it should be reasonable to setup repeatable scenarios and use some frames from the game to create a starting scenario for the simulation that is the same. Then the input from the player of the game could be used to drive the simulated version. You could go further and instrument events occurring in the game for direct comparison to the simulation. I’d be interested in setting a baseline for playtime of the level in question and using sessions of around that length as an ultimate test. There are some on obvious mechanical deficiencies seen in the videos they’ve published. One that really stood out to me was the damage taken when in the radioactive slime. So I don’t think the analysis would need to particularly deep to find differences.