5 ms·

> League of Legends does not use a fixed-step server clock, but rather a measured frame delta time which we throttle to a target frame rate. This makes the serv

> League of Legends does not use a fixed-step server clock, but rather a measured frame delta time which we throttle to a target frame rate. This makes the server more resilient to system-wide performance spikes, but this pattern definitely complicated determinism. This could use some elaboration. I don't even know how it's possible to have determinism without a fixed-step. I must misunderstand what this means. Every simulation tick in league of legends has gotta be the same duration, right? ie: 1.0 / 60.0?

tylerhou 2y ago
You can still achieve determinism by treating the measured delta-time as an input, and resimulating the game with recorded delta-time. This may not be feasible on public servers, but is definitely feasible for professional play where determinism is important to rewind games due to bugs which affect competitive integrity.
- maest 2y ago
  They have so many bugs that there's a procedure for rewinding the game during comp plays? Did I misunderstand?
  - pohuing 2y ago
    This is not unusual. Bugs happen and when there's a lot of money on the line you need a rollback system.
    - maest 2y ago
      Interesting. I don't think I've seen this in any other competitive game, which is why I'm surprised. But maybe I'm just lacking information
      - mackman 2y ago
        I implemented a debug replay system for an American Football game when I worked at Sony. The AI hierarchy and all animation and physics state was recorded and game could be rewound to figure out why an NPC made a poor decision. Only needed to track state for the duration of a single play. This was on PS2 which I think had 32MB of memory but the dev systems had an extra 96MB which we could use for this sort of thing.
        chowells 2y ago
        Hell. Replay review in American Football and other sports is a form of rewinding and fixing bugs in officiating. So it's certainly done in real life when money's on the line.
      - nightowl_games 2y ago
        Starcraft 2 lets you resume a game from a replay.
      - tylerhou 2y ago
        In StarCraft II you can resume from replay (and maybe in previous games as well), but I don't know if this has ever actually been used due to bugs in the game engine during competitive play.
  - IanGabes 2y ago
    They do! Its called Chronobreak. They have used it many times in professional matches successfully, but it doesn't work in 100% of scenarios.
- nightowl_games 2y ago
  Yeah you _could_ but I fail to see why you would ever do this. It's also not feasible to synchronize the client with the server in real time with this input.
  - tylerhou 2y ago
    You don't need to synchronize the server and the client. The point of determinism is to be able to reproduce game states deterministically, not to have people play the video game deterministically. Again, this is to "undo" bugs in pro play (Chronobreak).
DarthBatman 2y ago
Thought I heard my ears ringing... Blast from the past here. That's the beauty part of recording server state and playing it back -- we included a recording of whatever accurate integer-based OS time I could get on each platform. On Windows, for example, I record the raw values of QPC ticks and the tick rate into the recording. Then on playback, I play back the clocks themselves as if the hardware has that tick rate and tick count are from the recorded computer. So if frame 10 took 33.3ms and frame 11 took 66.7ms, we'd play them back that way. Note that playbacks are almost never in real-time. The reason a typical server frame takes 33ms is that we might be processing for 5ms and then yielding the thread (sleeping) for the remaining time during real-time gameplay. On playback, we don't yield the thread, so we play back faster than real-time when doing something like a live Chronobreak (disaster recovery.) Strictly speaking, you are right. Let's say we had a game with no player inputs (like a bots-only game.) It's possible that the outcomes might vary between two games with the same random seeds if they run different numbers of frames over the same time period. That said, we've actually tried this a few times, and our game code is surprisingly resilient to this -- only in a few instances have we found issues where subtle variations in frame time could impact player outcomes (and those are generally corrected when we find them -- this is an esports game after all.) A lot of this comes down to tradeoffs. Yes we might have those subtle issues, but if we fixed our timestep, then we'd have the other problem of the game potentially going into slow-motion during (the admittedly rare) periods of long frame times. In practice, we decided time dilation is more negative for player performance than loss of interactive resolution; humans are great predictors of motion with limited information, but they're not so great when the rules of localized reality bend. Edit: typo
- nightowl_games 2y ago
  > It's possible that the outcomes might vary between two games So just to clarify: in League, the 'dt' passed into each 'simulation step' is NOT constant? Isnt this kinda crazy? In your later articles, floating point imprecision is talked about. Couldnt this variance in dt create odd behaviour and ultimate contribute to weird things like 'character movement speed' being not _exactly_ the same between games? (like really small, but still...) And beyond that, how does the client and the server synchronize with each other if the frame #s represent different positions in time? My mind is blown right now... Note: I've worked on many networked games, and have written rollback/resimulate/replay code. I don't really understand _why_ League wouldnt use a fixed time step. Whats the advantage? In our games, the rendering of course uses the real dt passed in from the OS, but the simulation step is always fixed. This means in our games during a replay, your computer could render the outcome differently if your frame rate is different, but the raw simulation is always the same. For context, to show I at least have some idea what I'm talking about, I made this replay system (and the game has rollback multiplayer): https://gooberdash.winterpixel.io/?play=5b74f7c0-8591-40dc-b0f4-f8e9d3c8cb57&ghost=52721394-f918-4d62-aa5d-d50fcf3e8d68 https://gooberdash.winterpixel.io/?play=5b74f7c0-8591-40dc-b... I havent played a lot of League, but I always assumed it would use deterministic lock step networking (like it's predecessor, Warcraft 3)
mackman 2y ago
The physics game sim step had to be hardcoded but you could vary the graphics rendering independently. That did mean you needed to use a different RNG for sim objects and graphics-only particle effects. I worked on the deterministic instant-replay system for a racing sim on PS3. The most interesting thing was that on PS3 we had main processor cores (PPU) and helper cores (SPU) which had their own instruction set. Our physics sim could distribute work across both, but the thread assignment had to be deterministic because PPU and SPU got marginally different floating point results for the same instructuion and inputs. That was a fun one to figure out!
- hamish-b 2y ago
  Incredible insight, thank you! Could you elaborate on any more of your experiences with PPU/SPU programming? Also, what have you gone on to do since then?

tylerhou 2y ago

You can still achieve determinism by treating the measured delta-time as an input, and resimulating the game with recorded delta-time. This may not be feasible on public servers, but is definitely feasible for professional play where determinism is important to rewind games due to bugs which affect competitive integrity.

maest 2y ago
They have so many bugs that there's a procedure for rewinding the game during comp plays? Did I misunderstand?
- pohuing 2y ago
  This is not unusual. Bugs happen and when there's a lot of money on the line you need a rollback system.
  - maest 2y ago
    Interesting. I don't think I've seen this in any other competitive game, which is why I'm surprised. But maybe I'm just lacking information
    - mackman 2y ago
      I implemented a debug replay system for an American Football game when I worked at Sony. The AI hierarchy and all animation and physics state was recorded and game could be rewound to figure out why an NPC made a poor decision. Only needed to track state for the duration of a single play. This was on PS2 which I think had 32MB of memory but the dev systems had an extra 96MB which we could use for this sort of thing.
      - chowells 2y ago
        Hell. Replay review in American Football and other sports is a form of rewinding and fixing bugs in officiating. So it's certainly done in real life when money's on the line.
    - nightowl_games 2y ago
      Starcraft 2 lets you resume a game from a replay.
    - tylerhou 2y ago
      In StarCraft II you can resume from replay (and maybe in previous games as well), but I don't know if this has ever actually been used due to bugs in the game engine during competitive play.
- IanGabes 2y ago
  They do! Its called Chronobreak. They have used it many times in professional matches successfully, but it doesn't work in 100% of scenarios.
nightowl_games 2y ago
Yeah you _could_ but I fail to see why you would ever do this. It's also not feasible to synchronize the client with the server in real time with this input.
- tylerhou 2y ago
  You don't need to synchronize the server and the client. The point of determinism is to be able to reproduce game states deterministically, not to have people play the video game deterministically. Again, this is to "undo" bugs in pro play (Chronobreak).

DarthBatman 2y ago

Thought I heard my ears ringing... Blast from the past here. That's the beauty part of recording server state and playing it back -- we included a recording of whatever accurate integer-based OS time I could get on each platform. On Windows, for example, I record the raw values of QPC ticks and the tick rate into the recording. Then on playback, I play back the clocks themselves as if the hardware has that tick rate and tick count are from the recorded computer. So if frame 10 took 33.3ms and frame 11 took 66.7ms, we'd play them back that way. Note that playbacks are almost never in real-time. The reason a typical server frame takes 33ms is that we might be processing for 5ms and then yielding the thread (sleeping) for the remaining time during real-time gameplay. On playback, we don't yield the thread, so we play back faster than real-time when doing something like a live Chronobreak (disaster recovery.) Strictly speaking, you are right. Let's say we had a game with no player inputs (like a bots-only game.) It's possible that the outcomes might vary between two games with the same random seeds if they run different numbers of frames over the same time period. That said, we've actually tried this a few times, and our game code is surprisingly resilient to this -- only in a few instances have we found issues where subtle variations in frame time could impact player outcomes (and those are generally corrected when we find them -- this is an esports game after all.) A lot of this comes down to tradeoffs. Yes we might have those subtle issues, but if we fixed our timestep, then we'd have the other problem of the game potentially going into slow-motion during (the admittedly rare) periods of long frame times. In practice, we decided time dilation is more negative for player performance than loss of interactive resolution; humans are great predictors of motion with limited information, but they're not so great when the rules of localized reality bend. Edit: typo

nightowl_games 2y ago
> It's possible that the outcomes might vary between two games So just to clarify: in League, the 'dt' passed into each 'simulation step' is NOT constant? Isnt this kinda crazy? In your later articles, floating point imprecision is talked about. Couldnt this variance in dt create odd behaviour and ultimate contribute to weird things like 'character movement speed' being not _exactly_ the same between games? (like really small, but still...) And beyond that, how does the client and the server synchronize with each other if the frame #s represent different positions in time? My mind is blown right now... Note: I've worked on many networked games, and have written rollback/resimulate/replay code. I don't really understand _why_ League wouldnt use a fixed time step. Whats the advantage? In our games, the rendering of course uses the real dt passed in from the OS, but the simulation step is always fixed. This means in our games during a replay, your computer could render the outcome differently if your frame rate is different, but the raw simulation is always the same. For context, to show I at least have some idea what I'm talking about, I made this replay system (and the game has rollback multiplayer): https://gooberdash.winterpixel.io/?play=5b74f7c0-8591-40dc-b0f4-f8e9d3c8cb57&ghost=52721394-f918-4d62-aa5d-d50fcf3e8d68 https://gooberdash.winterpixel.io/?play=5b74f7c0-8591-40dc-b... I havent played a lot of League, but I always assumed it would use deterministic lock step networking (like it's predecessor, Warcraft 3)

mackman 2y ago

The physics game sim step had to be hardcoded but you could vary the graphics rendering independently. That did mean you needed to use a different RNG for sim objects and graphics-only particle effects. I worked on the deterministic instant-replay system for a racing sim on PS3. The most interesting thing was that on PS3 we had main processor cores (PPU) and helper cores (SPU) which had their own instruction set. Our physics sim could distribute work across both, but the thread assignment had to be deterministic because PPU and SPU got marginally different floating point results for the same instructuion and inputs. That was a fun one to figure out!

hamish-b 2y ago
Incredible insight, thank you! Could you elaborate on any more of your experiences with PPU/SPU programming? Also, what have you gone on to do since then?