6 ms·

I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistak

I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistaken Broodwar replay files contain a lot less information. They mainly contain player actions, i.e. you don't even know when a marine died -- the game has to literally be replayed to get that information.

ktpsns 3y ago
The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.
- icza 3y ago
  Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals. In the new game engine no probe will be built as the player does not have enough minerals. If this new probe was used to build other buildings, those also will not be built in the new game engine. Butterfly effect.
  - rcoveson 3y ago
    > imagine the cost of Probe getting increased from 50 minerals to 60 Yes, please. - T, Z
    - jsphweid 3y ago
      Artosis alt account?! :P
    - marginalia_nu 3y ago
      60 mins, 15 gas and a forge.
  - sebosp 3y ago
    This brings another question, if I watch an old replay, does it mean it has to have the versioned game engine stored in the computer? Does it mean with every update it gets increasingly bigger? Or is there a number of supported versions you could replay? I only started playing like 3 years ago...
    - deleted 3y ago
      [deleted]
    - reaperman 3y ago
      SC2 patches to an older version if you load an old version. Then patches back to current version when you want to play again. It requires the game to restart -- and while SC2 has a feature where you can watch a live replay with your friends (by all joining the same game which is hosted as type "replay") -- that "watch replay with others" feature only works with replays which were created in the latest version because only the latest version of the client can connect to the multiplayer servers. I'm not sure if the older patches are hot-downloaded on-demand when you load an old replay, I think they actually might be -- but I don't know. You can also "resume" a replay from any point and you and a friend can take over the game from that point. I think Harstem (both a professional player and a big streamer/ YouTuber) has a few YouTube videos where he explores a famous game where one pro player "had clearly lost and had absolutely no chance to win" but the losing player stayed in the game for a really, really long time and people were talking about whether and to what degree that was "unsportsmanlike" (similar to not resigning in a clearly-lost chess game). Harstem would take the losing side at the point in time that was generally agreed to be "lost" and some of his viewers of various ability levels and IIRC he beat all of them.
      - sebosp 3y ago
        Wow that is interesting that it patches back and forth, super cool! How did you find out about this? I wonder now and then if one can "consume" the events on a live game somehow, like connect as a bot or sniff the events before they are encrypted through the network... Also curious about the drone kill counter mod/overlay that Harstem uses, do you know of the name of that? I can't figure it out but would love to enrich it with stats from the data analysis
  - beebeepka 3y ago
    Exactly. Divergence is immediate and you wasted your high waiting for something to happen, only to discover that replay was generated by another game version. There was no youtube at the time and downloading replays was the only option and the good players weren't exactly thrilled to share them.
- sebosp 3y ago
  I fantasize if training an AI to watch all the replays of a map on YouTube can create a repro of the state machine logic that moves the units through the game, or to find the parts of a map that cannot be walked upon. In the replay it emits an event when the user wants a unit to move to x, y but then it has to path its way there, and I cannot predict (without their logic) how it would move there
- sebosp 3y ago
  Btw such move instructions I visualize here with the amazing rerun.io https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO16-GroupA-Winners-Match.html?url=https://sebosp.github.io/swarmy/public/0.5.1/assets/2023-GSL-S1-RO16-GroupA-Winners-Match.rrd https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO1...
- kirrent 3y ago
  The same can happen in a live game. Apparently LAN games don't check for consistency in game state so you can just diverge over time. One of the observers in ASL desynced in the RO8 just the other day so it travelled over into remastered.
  - tec27 3y ago
    This isn't a deficiency of LAN games, it's a deficiency of the observer mode Blizzard added to the game when they remastered it. The code they wrote to deal with stealthed units as an observer mistakenly changes the actual game state when you change player vision (i.e. set the game to only show one player's vision), which leads to the game state desyncing for the observer that does this. I run a project (ShieldBattery) where we've developed numerous patches to the game including a fix for this bug, but pro games don't use our stuff so they're stuck with the bug unless Blizzard decides to maintain the game again.
    - kirrent 3y ago
      Oh, that's good to know. Shield battery is a great project. Thanks for your work.
sebosp 3y ago
Is there a general name to the way the data is compressed/serialized in SC2 Replays? Yesterday somebody was talking about other ways to serialize/pack data with borsh but not sure it's at all similar... Is the SC1 broodwar replay format similar with byte-alignment and bit "packing"? BTW I call the parsers "bitpacked" or "byte-aligned", but I'm not sure what the right name for them is...
- icza 3y ago
  SC2 replays are MPQ files, which is a proprietary format created and used by Blizzard. It's an archive that may contain multiple files stored with different compression and optionally encrypted. I wrote a lib to parse MPQ files that embodies SC2Replays: https://github.com/icza/mpq https://github.com/icza/mpq. I also wrote an SC2 replay parser that is more or less a port of the official s2protocol: https://github.com/icza/s2prot https://github.com/icza/s2prot
  - sebosp 3y ago
    I mean for example https://github.com/sebosp/s2protocol-rs/blob/755098fb86ab6b1e99f8110d62840021070a9632/src/versions/protocol87702/mod.rs#L5849 https://github.com/sebosp/s2protocol-rs/blob/755098fb86ab6b1... I hacked my way around the json protocol specification, an enum has n types, using log I can find the number of bits I need to read to uniquely identify each variant, that kind of serialization I wonder the name of.
    - icza 3y ago
      I'm not sure about an official name, I also just call it "bitpacked".
  - arwineap 3y ago
    Thank you for your work on s2prot I spent a lot of time with it and file watchers to gather stats about my matches while I sat in queues

ktpsns 3y ago

The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.

icza 3y ago
Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals. In the new game engine no probe will be built as the player does not have enough minerals. If this new probe was used to build other buildings, those also will not be built in the new game engine. Butterfly effect.
- rcoveson 3y ago
  > imagine the cost of Probe getting increased from 50 minerals to 60 Yes, please. - T, Z
  - jsphweid 3y ago
    Artosis alt account?! :P
  - marginalia_nu 3y ago
    60 mins, 15 gas and a forge.
- sebosp 3y ago
  This brings another question, if I watch an old replay, does it mean it has to have the versioned game engine stored in the computer? Does it mean with every update it gets increasingly bigger? Or is there a number of supported versions you could replay? I only started playing like 3 years ago...
  - deleted 3y ago
    [deleted]
  - reaperman 3y ago
    SC2 patches to an older version if you load an old version. Then patches back to current version when you want to play again. It requires the game to restart -- and while SC2 has a feature where you can watch a live replay with your friends (by all joining the same game which is hosted as type "replay") -- that "watch replay with others" feature only works with replays which were created in the latest version because only the latest version of the client can connect to the multiplayer servers. I'm not sure if the older patches are hot-downloaded on-demand when you load an old replay, I think they actually might be -- but I don't know. You can also "resume" a replay from any point and you and a friend can take over the game from that point. I think Harstem (both a professional player and a big streamer/ YouTuber) has a few YouTube videos where he explores a famous game where one pro player "had clearly lost and had absolutely no chance to win" but the losing player stayed in the game for a really, really long time and people were talking about whether and to what degree that was "unsportsmanlike" (similar to not resigning in a clearly-lost chess game). Harstem would take the losing side at the point in time that was generally agreed to be "lost" and some of his viewers of various ability levels and IIRC he beat all of them.
    - sebosp 3y ago
      Wow that is interesting that it patches back and forth, super cool! How did you find out about this? I wonder now and then if one can "consume" the events on a live game somehow, like connect as a bot or sniff the events before they are encrypted through the network... Also curious about the drone kill counter mod/overlay that Harstem uses, do you know of the name of that? I can't figure it out but would love to enrich it with stats from the data analysis
- beebeepka 3y ago
  Exactly. Divergence is immediate and you wasted your high waiting for something to happen, only to discover that replay was generated by another game version. There was no youtube at the time and downloading replays was the only option and the good players weren't exactly thrilled to share them.
sebosp 3y ago
I fantasize if training an AI to watch all the replays of a map on YouTube can create a repro of the state machine logic that moves the units through the game, or to find the parts of a map that cannot be walked upon. In the replay it emits an event when the user wants a unit to move to x, y but then it has to path its way there, and I cannot predict (without their logic) how it would move there
sebosp 3y ago
Btw such move instructions I visualize here with the amazing rerun.io https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO16-GroupA-Winners-Match.html?url=https://sebosp.github.io/swarmy/public/0.5.1/assets/2023-GSL-S1-RO16-GroupA-Winners-Match.rrd https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO1...
kirrent 3y ago
The same can happen in a live game. Apparently LAN games don't check for consistency in game state so you can just diverge over time. One of the observers in ASL desynced in the RO8 just the other day so it travelled over into remastered.
- tec27 3y ago
  This isn't a deficiency of LAN games, it's a deficiency of the observer mode Blizzard added to the game when they remastered it. The code they wrote to deal with stealthed units as an observer mistakenly changes the actual game state when you change player vision (i.e. set the game to only show one player's vision), which leads to the game state desyncing for the observer that does this. I run a project (ShieldBattery) where we've developed numerous patches to the game including a fix for this bug, but pro games don't use our stuff so they're stuck with the bug unless Blizzard decides to maintain the game again.
  - kirrent 3y ago
    Oh, that's good to know. Shield battery is a great project. Thanks for your work.

sebosp 3y ago

Is there a general name to the way the data is compressed/serialized in SC2 Replays? Yesterday somebody was talking about other ways to serialize/pack data with borsh but not sure it's at all similar... Is the SC1 broodwar replay format similar with byte-alignment and bit "packing"? BTW I call the parsers "bitpacked" or "byte-aligned", but I'm not sure what the right name for them is...

icza 3y ago
SC2 replays are MPQ files, which is a proprietary format created and used by Blizzard. It's an archive that may contain multiple files stored with different compression and optionally encrypted. I wrote a lib to parse MPQ files that embodies SC2Replays: https://github.com/icza/mpq https://github.com/icza/mpq. I also wrote an SC2 replay parser that is more or less a port of the official s2protocol: https://github.com/icza/s2prot https://github.com/icza/s2prot
- sebosp 3y ago
  I mean for example https://github.com/sebosp/s2protocol-rs/blob/755098fb86ab6b1e99f8110d62840021070a9632/src/versions/protocol87702/mod.rs#L5849 https://github.com/sebosp/s2protocol-rs/blob/755098fb86ab6b1... I hacked my way around the json protocol specification, an enum has n types, using log I can find the number of bits I need to read to uniquely identify each variant, that kind of serialization I wonder the name of.
  - icza 3y ago
    I'm not sure about an official name, I also just call it "bitpacked".
- arwineap 3y ago
  Thank you for your work on s2prot I spent a lot of time with it and file watchers to gather stats about my matches while I sat in queues