7 ms·

Show HN: A nom parser for the Starcraft 2 Protocol Replay format

153 points by sebosp 3y ago

Been having a lot of fun reading an SC2Replay collection through nom parsers, serializing into Arrow files so that pola.rs can read them and perform data analysis with jupyter lab, plotly or interact with SQL operations, etc. Looking for feedback and ideas on what to progress on. For example, "through history, are my timings getting better?". etc. Also would love to have ideas on what libraries to use to perform forecasting.

benmanns 3y ago
I may be the only one not familiar, but nom refers to https://github.com/rust-bakery/nom https://github.com/rust-bakery/nom which looks like a pretty handy way to parse binary data in Rust.
- brundolf 3y ago
  It's a pretty handy way to parse any sequence of things in Rust! I'm using it to write a compiler for a toy language and it's been a delight
  - karelpeeters 3y ago
    Could you share the repo? I've been curious what nontrivial parsers look like woth nom, I always lose the overview pretty quickly.
    - TimWolla 3y ago
      I'm using nom to parse a simple access control DSL in https://github.com/wbbaddons/Tims-PackageServer/blob/master/src/auth/acl/ruleset.rs https://github.com/wbbaddons/Tims-PackageServer/blob/master/.... You can see how it looks in the tests of the linked file. The DSL supports comparison operators, '&&', '||' and nested expressions, while preventing mixing of '&&' and '||' without making precedence clear using parentheses. This DSL should fit 'non-trivial', but it still should be simple enough to easily understand it.
  - IshKebab 3y ago
    I would consider Chumsky for a programming language. It has recoverable parsing which is important for IDE support.
- deleted 3y ago
  [deleted]
- epage 3y ago
  For a very rough comparison of parsers, see https://github.com/rosetta-rs/parse-rosetta-rs https://github.com/rosetta-rs/parse-rosetta-rs (which both includes a full json parser for each parser)
jsphweid 3y ago
I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistaken Broodwar replay files contain a lot less information. They mainly contain player actions, i.e. you don't even know when a marine died -- the game has to literally be replayed to get that information.
- ktpsns 3y ago
  The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.
  - icza 3y ago
    Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals. In the new game engine no probe will be built as the player does not have enough minerals. If this new probe was used to build other buildings, those also will not be built in the new game engine. Butterfly effect.
    - rcoveson 3y ago
      > imagine the cost of Probe getting increased from 50 minerals to 60 Yes, please. - T, Z
      - jsphweid 3y ago
        Artosis alt account?! :P
      - marginalia_nu 3y ago
tornato7 3y ago
This is a super cool project! It would be especially interesting to use on pro-level games, maybe you could even sell services to pros who want to get better. Like being able to say "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" etc. With a relatively larger dataset you could come up with some real interesting statistics on individual players performance, use of certain units, the success of various strategies and build orders, etc. It would be real fun to try to predict the outcome of a game based on the first 3 minutes or something.
- sebosp 3y ago
  Indeed, another thing is that imho, casters and you tubers are data scientists that have the indexed dB in their heads, they say "at second X in this map Y you would have Z upgrade" and they talk about efficiency of mining, but I always wanted to see these things as statistics visually, like, how much faster or slower is player X in this game compared to the standard? How does the meta evolve? Or worse... If I check your replays, and you never ever scout this area, I could proxy there. You could find the build that in this map you never prepare for because let's be honest we are much more predictable than what we'd like to admit... I hope this project unlocks these kind of possibilities
- jsphweid 3y ago
  I'm messing around with a project now to featurize game states at arbitrary points and predict the winner with the idea of creating some sort of "eval bar" like in chess. Supply is a pretty good indicator but it's not always helpful. Definitely not in broodwar. If it's quite good it could be useful as a tool for people trying to get better (although various tooling and custom maps have been around for these types of purposes since the game came out and most people even pros seem to just grind ladder for practice). I was also hoping it could be used to analyze a replay and indicate how "back and forth" the game was, or if it was more one-sided, etc. Or if the top 100 pros were part of the featurization, then you could potentially determine how the odds change given a certain game state depending on who the pro is.
- reaperman 3y ago
  https://sc2replaystats.com https://sc2replaystats.com is one service already doing that, at least to some degree. Some features are locked behind a subscription payment. There's definitely room for improvement -- but some of what you're looking for is "unsolved"...optimal strategies are unknown, and the best strategy for a professional player may not be the best strategy for you. You'd likely want to practice strategies which won't get you top-100 in the world, but will work better with lower APM and lower overall "game sense".
comagoosie 3y ago
Thanks for sharing, very inspiring. I love Rust for parsing video game replays / save files. I've authored a Rocket League replay parser (boxcars) and an entire suite of web visualizations (via Webassembly) for EU4 called pdx.tools https://pdx.tools https://pdx.tools It's not easy to work with proprietary formats, but they've both become pretty popular, so I would 100% recommend sinking more time into this project as long as it scratches your itch. Gamers are always looking for more stats and deeper insights
- sebosp 3y ago
  Have you seen rerun.io? Also web assembly, sharing again the link from above https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO16-GroupA-Winners-Match.html?url=https://sebosp.github.io/swarmy/public/0.5.1/assets/2023-GSL-S1-RO16-GroupA-Winners-Match.rrd https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO1... Maybe you can use it too in your project! The guys from rerun are amazing and super helpful, had a call with them to discuss questions and possible uses too
  - comagoosie 3y ago
    No joke, rerun seems perfect for visualizing a game like rocket league. Thank you for the recommendation!
deleted 3y ago
[deleted]
munk-a 3y ago
It'd be great to use a non-Blizzard property for this analysis since that company is outstandingly despicable. Plenty of other RTSes exist and I know that AoE2 has an extremely strong data analytics scene.
- 0cf8612b2e1e 3y ago
  Where would I go to see some of these analyses?
  - munk-a 3y ago
    A few places but I'd suggest starting with AOE stats[1]. 1. https://aoestats.io/ https://aoestats.io/
celdon25 3y ago
I'm curious to see how well this works with Direct Strike (custom game mode for SC2)
- sebosp 3y ago
  I only had one replay with direct strike in it's name but I can't remember watching it, I thought it was just one more map... I know it was failing initially to decode but in the end I fixed a couple of bugs and then it was working with this file. Could you explain what is significant go direct strike? How is it different to a normal game?
  - celdon25 3y ago
    It has an active competitive scene, lobbies fill up in seconds sometimes. There are also tools such as dsstats that attempt to parse the replays already, but are limited by technology (maybe your tool can help with the limitations). If you search Direct Strike Starcraft on youtube you should be able to see the gameplay itself.
Rodeoclash 3y ago
This is very cool, I'd love to chat more about it. For context, I have an open source video player designed for esports coaches. The main feature is that you can load in multiple video streams at once at synchronise them together. Mainly for FPS games like Valorant / Apex Legends (https://www.vodon.gg/ https://www.vodon.gg/ if you want to explore it). I'm starting to get access to some streams of data from the games via coaches that use the tool. My very naïve approach was simply to load game events into the video timeline (so you could easily skip forwards to deaths, kills etc) but I hadn't thought about loading this into data analysis tools. The game events themselves for Valorant seem like they'd be enough to almost construct an online replay from them as well, which could compliment the recorded gameplay (i.e. construct a dynamic map of where everybody is that could be brought up on screen). It's a very cool space, if you'd like to chat more my email is in my profile.
ZephyrBlu 3y ago
This is insanely cool! Very impressed you managed to implement a full parser in Rust. I implemented a basic one in Rust a while back: https://github.com/ZephyrBlu/rust-parser https://github.com/ZephyrBlu/rust-parser And a full one in Python with a few bells and whistles ages ago: https://github.com/ZephyrBlu/zephyrus-sc2-parser https://github.com/ZephyrBlu/zephyrus-sc2-parser Don't maintain either of them though :(, and the Rust one is super rough. SC2 is a very interesting area for data analysis, but at the same time I found it very challenging. There is so much nuance and inconsistency across games it can be really hard to do accurately do things like categorize builds or measure build timings. The area I ended up focusing on was builds, and I feel like I did some interesting stuff there: https://sc2.gg/reports/top-openings-2022/ https://sc2.gg/reports/top-openings-2022/. I found personal statistics less interesting than aggregate statistics. Even pro games are very volatile, ladder games even more so. Extremely hard to get reliable signal out of them if you're trying to track things across games. Even simple things like Collection Rate are poor indicators without significant categorization work (Matchup, build, opponent build, etc).
Kaszanas 3y ago
I am currently conducting research on esprots as a whole with the main target being StarCraft since around 2019. I was able to publish a nice dataset of pro replays, and resulting JSON files after using one of the Golang parsers written by Icza. https://www.nature.com/articles/s41597-023-02510-7 https://www.nature.com/articles/s41597-023-02510-7 Feel free to look into the tools on my GitHub (https://github.com/Kaszanas https://github.com/Kaszanas). Since this is mostly the topic of my PhD I guess I will be updating the dataset in the near future. You may want to try and test your parser against it. further research for you would probably include running Logistic Regression on aggregated data from each of the replays to try and have a model that can discern between winners and losers and see which parameters are key in your data. Example: https://www.researchgate.net/publication/363613604_Determinants_of_victory_in_Esports_-_StarCraft_II https://www.researchgate.net/publication/363613604_Determina... And even further embedding the games as timeseries data via various methods.
dleeftink 3y ago
On the data analytical site, I always wondered if complex game state patterns could be deconstructed using techniques from the JKU Visual Data Science Lab [0][1]. Here they demonstrate chess board win states from competitive data, and I wonder what the state relationships would look like for different (video) game matches and if new strategies could be derived from it. [0]: https://youtu.be/yBCe8SqGwK8 https://youtu.be/yBCe8SqGwK8 [1]: https://jku-vds-lab.at/publications/2022_embedding_structure/ https://jku-vds-lab.at/publications/2022_embedding_structure...
noobly 3y ago
Wow, nice. Bookmarked for when I get my desktop back up.
mmkkokk 3y ago
[flagged]

153 points by sebosp 3y ago

benmanns 3y ago

I may be the only one not familiar, but nom refers to https://github.com/rust-bakery/nom https://github.com/rust-bakery/nom which looks like a pretty handy way to parse binary data in Rust.

brundolf 3y ago
It's a pretty handy way to parse any sequence of things in Rust! I'm using it to write a compiler for a toy language and it's been a delight
- karelpeeters 3y ago
  Could you share the repo? I've been curious what nontrivial parsers look like woth nom, I always lose the overview pretty quickly.
  - TimWolla 3y ago
    I'm using nom to parse a simple access control DSL in https://github.com/wbbaddons/Tims-PackageServer/blob/master/src/auth/acl/ruleset.rs https://github.com/wbbaddons/Tims-PackageServer/blob/master/.... You can see how it looks in the tests of the linked file. The DSL supports comparison operators, '&&', '||' and nested expressions, while preventing mixing of '&&' and '||' without making precedence clear using parentheses. This DSL should fit 'non-trivial', but it still should be simple enough to easily understand it.
- IshKebab 3y ago
  I would consider Chumsky for a programming language. It has recoverable parsing which is important for IDE support.
deleted 3y ago
[deleted]
epage 3y ago
For a very rough comparison of parsers, see https://github.com/rosetta-rs/parse-rosetta-rs https://github.com/rosetta-rs/parse-rosetta-rs (which both includes a full json parser for each parser)

jsphweid 3y ago

I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistaken Broodwar replay files contain a lot less information. They mainly contain player actions, i.e. you don't even know when a marine died -- the game has to literally be replayed to get that information.

ktpsns 3y ago
The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.
- icza 3y ago
  Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals. In the new game engine no probe will be built as the player does not have enough minerals. If this new probe was used to build other buildings, those also will not be built in the new game engine. Butterfly effect.
  - rcoveson 3y ago
    > imagine the cost of Probe getting increased from 50 minerals to 60 Yes, please. - T, Z
    - jsphweid 3y ago
      Artosis alt account?! :P
    - marginalia_nu 3y ago

tornato7 3y ago

This is a super cool project! It would be especially interesting to use on pro-level games, maybe you could even sell services to pros who want to get better. Like being able to say "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" etc. With a relatively larger dataset you could come up with some real interesting statistics on individual players performance, use of certain units, the success of various strategies and build orders, etc. It would be real fun to try to predict the outcome of a game based on the first 3 minutes or something.

sebosp 3y ago
Indeed, another thing is that imho, casters and you tubers are data scientists that have the indexed dB in their heads, they say "at second X in this map Y you would have Z upgrade" and they talk about efficiency of mining, but I always wanted to see these things as statistics visually, like, how much faster or slower is player X in this game compared to the standard? How does the meta evolve? Or worse... If I check your replays, and you never ever scout this area, I could proxy there. You could find the build that in this map you never prepare for because let's be honest we are much more predictable than what we'd like to admit... I hope this project unlocks these kind of possibilities
jsphweid 3y ago
I'm messing around with a project now to featurize game states at arbitrary points and predict the winner with the idea of creating some sort of "eval bar" like in chess. Supply is a pretty good indicator but it's not always helpful. Definitely not in broodwar. If it's quite good it could be useful as a tool for people trying to get better (although various tooling and custom maps have been around for these types of purposes since the game came out and most people even pros seem to just grind ladder for practice). I was also hoping it could be used to analyze a replay and indicate how "back and forth" the game was, or if it was more one-sided, etc. Or if the top 100 pros were part of the featurization, then you could potentially determine how the odds change given a certain game state depending on who the pro is.
reaperman 3y ago
https://sc2replaystats.com https://sc2replaystats.com is one service already doing that, at least to some degree. Some features are locked behind a subscription payment. There's definitely room for improvement -- but some of what you're looking for is "unsolved"...optimal strategies are unknown, and the best strategy for a professional player may not be the best strategy for you. You'd likely want to practice strategies which won't get you top-100 in the world, but will work better with lower APM and lower overall "game sense".

comagoosie 3y ago

Thanks for sharing, very inspiring. I love Rust for parsing video game replays / save files. I've authored a Rocket League replay parser (boxcars) and an entire suite of web visualizations (via Webassembly) for EU4 called pdx.tools https://pdx.tools https://pdx.tools It's not easy to work with proprietary formats, but they've both become pretty popular, so I would 100% recommend sinking more time into this project as long as it scratches your itch. Gamers are always looking for more stats and deeper insights

sebosp 3y ago
Have you seen rerun.io? Also web assembly, sharing again the link from above https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO16-GroupA-Winners-Match.html?url=https://sebosp.github.io/swarmy/public/0.5.1/assets/2023-GSL-S1-RO16-GroupA-Winners-Match.rrd https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO1... Maybe you can use it too in your project! The guys from rerun are amazing and super helpful, had a call with them to discuss questions and possible uses too
- comagoosie 3y ago
  No joke, rerun seems perfect for visualizing a game like rocket league. Thank you for the recommendation!

deleted 3y ago

[deleted]

munk-a 3y ago

It'd be great to use a non-Blizzard property for this analysis since that company is outstandingly despicable. Plenty of other RTSes exist and I know that AoE2 has an extremely strong data analytics scene.

0cf8612b2e1e 3y ago
Where would I go to see some of these analyses?
- munk-a 3y ago
  A few places but I'd suggest starting with AOE stats[1]. 1. https://aoestats.io/ https://aoestats.io/

celdon25 3y ago

I'm curious to see how well this works with Direct Strike (custom game mode for SC2)

sebosp 3y ago
I only had one replay with direct strike in it's name but I can't remember watching it, I thought it was just one more map... I know it was failing initially to decode but in the end I fixed a couple of bugs and then it was working with this file. Could you explain what is significant go direct strike? How is it different to a normal game?
- celdon25 3y ago
  It has an active competitive scene, lobbies fill up in seconds sometimes. There are also tools such as dsstats that attempt to parse the replays already, but are limited by technology (maybe your tool can help with the limitations). If you search Direct Strike Starcraft on youtube you should be able to see the gameplay itself.

Rodeoclash 3y ago

This is very cool, I'd love to chat more about it. For context, I have an open source video player designed for esports coaches. The main feature is that you can load in multiple video streams at once at synchronise them together. Mainly for FPS games like Valorant / Apex Legends (https://www.vodon.gg/ https://www.vodon.gg/ if you want to explore it). I'm starting to get access to some streams of data from the games via coaches that use the tool. My very naïve approach was simply to load game events into the video timeline (so you could easily skip forwards to deaths, kills etc) but I hadn't thought about loading this into data analysis tools. The game events themselves for Valorant seem like they'd be enough to almost construct an online replay from them as well, which could compliment the recorded gameplay (i.e. construct a dynamic map of where everybody is that could be brought up on screen). It's a very cool space, if you'd like to chat more my email is in my profile.

ZephyrBlu 3y ago

This is insanely cool! Very impressed you managed to implement a full parser in Rust. I implemented a basic one in Rust a while back: https://github.com/ZephyrBlu/rust-parser https://github.com/ZephyrBlu/rust-parser And a full one in Python with a few bells and whistles ages ago: https://github.com/ZephyrBlu/zephyrus-sc2-parser https://github.com/ZephyrBlu/zephyrus-sc2-parser Don't maintain either of them though :(, and the Rust one is super rough. SC2 is a very interesting area for data analysis, but at the same time I found it very challenging. There is so much nuance and inconsistency across games it can be really hard to do accurately do things like categorize builds or measure build timings. The area I ended up focusing on was builds, and I feel like I did some interesting stuff there: https://sc2.gg/reports/top-openings-2022/ https://sc2.gg/reports/top-openings-2022/. I found personal statistics less interesting than aggregate statistics. Even pro games are very volatile, ladder games even more so. Extremely hard to get reliable signal out of them if you're trying to track things across games. Even simple things like Collection Rate are poor indicators without significant categorization work (Matchup, build, opponent build, etc).

Kaszanas 3y ago

I am currently conducting research on esprots as a whole with the main target being StarCraft since around 2019. I was able to publish a nice dataset of pro replays, and resulting JSON files after using one of the Golang parsers written by Icza. https://www.nature.com/articles/s41597-023-02510-7 https://www.nature.com/articles/s41597-023-02510-7 Feel free to look into the tools on my GitHub (https://github.com/Kaszanas https://github.com/Kaszanas). Since this is mostly the topic of my PhD I guess I will be updating the dataset in the near future. You may want to try and test your parser against it. further research for you would probably include running Logistic Regression on aggregated data from each of the replays to try and have a model that can discern between winners and losers and see which parameters are key in your data. Example: https://www.researchgate.net/publication/363613604_Determinants_of_victory_in_Esports_-_StarCraft_II https://www.researchgate.net/publication/363613604_Determina... And even further embedding the games as timeseries data via various methods.

dleeftink 3y ago

On the data analytical site, I always wondered if complex game state patterns could be deconstructed using techniques from the JKU Visual Data Science Lab [0][1]. Here they demonstrate chess board win states from competitive data, and I wonder what the state relationships would look like for different (video) game matches and if new strategies could be derived from it. [0]: https://youtu.be/yBCe8SqGwK8 https://youtu.be/yBCe8SqGwK8 [1]: https://jku-vds-lab.at/publications/2022_embedding_structure/ https://jku-vds-lab.at/publications/2022_embedding_structure...

noobly 3y ago

Wow, nice. Bookmarked for when I get my desktop back up.

mmkkokk 3y ago

[flagged]