4 ms·
This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which c
by lclarkmichalek 13y ago
This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over.
Before someone comes along and says "this isn't big data!", I know. It's medium data at best. However, we are bound by CPU in a big way, so between throwing more cores at the problem and rewriting everything we can in C, we think we can reduce processing times to an acceptable point (currently about ~4 mins, hoping to hit <30s).
- baudehlo 13y agoThis sounds like something you could just do in SQL and have it all done in milliseconds.
- collyw 13y agoMost of the NoSQL cases I have heard seem to be that they could be done at least as well in SQL. I have asked a lot of questions on this topic, and no one has yet convinced me (please do if you have a legitimate NoSQL case).
- msellout 13y agoExecution time, after a point, is less important than development time. NoSQL is often faster for development and refactoring, because the schema is easier to change in code than in the database. Also there's the scale issue. Append-only is helpful. Why have SQL if you can't use all the features?
- lclarkmichalek 13y agoFor our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.
- alecco 13y agoMaybe you should consider transforming the data to structure it a bit. Have a unique identifier per object and arrays for each seen characteristic with (value, id), and it's reverse index. Then decompose the processing of each object to sub-problems matching n-way the characteristics. It doesn't have to be in SQL, though. I'd try a columnar RDBMS. YMMV
- baudehlo 13y agoI have extremely strong doubts about that. Everything can be modelled in modern databases, and SQL is probably much more powerful than you understand here.
- ameen 13y agoAre you talking about match stastics or player stats? Dota 2 has a logging system that records various events? during a match.
- prawks 13y agoReplays contain more detailed data (I believe) like player positioning and actions. I think logging is pretty limited.
- lclarkmichalek 13y agoMatch stats atm, though we can get much everything. We can go far beyond the combat log, reproducing an entire minimap, along with things like hero position trails, map control, etc etc. Here's a screenshot of a minimap recreation we produced for the recent Fragbite Masters tournament: http://i.imgur.com/kuOvUZX.jpg http://i.imgur.com/kuOvUZX.jpg. We have more complex tools such as XP and Gold source breakdowns over time (i.e. jungle creeps, lane creeps, kills, assists etc), along with other fun stuff still in the pipeline.
- tumanian 13y agoWait, it takes more then a few minutes to spin up the hadoop mapreduce job. Are you using mapreduce for preprocessing or did you optimize hadoop for this scenario?
- StavrosK 13y agoWhich service is this for? It sounds very interesting.