8 ms·

MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

234 points by buba 3y ago

buba 3y ago
Interesting aproach to data ownership philosophy, I/O techniques and source of truth fuckery in MMO-like systems
- hiatus 3y ago
  Are you the author?
  - buba 3y ago
    yep
    - Uehreka 3y ago
      …and you wrote a comment complimenting your own article?
      - xwowsersx 3y ago
        haha, no I think they were giving a heads up to potential readers as to what they think is the interesting or unique points in the post. Still funny though :)
      - buba 3y ago
        Yep, I noticed. It's an interesting approach tho xD
      - aleph_minus_one 3y ago
        The author is either doing SEO or wants to teach the AIs why his approach is great. :-)
- zinodaur 3y ago
  I liked the article! Do you know of MMOs that try to use database techniques like write-ahead-logs and log-sequence-numbers for persistence/replication? The nice thing about these techniques is that you can replicate state in a consistent way - so you could have multiple game state services all providing equivalent reads
Eumenes 3y ago
Early MMOs (WoW, Asheron's Call, Everquest, Daoc) were very impressive in terms of distributed computing.
- Thaxll 3y ago
  They were not really distributed though, I think one of the first that really started was Guild Wars 2. https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Presentations/Clarke-Willson_Guild%20Wars%202%20microservices.pdf https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Pres... DAoC was basically a Linux box with a bunch of processes connected to MySQL. https://www.gamedeveloper.com/disciplines/postmortem-mythic-entertainment-s-i-dark-age-of-camelot-i- https://www.gamedeveloper.com/disciplines/postmortem-mythic-...
  - deleted 3y ago
    [deleted]
  - Eumenes 3y ago
    Would this count? https://www.gamedeveloper.com/design/classic-postmortem-i-asheron-s-call-i- https://www.gamedeveloper.com/design/classic-postmortem-i-as... > One the most impressive features of the Turbine engine is the continuous outdoor environment. This is made possible thanks to dynamic load balancing, which is a scalable serverside architecture. The easiest way to appreciate the need for dynamic load balancing is to consider the following scenario. > Dynamic load balancing solves this overloaded server problem. Instead of assigning a static geographic area to each server, the individual servers can divide up the game world based on the relative processor load of each server. In the previous example, instead of remaining idle, all four servers would divide the load equally among themselves, ensuring the most efficient use of the hardware’s processing capacity.
    - Animats 3y ago
      That's a useful technology. Second Life / Open Simulator do not have that, and need it. It's good to hear about a success with that approach. Improbable tried that, dividing the world into regions but moving the region boundaries around based on player density. This worked, but apparently required huge amounts of inter-server traffic. The system was too expensive to operate. (Running it on Google Cloud with metering for every client/server transaction didn't help.) Five indy free to play games, some of them good (look up Worlds Adrift), went bust because of server cost. Improbable then pivoted to simulators for the UK military, a much less cost-sensitive market. That worked, but they had way too much company and funding for that niche. Then they tried to pivot to crypto metaverses, two years too late, and hooked up with the Yuga Labs (Bored Ape, Otherside) crowd. Lately, they're trying to do something with US Major League Baseball. Their solution to the cost problem is to only run special events that last a few hours, for which they can short term rent some huge number of servers from AWS or somebody. There's still no good off the shelf solution for this kind of scaling, with big worlds and big moving crowds. Epic and Roblox were making noises about working on this problem a year ago, but not much has been heard recently. Now both are in money-losing and layoff mode.
quadral 3y ago
You missed the opportunity to talk about WoW private servers like Trinitycore. Trinitycore emulator can handle 10k+ players on a single server.
- buba 3y ago
  to be honest, i know nothing about private WoW servers but i promise i'll check it out! Thanks!
  - sshagent 3y ago
    azerothcore is probably the best and most polished, if you don't mind wotlk
    - sleepybrett 3y ago
      the last expansion that was any good?
      - PartiallyTyped 3y ago
        Depending on who you ask. Tangent: Imho, the only reason it is good is because it's not as grindy and / or the community just didn't put as much emphasis on min-maxing things. GearScore was a thing of course, but theory crafting wasn't anywhere close to what we have now.
        sleepybrett 3y ago
        Not sure that's true for me, very good raids (minus the trial). Ulduar and Icecrown being the highlights for my guild. Though didn't mind the single boss ones either. Trial was a little janky though but the fights were fun. The big progression guild we were part of up through burning crusade wanted to leave for greener server pastures, a handful of us were kinda done with hard progression and the cuthroat nature of it. Picked up a few more and did 10mans/hardmodes mostly. Hooked up with another guild like us for 25man but we only cleared those hard modes once.. 25man was still a management nightmare... Played through pandaland, skipped warlords, came back for some of legion and then finally broke the habit.
monlockandkey 3y ago
Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.
- hmmokidk 3y ago
  Just keep it simple and avoid optimization like the plague. you can spend 1 month building a prototype or >3 months perfecting a single piece. Just know that it is all smoke and mirrors and don’t worry about that.
- 63 3y ago
  I'd wager a guess that getting players will be the most difficult part by far, at least in the beginning. Make an MVP and focus on building a playerbase first, then come back to architecture when you're suffering from success if you get that far.
  - yetihehe 3y ago
    > then come back to architecture when you're suffering from success if you get that far. Then it will be too late because you will essentially have to rewrite half of your project while your userbase is leaving due to unplayable game. Better to make good architectural decisions from the start, and make small optimizations when needed.
- buba 3y ago
  I've some other posts planned about this topic, I don't know when or even if im going to deliver, but you are free to follow the blog and receive the update if I ever do.
- net_ 3y ago
  I made a video series on networking theory for virtual worlds that has been well received: https://youtu.be/0wOZusuMIIM https://youtu.be/0wOZusuMIIM I've also been working on an engine for the past few years if you want some code examples: https://github.com/Net5F/AmalgamEngine https://github.com/Net5F/AmalgamEngine
Kiro 3y ago
> the I/O bottleneck in the database I thought most MMOs kept everything in memory and only offloaded to the database periodically. I clearly remember rollbacks to fix times (XX:00) when things went down. Edit: Sorry, should have read the whole article before commenting.
- izend 3y ago
  Exactly, there is no way the state can be persisted on absolutely every change. It has to be periodically dumped to the database.
justinlloyd 3y ago
The "cache, lots of cache" statement is the most true of any MMO architecture we can build. I did some optimization work earlier this year on a project where the single back-end server is now handling 2 billion requests per minute and had around 3TB of RAM for cache (I think the final production system was aiming for 12TB of RAM). There's concerns around race conditions as you pointed out, message passing from client to server, and server to server, client hand-off between sharded servers. Those synchronization problems will haunt your dreams. I think the biggest issue I still struggle with is tracking those ephemeral problems that only happen on one shard, or only when going between this shard and this shard, but not the other way. One useful trick is obviously message prioritization and different messages heading to different servers - though these days I'd put a message router in front of the shards and the router handles persistent connections other than the usual technique of direct connection I've employed in the past. Contributed to an MMO game that involves waving light sabers around, another where you defeat the ultimate prime evil (though I was more on the fraud detection on that one), an unpublished MMO that unceremoniously died during the 2008 financial crash, a "shared world" game that involved animals, an open-world game that involves driving cars and running pedestrians over, a few "internet scale" websites, and am currently lead back-end on another MMO - though our database requirements are relatively simple this time around, but it is still again, read-at-start-up, write-only-when-necessary.
- Stevvo 3y ago
  How did they get the architecture so wrong on that "open-world game that involves driving cars and running pedestrians over"?
  - dmoy 3y ago
    I assume that is a tongue-in-cheek description for Grand Theft Auto (whichever one they made online). That game series has had various moral discussions / controversy surrounding it since before the first game came out like 25 years ago.
    - anon56 3y ago
Animats 3y ago
I wrote about some of these issues client-side, in a previous post about a Rust metaverse client. This is a much worse problem in a metaverse system, because there are no static game level maps. Every object in the world is in a database somewhere. Second Life / Open Simulator makes a big distinction between assets, inventories and area state. Assets (meshes, textures, animations, sounds) are immutable, and are stored more or less permanently. (There's a garbage collection batch job that runs monthly or so) Those are basically files. There's a vanilla web service running on an AWS web server, and Akamai, both heavily cached. Inventories are like file directories. They have asset UUIDs and some metadata (name, etc.) Those are in a database, but that data is dynamic and not cached. Each user has an inventory, of course, and it can be huge. 50,000 items are not unheard of. This is a metaverse; you can build stuff. Area state is in server memory for each region. That's saved periodically, once a minute or so. This is a backup file, not a database. If you wanted a more continuous save process, you could keep a log of recent changes on a different machine than the server. After a crash, reload the server state and rerun the recent changes. Area state is under a gigabyte per region (A region is 256x256 meters). With a three level system like this, none of the levels are severely overloaded. The greatest data volume is from the asset store, and because that's immutable, it can be and is cached extensively. There are three levels of caches - asset server, CDN, and client. It's still a problem getting assets out to the clients fast enough, but with prioritization and concurrency, that's solveable. The inventory database is mostly-read, so the usual scaling techniques for mostly-read databases work. Area state is in memory. The main trick is taking a clean backup without visibly freezing the system.
mannyv 3y ago
In most data architectures the DB is only the backing store, because no matter how fast your database is it's going to be slower than RAM. Once you start caring about the performance the second thing you do is stick a cache layer of one sort or another in front of the database; the first thing should be making sure you have the correct indexes. In any case, it sounds like a distributed cache problem. I wonder if you could just abuse redis for your game backend?
- kylestlb 3y ago
  may help to read the article, redis is mentioned
- justinlloyd 3y ago
  You can use redis or memcached, but every MMO or online game I've been involved with, unless it was a "web game", has eschewed those for the most part. The game server maintains the state, knows all the objects in the universe, or at least its portion of the universe, and is responsible for retrieving and updating those objects. Even redis and memcached would be considered slow by comparison. Those game objects/world objects/MOBs may eventually be pushed out to a key-store server, but generally are not. The only portion of the database on any MMO I've worked on that has cared about "proper indexes" has been the area dealing with account retrieval. Traditional databases, at least on the non-web MMOs I've been involved with, when it comes to game state, are not normally used. RDBMS are used for boring things like account management, customer management, and so forth. Our database on the current (non-web) MMO uses a few more web technologies than I have in the past for this particular problem, but once the shard is loaded, and the user is connected, it is back to tradition, for the most part.
  - machiaweliczny 3y ago
    What protocols are used to stream updates to client? Or it’s simulation on client and state dump on fails?
whartung 3y ago
Speaking about WoW specifically, since I'm not familiar with the others, I've always been curious about their quest system. Specifically keeping track of what are available for the character efficiently along with the event system to flag quests as completed, etc. There's so many of them. I have to assume they're spatially limited. You enter a zone, or an area, and the system loads up all of the quests located in that space, then it runs through to determine whether you qualify for them. As for quests that you're on, that's a bit more straightforward, since you're so limited to how many you can carry around with you at any one time. Then, every event can practically just be brute forced across your pending quests to see which ones get progressed, etc. But it was always a curiosity to me considering the magnitude of the quests available how most anything can trigger quest progress. There's also the whole achievement system, which perhaps is similar in design.
- mtve 3y ago
  Sorry in advance for not answering right to your question, but you may check sources of ManGOS/TrinityCore/family WoW servers for that. In short, from what I know, yes, quests are stored in "quest log" fields of character data in server DB, and they are tracked by the clients and checked by the server. Some simple auto quests like "find this item" are not even tracked by server and only stored on completion. Since both client and server have all game data, the client knows about all possible quests and only shows to the player what is appropriate at a current state.
- genocidicbunny 3y ago
  > Specifically keeping track of what are available for the character efficiently along with the event system to flag quests as completed, etc. This is not necessarily that difficult, at least the first part. A lot of games will have quests be given out by a 'quest giver' character of some sort, or they will activate at specific interaction points on the map. You can do some cheap 'has-player-finished-quest' type of checks to determine if for example the quest giver has some sort of UI to indicate they have a quest available that activate when the quest giver first comes into view range. Quests with more initial conditions can hide their checks behind the interaction with the quest giver. Doing quest progression can be a bit more challenging. You need to determine when to do the checks for progress, and also how comprehensive you want them to be. The more complex the check, the less often you can run it without affecting game performance. I've seen designers use all sorts of tricks depending on the specific quest. Interaction volumes that run checks, periodic ticks, on entity flash messages..etc. > Then, every event can practically just be brute forced across your pending quests to see which ones get progressed This only works for games that have a small number of active quests and not a lot of events. And with MMO's, you really need to be considerate of the accidental quadratic performance problem.
gabereiser 3y ago
Oh I feel for the author. I’ve made games. I’ve made distributed web platforms. I have profound respect for modern mmo architectures because of one, nasty, “I wish this wasn’t a thing” class of data. State. Who, where, what animation, what modeled entity, is in my party, on my screen, under my axe. Synchronized playback of my swing to my party members so we all yell in excitement at the same time when the boss falls. This level of synchronization across shards (server clusters of servers) is enormously complex. Not to mention just writing “net code” in general. Network speed is the biggest issue and often TCP isn’t enough. You need network prediction. Where will they be based on position, direction, etc until I receive the next packet. I can then error check the prediction with the actual and correct. If UDP is available to you, you use it so you can deliver that state as fast as possible, with no ACK back and forth. A combination of UDP state transfer peer2peer for animation and basic state, TCP network connections for services and server state, REST for that auction house. SQS or pub/sub for that item delivery and party/match making/world chat. It’s a beast of a problem. Rewind 15-20 years ago and all the folks who wanted to make a game, their first game, and they want to build an mmo. None of them succeeded. Not 1. The only ones since were from people who knew the ask. Or had a crowdfunded ponzi scheme.
- Strom 3y ago
  > Rewind 15-20 years ago and all the folks who wanted to make a game, their first game, and they want to build an mmo. None of them succeeded. Not 1. I guess it depends on how you define success, but I would posit FOnline [1][2] as a success story. FOnline is a fan made MMO of Fallout by a single guy, using the assets of the original Fallout 1 & 2 single-player games. Having these assets and also general game mechanics already finished definitely played a huge role in getting it to a playable state in reasonable time. Still, FOnline is a from scratch code base not a mod of the originals. Also it changed plenty of mechanics too, most notably being real-time while the original games were turn-based. It's still being pushed forward even today after 20 years of development by this one guy but it was playable in late 2000s already. Peak concurrent players that I remember seeing was a few thousand. Definitely not AAA level, but way past simple multiplayer. Would have gone higher due to the hype at the time, but the server started to really struggle at that point. After a few years of being a closed source free game it got converted into a SDK and spawned a dozen new fan games using that engine. Perhaps even more importantly, it was extremely fun in the early days. PvP gained you experience and all the other player's loot. Later on the PvP was limited due to PvE lobbyists, but perhaps it made the game more fun for PvE lovers. Here's a random screenshot from my personal archive that shows a bunch of players on the screen at once. [3] In any case, I view it as a great example of a single person MMO success. -- [1] https://fonline.ru/ https://fonline.ru/ [2] https://falloutmods.fandom.com/wiki/FOnline_Engine https://falloutmods.fandom.com/wiki/FOnline_Engine [3] https://imgur.com/9sMJNE5 https://imgur.com/9sMJNE5
slowhadoken 3y ago
DaFluffyPotato made a pretty good video on simple multiplayer game dev recently https://www.youtube.com/watch?v=_hh7Oe1ohQU https://www.youtube.com/watch?v=_hh7Oe1ohQU He even goes over cost based on CPU usage.
charcircuit 3y ago
>And sooner or later, we will hit hell, an evil that lurks behind every MMO, the I/O bottleneck in the database. When I worked on a MMO with a five figure concurrent player count we got by fine with a single database server. The much bigger I/O bottleneck were with the load balancers.

234 points by buba 3y ago

buba 3y ago

Interesting aproach to data ownership philosophy, I/O techniques and source of truth fuckery in MMO-like systems

hiatus 3y ago
Are you the author?
- buba 3y ago
  yep
  - Uehreka 3y ago
    …and you wrote a comment complimenting your own article?
    - xwowsersx 3y ago
      haha, no I think they were giving a heads up to potential readers as to what they think is the interesting or unique points in the post. Still funny though :)
    - buba 3y ago
      Yep, I noticed. It's an interesting approach tho xD
    - aleph_minus_one 3y ago
      The author is either doing SEO or wants to teach the AIs why his approach is great. :-)
zinodaur 3y ago
I liked the article! Do you know of MMOs that try to use database techniques like write-ahead-logs and log-sequence-numbers for persistence/replication? The nice thing about these techniques is that you can replicate state in a consistent way - so you could have multiple game state services all providing equivalent reads

Eumenes 3y ago

Early MMOs (WoW, Asheron's Call, Everquest, Daoc) were very impressive in terms of distributed computing.

Thaxll 3y ago
They were not really distributed though, I think one of the first that really started was Guild Wars 2. https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Presentations/Clarke-Willson_Guild%20Wars%202%20microservices.pdf https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Pres... DAoC was basically a Linux box with a bunch of processes connected to MySQL. https://www.gamedeveloper.com/disciplines/postmortem-mythic-entertainment-s-i-dark-age-of-camelot-i- https://www.gamedeveloper.com/disciplines/postmortem-mythic-...
- deleted 3y ago
  [deleted]
- Eumenes 3y ago
  Would this count? https://www.gamedeveloper.com/design/classic-postmortem-i-asheron-s-call-i- https://www.gamedeveloper.com/design/classic-postmortem-i-as... > One the most impressive features of the Turbine engine is the continuous outdoor environment. This is made possible thanks to dynamic load balancing, which is a scalable serverside architecture. The easiest way to appreciate the need for dynamic load balancing is to consider the following scenario. > Dynamic load balancing solves this overloaded server problem. Instead of assigning a static geographic area to each server, the individual servers can divide up the game world based on the relative processor load of each server. In the previous example, instead of remaining idle, all four servers would divide the load equally among themselves, ensuring the most efficient use of the hardware’s processing capacity.
  - Animats 3y ago
    That's a useful technology. Second Life / Open Simulator do not have that, and need it. It's good to hear about a success with that approach. Improbable tried that, dividing the world into regions but moving the region boundaries around based on player density. This worked, but apparently required huge amounts of inter-server traffic. The system was too expensive to operate. (Running it on Google Cloud with metering for every client/server transaction didn't help.) Five indy free to play games, some of them good (look up Worlds Adrift), went bust because of server cost. Improbable then pivoted to simulators for the UK military, a much less cost-sensitive market. That worked, but they had way too much company and funding for that niche. Then they tried to pivot to crypto metaverses, two years too late, and hooked up with the Yuga Labs (Bored Ape, Otherside) crowd. Lately, they're trying to do something with US Major League Baseball. Their solution to the cost problem is to only run special events that last a few hours, for which they can short term rent some huge number of servers from AWS or somebody. There's still no good off the shelf solution for this kind of scaling, with big worlds and big moving crowds. Epic and Roblox were making noises about working on this problem a year ago, but not much has been heard recently. Now both are in money-losing and layoff mode.

quadral 3y ago

You missed the opportunity to talk about WoW private servers like Trinitycore. Trinitycore emulator can handle 10k+ players on a single server.

buba 3y ago
to be honest, i know nothing about private WoW servers but i promise i'll check it out! Thanks!
- sshagent 3y ago
  azerothcore is probably the best and most polished, if you don't mind wotlk
  - sleepybrett 3y ago
    the last expansion that was any good?
    - PartiallyTyped 3y ago
      Depending on who you ask. Tangent: Imho, the only reason it is good is because it's not as grindy and / or the community just didn't put as much emphasis on min-maxing things. GearScore was a thing of course, but theory crafting wasn't anywhere close to what we have now.
      - sleepybrett 3y ago
        Not sure that's true for me, very good raids (minus the trial). Ulduar and Icecrown being the highlights for my guild. Though didn't mind the single boss ones either. Trial was a little janky though but the fights were fun. The big progression guild we were part of up through burning crusade wanted to leave for greener server pastures, a handful of us were kinda done with hard progression and the cuthroat nature of it. Picked up a few more and did 10mans/hardmodes mostly. Hooked up with another guild like us for 25man but we only cleared those hard modes once.. 25man was still a management nightmare... Played through pandaland, skipped warlords, came back for some of legion and then finally broke the habit.

monlockandkey 3y ago

Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.

hmmokidk 3y ago
Just keep it simple and avoid optimization like the plague. you can spend 1 month building a prototype or >3 months perfecting a single piece. Just know that it is all smoke and mirrors and don’t worry about that.
63 3y ago
I'd wager a guess that getting players will be the most difficult part by far, at least in the beginning. Make an MVP and focus on building a playerbase first, then come back to architecture when you're suffering from success if you get that far.
- yetihehe 3y ago
  > then come back to architecture when you're suffering from success if you get that far. Then it will be too late because you will essentially have to rewrite half of your project while your userbase is leaving due to unplayable game. Better to make good architectural decisions from the start, and make small optimizations when needed.
buba 3y ago
I've some other posts planned about this topic, I don't know when or even if im going to deliver, but you are free to follow the blog and receive the update if I ever do.
net_ 3y ago
I made a video series on networking theory for virtual worlds that has been well received: https://youtu.be/0wOZusuMIIM https://youtu.be/0wOZusuMIIM I've also been working on an engine for the past few years if you want some code examples: https://github.com/Net5F/AmalgamEngine https://github.com/Net5F/AmalgamEngine

Kiro 3y ago

> the I/O bottleneck in the database I thought most MMOs kept everything in memory and only offloaded to the database periodically. I clearly remember rollbacks to fix times (XX:00) when things went down. Edit: Sorry, should have read the whole article before commenting.

izend 3y ago
Exactly, there is no way the state can be persisted on absolutely every change. It has to be periodically dumped to the database.

justinlloyd 3y ago

The "cache, lots of cache" statement is the most true of any MMO architecture we can build. I did some optimization work earlier this year on a project where the single back-end server is now handling 2 billion requests per minute and had around 3TB of RAM for cache (I think the final production system was aiming for 12TB of RAM). There's concerns around race conditions as you pointed out, message passing from client to server, and server to server, client hand-off between sharded servers. Those synchronization problems will haunt your dreams. I think the biggest issue I still struggle with is tracking those ephemeral problems that only happen on one shard, or only when going between this shard and this shard, but not the other way. One useful trick is obviously message prioritization and different messages heading to different servers - though these days I'd put a message router in front of the shards and the router handles persistent connections other than the usual technique of direct connection I've employed in the past. Contributed to an MMO game that involves waving light sabers around, another where you defeat the ultimate prime evil (though I was more on the fraud detection on that one), an unpublished MMO that unceremoniously died during the 2008 financial crash, a "shared world" game that involved animals, an open-world game that involves driving cars and running pedestrians over, a few "internet scale" websites, and am currently lead back-end on another MMO - though our database requirements are relatively simple this time around, but it is still again, read-at-start-up, write-only-when-necessary.

Stevvo 3y ago
How did they get the architecture so wrong on that "open-world game that involves driving cars and running pedestrians over"?
- dmoy 3y ago
  I assume that is a tongue-in-cheek description for Grand Theft Auto (whichever one they made online). That game series has had various moral discussions / controversy surrounding it since before the first game came out like 25 years ago.
  - anon56 3y ago

Animats 3y ago

I wrote about some of these issues client-side, in a previous post about a Rust metaverse client. This is a much worse problem in a metaverse system, because there are no static game level maps. Every object in the world is in a database somewhere. Second Life / Open Simulator makes a big distinction between assets, inventories and area state. Assets (meshes, textures, animations, sounds) are immutable, and are stored more or less permanently. (There's a garbage collection batch job that runs monthly or so) Those are basically files. There's a vanilla web service running on an AWS web server, and Akamai, both heavily cached. Inventories are like file directories. They have asset UUIDs and some metadata (name, etc.) Those are in a database, but that data is dynamic and not cached. Each user has an inventory, of course, and it can be huge. 50,000 items are not unheard of. This is a metaverse; you can build stuff. Area state is in server memory for each region. That's saved periodically, once a minute or so. This is a backup file, not a database. If you wanted a more continuous save process, you could keep a log of recent changes on a different machine than the server. After a crash, reload the server state and rerun the recent changes. Area state is under a gigabyte per region (A region is 256x256 meters). With a three level system like this, none of the levels are severely overloaded. The greatest data volume is from the asset store, and because that's immutable, it can be and is cached extensively. There are three levels of caches - asset server, CDN, and client. It's still a problem getting assets out to the clients fast enough, but with prioritization and concurrency, that's solveable. The inventory database is mostly-read, so the usual scaling techniques for mostly-read databases work. Area state is in memory. The main trick is taking a clean backup without visibly freezing the system.

mannyv 3y ago

In most data architectures the DB is only the backing store, because no matter how fast your database is it's going to be slower than RAM. Once you start caring about the performance the second thing you do is stick a cache layer of one sort or another in front of the database; the first thing should be making sure you have the correct indexes. In any case, it sounds like a distributed cache problem. I wonder if you could just abuse redis for your game backend?

kylestlb 3y ago
may help to read the article, redis is mentioned
justinlloyd 3y ago
You can use redis or memcached, but every MMO or online game I've been involved with, unless it was a "web game", has eschewed those for the most part. The game server maintains the state, knows all the objects in the universe, or at least its portion of the universe, and is responsible for retrieving and updating those objects. Even redis and memcached would be considered slow by comparison. Those game objects/world objects/MOBs may eventually be pushed out to a key-store server, but generally are not. The only portion of the database on any MMO I've worked on that has cared about "proper indexes" has been the area dealing with account retrieval. Traditional databases, at least on the non-web MMOs I've been involved with, when it comes to game state, are not normally used. RDBMS are used for boring things like account management, customer management, and so forth. Our database on the current (non-web) MMO uses a few more web technologies than I have in the past for this particular problem, but once the shard is loaded, and the user is connected, it is back to tradition, for the most part.
- machiaweliczny 3y ago
  What protocols are used to stream updates to client? Or it’s simulation on client and state dump on fails?

whartung 3y ago

Speaking about WoW specifically, since I'm not familiar with the others, I've always been curious about their quest system. Specifically keeping track of what are available for the character efficiently along with the event system to flag quests as completed, etc. There's so many of them. I have to assume they're spatially limited. You enter a zone, or an area, and the system loads up all of the quests located in that space, then it runs through to determine whether you qualify for them. As for quests that you're on, that's a bit more straightforward, since you're so limited to how many you can carry around with you at any one time. Then, every event can practically just be brute forced across your pending quests to see which ones get progressed, etc. But it was always a curiosity to me considering the magnitude of the quests available how most anything can trigger quest progress. There's also the whole achievement system, which perhaps is similar in design.

mtve 3y ago
Sorry in advance for not answering right to your question, but you may check sources of ManGOS/TrinityCore/family WoW servers for that. In short, from what I know, yes, quests are stored in "quest log" fields of character data in server DB, and they are tracked by the clients and checked by the server. Some simple auto quests like "find this item" are not even tracked by server and only stored on completion. Since both client and server have all game data, the client knows about all possible quests and only shows to the player what is appropriate at a current state.
genocidicbunny 3y ago
> Specifically keeping track of what are available for the character efficiently along with the event system to flag quests as completed, etc. This is not necessarily that difficult, at least the first part. A lot of games will have quests be given out by a 'quest giver' character of some sort, or they will activate at specific interaction points on the map. You can do some cheap 'has-player-finished-quest' type of checks to determine if for example the quest giver has some sort of UI to indicate they have a quest available that activate when the quest giver first comes into view range. Quests with more initial conditions can hide their checks behind the interaction with the quest giver. Doing quest progression can be a bit more challenging. You need to determine when to do the checks for progress, and also how comprehensive you want them to be. The more complex the check, the less often you can run it without affecting game performance. I've seen designers use all sorts of tricks depending on the specific quest. Interaction volumes that run checks, periodic ticks, on entity flash messages..etc. > Then, every event can practically just be brute forced across your pending quests to see which ones get progressed This only works for games that have a small number of active quests and not a lot of events. And with MMO's, you really need to be considerate of the accidental quadratic performance problem.

gabereiser 3y ago

Oh I feel for the author. I’ve made games. I’ve made distributed web platforms. I have profound respect for modern mmo architectures because of one, nasty, “I wish this wasn’t a thing” class of data. State. Who, where, what animation, what modeled entity, is in my party, on my screen, under my axe. Synchronized playback of my swing to my party members so we all yell in excitement at the same time when the boss falls. This level of synchronization across shards (server clusters of servers) is enormously complex. Not to mention just writing “net code” in general. Network speed is the biggest issue and often TCP isn’t enough. You need network prediction. Where will they be based on position, direction, etc until I receive the next packet. I can then error check the prediction with the actual and correct. If UDP is available to you, you use it so you can deliver that state as fast as possible, with no ACK back and forth. A combination of UDP state transfer peer2peer for animation and basic state, TCP network connections for services and server state, REST for that auction house. SQS or pub/sub for that item delivery and party/match making/world chat. It’s a beast of a problem. Rewind 15-20 years ago and all the folks who wanted to make a game, their first game, and they want to build an mmo. None of them succeeded. Not 1. The only ones since were from people who knew the ask. Or had a crowdfunded ponzi scheme.

Strom 3y ago
> Rewind 15-20 years ago and all the folks who wanted to make a game, their first game, and they want to build an mmo. None of them succeeded. Not 1. I guess it depends on how you define success, but I would posit FOnline [1][2] as a success story. FOnline is a fan made MMO of Fallout by a single guy, using the assets of the original Fallout 1 & 2 single-player games. Having these assets and also general game mechanics already finished definitely played a huge role in getting it to a playable state in reasonable time. Still, FOnline is a from scratch code base not a mod of the originals. Also it changed plenty of mechanics too, most notably being real-time while the original games were turn-based. It's still being pushed forward even today after 20 years of development by this one guy but it was playable in late 2000s already. Peak concurrent players that I remember seeing was a few thousand. Definitely not AAA level, but way past simple multiplayer. Would have gone higher due to the hype at the time, but the server started to really struggle at that point. After a few years of being a closed source free game it got converted into a SDK and spawned a dozen new fan games using that engine. Perhaps even more importantly, it was extremely fun in the early days. PvP gained you experience and all the other player's loot. Later on the PvP was limited due to PvE lobbyists, but perhaps it made the game more fun for PvE lovers. Here's a random screenshot from my personal archive that shows a bunch of players on the screen at once. [3] In any case, I view it as a great example of a single person MMO success. -- [1] https://fonline.ru/ https://fonline.ru/ [2] https://falloutmods.fandom.com/wiki/FOnline_Engine https://falloutmods.fandom.com/wiki/FOnline_Engine [3] https://imgur.com/9sMJNE5 https://imgur.com/9sMJNE5

slowhadoken 3y ago

DaFluffyPotato made a pretty good video on simple multiplayer game dev recently https://www.youtube.com/watch?v=_hh7Oe1ohQU https://www.youtube.com/watch?v=_hh7Oe1ohQU He even goes over cost based on CPU usage.

charcircuit 3y ago

>And sooner or later, we will hit hell, an evil that lurks behind every MMO, the I/O bottleneck in the database. When I worked on a MMO with a five figure concurrent player count we got by fine with a single database server. The much bigger I/O bottleneck were with the load balancers.