4 ms·

Does anyone have any thoughts/info as to why online play like this is always so hard to do right/scale? Many of these bugs and woes don't seem to be simply due

by tibbon 13y ago

Does anyone have any thoughts/info as to why online play like this is always so hard to do right/scale? Many of these bugs and woes don't seem to be simply due to emergent gameplay and bugs that emerge from state issues. It seems largely that they just need more servers. With the type of income (and expenses) that they've already incurred, setting up a few hundred more servers (a few clicks on EC2 if they have their Chef scripts done right) should do it largely right?

Negitivefrags 13y ago
This will happen with any system that needs that kind of scale right out of the gate. It's just that games are one of the few systems like that. We really just don't have good ways of knowing what will happen. Can you think of other online systems that need to scale to massive numbers of users on day one of it's release? If you can, how many of those systems survived? With my own game project, I used around 50,000 bots on AWS and attempted to model them as closely as possible to real players (using real player data taken from our Closed Beta). Looking at resource utilisation on the servers, it looked like we should be able to scale even further than those tests. By that I mean, the most overloaded systems were still only using about 35% of the system resources. Come open beta day, we got concurrency of around 75,000 at peak, higher than the optimistic estimate we had made of 50,000. New problems developed. In fact, new problems developed at around 45,000 that somehow the bots didn't manage to find. The worst problem was a deadlock condition that could take out one of the database servers randomly kicking off ten's of thousands of users at a time. It's hard to describe the feeling of knowing that you are frustrating so many people around the world. More servers would have done nothing to prevent these problems. Anyway, I guess what I'm saying is that scaling all at once is really hard. Even if you take lots of precautions, random problems will still develop.
- talmand 13y ago
  Curios, did your bots simulate connection issues as well? This isn't my line of work but it would seem simulating 50,000 bots in an environment is not quite the same as 50,000 humans from all over the place with their various connections speeds and/or issues. Plus, you'd have to teach your bots to try to do incredibly stupid things that humans tend to do, willingly or otherwise.
  - Negitivefrags 13y ago
    We basically attempted to get the bots to match each metric that we were measuring server side from a smaller sample of players during closed beta. The peak concurrency during closed beta was something like 2000 players, so we multiplied each metric out to get a target. Here are a few examples of metrics: Chat messages per second Party creations per second Average party size Number of zone transitions per second Average number of open zones per player Character saves per second PVP Match list requests per second And so on, and so on. Certainly bots are not going to be the same due to the issues you mentioned, but we do the best we can! We didn't simulate all kinds of connection issues a real player might have, but we did match the non-graceful connection termination metric.

by tibbon 13y ago

Negitivefrags 13y ago

This will happen with any system that needs that kind of scale right out of the gate. It's just that games are one of the few systems like that. We really just don't have good ways of knowing what will happen. Can you think of other online systems that need to scale to massive numbers of users on day one of it's release? If you can, how many of those systems survived? With my own game project, I used around 50,000 bots on AWS and attempted to model them as closely as possible to real players (using real player data taken from our Closed Beta). Looking at resource utilisation on the servers, it looked like we should be able to scale even further than those tests. By that I mean, the most overloaded systems were still only using about 35% of the system resources. Come open beta day, we got concurrency of around 75,000 at peak, higher than the optimistic estimate we had made of 50,000. New problems developed. In fact, new problems developed at around 45,000 that somehow the bots didn't manage to find. The worst problem was a deadlock condition that could take out one of the database servers randomly kicking off ten's of thousands of users at a time. It's hard to describe the feeling of knowing that you are frustrating so many people around the world. More servers would have done nothing to prevent these problems. Anyway, I guess what I'm saying is that scaling all at once is really hard. Even if you take lots of precautions, random problems will still develop.

talmand 13y ago
Curios, did your bots simulate connection issues as well? This isn't my line of work but it would seem simulating 50,000 bots in an environment is not quite the same as 50,000 humans from all over the place with their various connections speeds and/or issues. Plus, you'd have to teach your bots to try to do incredibly stupid things that humans tend to do, willingly or otherwise.
- Negitivefrags 13y ago
  We basically attempted to get the bots to match each metric that we were measuring server side from a smaller sample of players during closed beta. The peak concurrency during closed beta was something like 2000 players, so we multiplied each metric out to get a target. Here are a few examples of metrics: Chat messages per second Party creations per second Average party size Number of zone transitions per second Average number of open zones per player Character saves per second PVP Match list requests per second And so on, and so on. Certainly bots are not going to be the same due to the issues you mentioned, but we do the best we can! We didn't simulate all kinds of connection issues a real player might have, but we did match the non-graceful connection termination metric.