4 ms·

Here's how this adds up to me: - 215M users. - Up to 1000 servers per week. - Say, they've been doing this for 30 weeks. - So they need 1 server for every 7

by dpcan 16y ago

Here's how this adds up to me: - 215M users. - Up to 1000 servers per week. - Say, they've been doing this for 30 weeks. - So they need 1 server for every 7167 users? Is this normal for mmorpg type games? - Are 215M people playing around the clock? I'm just a little baffled by this number.

jdk 16y ago
Compare and contrast with World of Warcraft: http://www.crunchgear.com/2009/09/18/blizzard-reveals-some-technical-data-about-world-of-warcraft/ http://www.crunchgear.com/2009/09/18/blizzard-reveals-some-t... "...it takes roughly 20,000 computer systems, over a petabyte of storage, and over 4600 people. Using multiple data centers around the world, this works out to a total of 13,250 server blades, 75,000 CPU cores, and 112.5 terabytes of blade RAM." It doesn't say if that's total or just US (since WoW is hosted by different partners in Asia, last I paid any attention).
_delirium 16y ago
7000 users/server is actually pretty high for an MMO, though I don't know how Zynga's MMOs compare to something like World of Warcraft in terms of server resources needed. The numbers being floated around for World of Warcraft are around 75,000 cores for 10m subscribers, or about 133 subscribers per core. Obviously those aren't single-core servers, but it's still under 1k per server. Of course this and the Zynga numbers are also both subscriber numbers, not peak player numbers, and I'm not sure how play frequency/duration differs between the two companies' userbases. Some MMO designers actually really want to bring that number even lower, though lots of other people at game companies are looking at ways to pack more users on each core. But from a designer's perspective, if you typically get less than 1/50 of a CPU slice per user, maybe 1/100, and much of that is taken up by doing basic bookkeeping, it's hard to put in things that even 1990s RPGs took for granted, like NPCs with at least passably interesting AI.
- joshbuckley 16y ago
  One key difference may be the sheer amount of metrics that Zynga tracks. I've heard they keep stats of every (however mundane) action each user performs. It racks up to a huge amount of data.
- sbov 16y ago
  Just to emphasize your point in the last paragraph: comparing the game mechanics in Zynga games to those in traditional MMOs is laughable. Zynga games are more comparable to social websites than WoW.
  - crystalis 16y ago
    Could you point out a WoW mechanic that is actually taxing? From what I understand, WoW AI is a height agnostic beeline and battles are heavily scripted at best. The tricky part seems to be the real-time player interaction a.k.a 3d chatroom, which has nothing to do with game mechanics (e.g., IMVU).
    - bmelton 16y ago
      The easiest mechanic to point out is movement, and the synchronization of movement. At every step in the game, I'm moving along what would traditionally be game tiles, in a given orientation at a given velocity. While the 'world' is rendered around me on my PC, the movement and orientation changes all have to be upstreamed to the server and calculated for normalcy (to detect speed hacks, cheats, mountain climbing, etc).
    - sbov 16y ago
      Many standard rpg mechanics in a large, seamless, 3d world full of both NPCs and players can tax resources. Movement is one already mentioned, but in general, AI isn't only turned on in combat. NPCs move outside of combat - most creatures in the world wander about or follow a pre-defined path. I don't recall if they did on WoW, but I know in Everquest creatures used spells of their class both in and out of combat, such as buffs and heals - a creature in Everquest would even heal and buff up their allies. If monsters are buffing, you now are tracking those buffs & their durations - when we added monsters buffing to our multiplayer game, we made them infinite in duration to get acceptable performance. Another operation that could be taxing is calculating line of sight (LOS) - this factors into any action an NPC or player would do. You can't have monsters attacking players, or buffing allies, through walls.
      - 16y ago
MWarneford 16y ago
We develop casual MMOs (think Club Penguin). These are not nearly as complex as a WOW, much closer to Farmville. In our case, like most MMOs, nearly all our messages go back and forth over persistent socket connections, not HTTP requests. Firewalls cause problems and often block persistent connections, but we can get a much much better through put. For example, we developed a product for Turner on top of our platform. Take a worst case scenario - everyone in the game moves using arrow keys, and they're all moving. A typical room might have 25 people, each broadcasting their location roughly every 0.5 seconds. This location update is broadcast to all 24 other people in the room. So each person generates 2 messages into the server per second, but results in 48 broadcast messages per second. Of course there are 25 people in the room, all of whom are moving. So we have 2 x 25 = 50 messages into the server per second, and 48 x 25 = 1200 out going messages per second. A typical node of our platform might support 4000 players. If each room supports 25 people then we have 160 rooms. Every second the server needs to broadcast 1200 x 160 = 192,000 or nearly 200,000 messages, and receives close to 8000 messages. Movement packets are easy. We can route these through the system quickly, but then think about the "slow" requests, such as writes to the game state persistence store (Mongo DB) or slower still, writes to the SQL database. With 8000 incoming messages, and a typical 64 thread processing pool each thread needs to churn through 125 messages per second. Or one message every 0.008 seconds. Those slow SQL writes might take 0.5 second, during which time 0.5/0.008 = 62 messages have backed up. Many of our games (and im inferring Zynga's) store huge amounts of persistence / state data. Where has the player been, whats completed, timers, friends, etc, etc. Much more than you'd imagine - and thats not including the analytics data. These slower writes very quickly add up and reduce throughput. Of course there are lots of things to optimise the number of messages, but this starts to give an idea for how quickly messages back up in a MMO type environment. If you've bothered to read this far you might be interested in a short blog post I wrote about our casual MMO architecture http://dubitplatform.com/blog/2010/3/4/under-the-hood-dubit-platform-architecture.html http://dubitplatform.com/blog/2010/3/4/under-the-hood-dubit-...
jaspertheghost 16y ago
There are a couple of things: 1. The servers aren't all application servers. There's load balancers, memcache servers, lossy memcache servers, etc. 2. It's the cloud. It includes large instances, HA-Proxies, memcache servers, etc. 3. There's very high write characteristics for social gaming's data store, but the read is similar to normal web application. 4. He took an aggregate server #, and generalized it over a period of 30 weeks. It's directionally accurate, but I wouldn't take it literally.
MikeMacMan 16y ago
"Up to 1000 servers per week" could simply mean that, there was this one week where they added 1000 servers. I'm pretty sure it's spin. We don't know what they're adding on average each week.

by dpcan 16y ago

jdk 16y ago

Compare and contrast with World of Warcraft: http://www.crunchgear.com/2009/09/18/blizzard-reveals-some-technical-data-about-world-of-warcraft/ http://www.crunchgear.com/2009/09/18/blizzard-reveals-some-t... "...it takes roughly 20,000 computer systems, over a petabyte of storage, and over 4600 people. Using multiple data centers around the world, this works out to a total of 13,250 server blades, 75,000 CPU cores, and 112.5 terabytes of blade RAM." It doesn't say if that's total or just US (since WoW is hosted by different partners in Asia, last I paid any attention).

_delirium 16y ago

7000 users/server is actually pretty high for an MMO, though I don't know how Zynga's MMOs compare to something like World of Warcraft in terms of server resources needed. The numbers being floated around for World of Warcraft are around 75,000 cores for 10m subscribers, or about 133 subscribers per core. Obviously those aren't single-core servers, but it's still under 1k per server. Of course this and the Zynga numbers are also both subscriber numbers, not peak player numbers, and I'm not sure how play frequency/duration differs between the two companies' userbases. Some MMO designers actually really want to bring that number even lower, though lots of other people at game companies are looking at ways to pack more users on each core. But from a designer's perspective, if you typically get less than 1/50 of a CPU slice per user, maybe 1/100, and much of that is taken up by doing basic bookkeeping, it's hard to put in things that even 1990s RPGs took for granted, like NPCs with at least passably interesting AI.

joshbuckley 16y ago
One key difference may be the sheer amount of metrics that Zynga tracks. I've heard they keep stats of every (however mundane) action each user performs. It racks up to a huge amount of data.
sbov 16y ago
Just to emphasize your point in the last paragraph: comparing the game mechanics in Zynga games to those in traditional MMOs is laughable. Zynga games are more comparable to social websites than WoW.
- crystalis 16y ago
  Could you point out a WoW mechanic that is actually taxing? From what I understand, WoW AI is a height agnostic beeline and battles are heavily scripted at best. The tricky part seems to be the real-time player interaction a.k.a 3d chatroom, which has nothing to do with game mechanics (e.g., IMVU).
  - bmelton 16y ago
    The easiest mechanic to point out is movement, and the synchronization of movement. At every step in the game, I'm moving along what would traditionally be game tiles, in a given orientation at a given velocity. While the 'world' is rendered around me on my PC, the movement and orientation changes all have to be upstreamed to the server and calculated for normalcy (to detect speed hacks, cheats, mountain climbing, etc).
  - sbov 16y ago
    Many standard rpg mechanics in a large, seamless, 3d world full of both NPCs and players can tax resources. Movement is one already mentioned, but in general, AI isn't only turned on in combat. NPCs move outside of combat - most creatures in the world wander about or follow a pre-defined path. I don't recall if they did on WoW, but I know in Everquest creatures used spells of their class both in and out of combat, such as buffs and heals - a creature in Everquest would even heal and buff up their allies. If monsters are buffing, you now are tracking those buffs & their durations - when we added monsters buffing to our multiplayer game, we made them infinite in duration to get acceptable performance. Another operation that could be taxing is calculating line of sight (LOS) - this factors into any action an NPC or player would do. You can't have monsters attacking players, or buffing allies, through walls.
    - 16y ago

MWarneford 16y ago

We develop casual MMOs (think Club Penguin). These are not nearly as complex as a WOW, much closer to Farmville. In our case, like most MMOs, nearly all our messages go back and forth over persistent socket connections, not HTTP requests. Firewalls cause problems and often block persistent connections, but we can get a much much better through put. For example, we developed a product for Turner on top of our platform. Take a worst case scenario - everyone in the game moves using arrow keys, and they're all moving. A typical room might have 25 people, each broadcasting their location roughly every 0.5 seconds. This location update is broadcast to all 24 other people in the room. So each person generates 2 messages into the server per second, but results in 48 broadcast messages per second. Of course there are 25 people in the room, all of whom are moving. So we have 2 x 25 = 50 messages into the server per second, and 48 x 25 = 1200 out going messages per second. A typical node of our platform might support 4000 players. If each room supports 25 people then we have 160 rooms. Every second the server needs to broadcast 1200 x 160 = 192,000 or nearly 200,000 messages, and receives close to 8000 messages. Movement packets are easy. We can route these through the system quickly, but then think about the "slow" requests, such as writes to the game state persistence store (Mongo DB) or slower still, writes to the SQL database. With 8000 incoming messages, and a typical 64 thread processing pool each thread needs to churn through 125 messages per second. Or one message every 0.008 seconds. Those slow SQL writes might take 0.5 second, during which time 0.5/0.008 = 62 messages have backed up. Many of our games (and im inferring Zynga's) store huge amounts of persistence / state data. Where has the player been, whats completed, timers, friends, etc, etc. Much more than you'd imagine - and thats not including the analytics data. These slower writes very quickly add up and reduce throughput. Of course there are lots of things to optimise the number of messages, but this starts to give an idea for how quickly messages back up in a MMO type environment. If you've bothered to read this far you might be interested in a short blog post I wrote about our casual MMO architecture http://dubitplatform.com/blog/2010/3/4/under-the-hood-dubit-platform-architecture.html http://dubitplatform.com/blog/2010/3/4/under-the-hood-dubit-...

jaspertheghost 16y ago

There are a couple of things: 1. The servers aren't all application servers. There's load balancers, memcache servers, lossy memcache servers, etc. 2. It's the cloud. It includes large instances, HA-Proxies, memcache servers, etc. 3. There's very high write characteristics for social gaming's data store, but the read is similar to normal web application. 4. He took an aggregate server #, and generalized it over a period of 30 weeks. It's directionally accurate, but I wouldn't take it literally.

MikeMacMan 16y ago

"Up to 1000 servers per week" could simply mean that, there was this one week where they added 1000 servers. I'm pretty sure it's spin. We don't know what they're adding on average each week.