4 ms·

1000 new servers a week is a ridiculous number for this sort of company. Their code must be really, really bad. Anyone who works there care to chime in? Is it a

by swombat 16y ago

1000 new servers a week is a ridiculous number for this sort of company. Their code must be really, really bad. Anyone who works there care to chime in? Is it all written in ColdFusion or something?

sahillavingia 16y ago
"as many as 1,000 servers every week" Really this probably means that one week they added 1,000 but the actual average number is much lower. Makes for a good talking point worded that way!
- maukdaddy 16y ago
  Exactly. 1000 servers was probably a peak, probably a week when launching a new title - Farmville 3.0 - Return of the Wheat or whatever.
nethergoat 16y ago
Note that it's not 1000 servers every week, it's up to 1000 servers in a given week. This was likely done to support the launch of a new game, and may factor in server turnover (they're on EC2). That being said, launching 1000 nodes for non-batch jobs is still pretty remarkable.
- bd 16y ago
  Zynga can have absolutely crazy growth on launch. FarmVille was growing few million users a week. Also keep in mind interactive multiplayer games with persistent state are different beasts compared to usual web applications. Games are very write heavy and do not cache well.
zalew 16y ago
Is it all written in ColdFusion or something? http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ Zynga uses Apache PHP on the front end, memcached for active user play and MySQL on the back end. It uses memcached to store key value pairs to deal with active user play during sessions and then later writes it to disk.
- brown9-2 16y ago
  I'm obviously asking the wrong person, but if they use memcached as a temporary datastore until another process can come along later and write the data to disk, what do they do if a memcache node fails? Or would a node fail rare enough for the data loss to not matter much?
  - zalew 16y ago
    I imagine they have other memcached nodes for failover. Have no idea how it's all managed though.
cageface 16y ago
At this scale the performance advantages of a language like Scala start to seem pretty significant.
philfreo 16y ago
Zynga has acquired a bunch of different game studios. While I'm sure they've tried to retrofit the acquired games to better architectures, they probably weren't designed with scalability or efficiency in mind from the beginning.
portman 16y ago
Downvoted for not thinking through the numbers. When Zynga launches a new game, they can get over 1M new DAILY players on that game within the first week. [1] For their games, it seems that 7% of daily players are online at any given time. [2] So within the first week of launch, you have, about 100,000 people playing, simultaneously. Let's say the client is sending 5 requests per second to the game server. And let's say that this week, all 1000 servers went to the new game. Now, of course that's just an AVERAGE across the day. In reality, there will be a cresting factor, like any distributed system, and sometimes the load will spike, say, 8x. Is it ridiculous for a single server to process 500 requests per second on average and 4000 requests per second peak? That seems pretty standard by my book. [1] http://techcrunch.com/2010/06/14/pincus-frontierville/ http://techcrunch.com/2010/06/14/pincus-frontierville/ [2] http://techcrunch.com/2010/08/17/zynga-launches-first-localized-game-debuts-chinese-version-of-texas-poker/ http://techcrunch.com/2010/08/17/zynga-launches-first-locali...
- Retric 16y ago
  An average of 5 requests per second from a casual game's client is a sign of a poor design. Realistically you very close to 1 user action = 1 request. Now, a top tier starcraft micromanager may average pull 300 actions per (ed: minute), but an average user should be somewhere around 1 action per second to 6 actions a minute. PS: The key to pulling this off is running the same simulation on the client using the same random seed. User clicks attack blob with fork. > Client says: attacked blob at (timestamp, with fork) > Server says: blob took X damage at (timestamp) random seed at (value) > Client shows: A long string of actions that adds up to that value and or requests a new game state if something does not add up.
  - kingkilr 16y ago
    > 300 actions per second You mean per minute, right? Either that or I'm way worse at SCII than I thought.
    - Retric 16y ago
      Ops, fixed.

by swombat 16y ago

1000 new servers a week is a ridiculous number for this sort of company. Their code must be really, really bad. Anyone who works there care to chime in? Is it all written in ColdFusion or something?

sahillavingia 16y ago

"as many as 1,000 servers every week" Really this probably means that one week they added 1,000 but the actual average number is much lower. Makes for a good talking point worded that way!

maukdaddy 16y ago
Exactly. 1000 servers was probably a peak, probably a week when launching a new title - Farmville 3.0 - Return of the Wheat or whatever.

nethergoat 16y ago

Note that it's not 1000 servers every week, it's up to 1000 servers in a given week. This was likely done to support the launch of a new game, and may factor in server turnover (they're on EC2). That being said, launching 1000 nodes for non-batch jobs is still pretty remarkable.

bd 16y ago
Zynga can have absolutely crazy growth on launch. FarmVille was growing few million users a week. Also keep in mind interactive multiplayer games with persistent state are different beasts compared to usual web applications. Games are very write heavy and do not cache well.

zalew 16y ago

Is it all written in ColdFusion or something? http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ Zynga uses Apache PHP on the front end, memcached for active user play and MySQL on the back end. It uses memcached to store key value pairs to deal with active user play during sessions and then later writes it to disk.

brown9-2 16y ago
I'm obviously asking the wrong person, but if they use memcached as a temporary datastore until another process can come along later and write the data to disk, what do they do if a memcache node fails? Or would a node fail rare enough for the data loss to not matter much?
- zalew 16y ago
  I imagine they have other memcached nodes for failover. Have no idea how it's all managed though.

cageface 16y ago

At this scale the performance advantages of a language like Scala start to seem pretty significant.

philfreo 16y ago

Zynga has acquired a bunch of different game studios. While I'm sure they've tried to retrofit the acquired games to better architectures, they probably weren't designed with scalability or efficiency in mind from the beginning.

portman 16y ago

Downvoted for not thinking through the numbers. When Zynga launches a new game, they can get over 1M new DAILY players on that game within the first week. [1] For their games, it seems that 7% of daily players are online at any given time. [2] So within the first week of launch, you have, about 100,000 people playing, simultaneously. Let's say the client is sending 5 requests per second to the game server. And let's say that this week, all 1000 servers went to the new game. Now, of course that's just an AVERAGE across the day. In reality, there will be a cresting factor, like any distributed system, and sometimes the load will spike, say, 8x. Is it ridiculous for a single server to process 500 requests per second on average and 4000 requests per second peak? That seems pretty standard by my book. [1] http://techcrunch.com/2010/06/14/pincus-frontierville/ http://techcrunch.com/2010/06/14/pincus-frontierville/ [2] http://techcrunch.com/2010/08/17/zynga-launches-first-localized-game-debuts-chinese-version-of-texas-poker/ http://techcrunch.com/2010/08/17/zynga-launches-first-locali...

Retric 16y ago
An average of 5 requests per second from a casual game's client is a sign of a poor design. Realistically you very close to 1 user action = 1 request. Now, a top tier starcraft micromanager may average pull 300 actions per (ed: minute), but an average user should be somewhere around 1 action per second to 6 actions a minute. PS: The key to pulling this off is running the same simulation on the client using the same random seed. User clicks attack blob with fork. > Client says: attacked blob at (timestamp, with fork) > Server says: blob took X damage at (timestamp) random seed at (value) > Client shows: A long string of actions that adds up to that value and or requests a new game state if something does not add up.
- kingkilr 16y ago
  > 300 actions per second You mean per minute, right? Either that or I'm way worse at SCII than I thought.
  - Retric 16y ago
    Ops, fixed.