7 ms·
Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
- IgorPartola 16y agoHow does this compare? How many does Google/Amazon/Facebook/Dropbox add a week?
- swombat 16y ago1000 new servers a week is a ridiculous number for this sort of company. Their code must be really, really bad. Anyone who works there care to chime in? Is it all written in ColdFusion or something?
- sahillavingia 16y ago"as many as 1,000 servers every week" Really this probably means that one week they added 1,000 but the actual average number is much lower. Makes for a good talking point worded that way!
- maukdaddy 16y agoExactly. 1000 servers was probably a peak, probably a week when launching a new title - Farmville 3.0 - Return of the Wheat or whatever.
- nethergoat 16y agoNote that it's not 1000 servers every week, it's up to 1000 servers in a given week. This was likely done to support the launch of a new game, and may factor in server turnover (they're on EC2). That being said, launching 1000 nodes for non-batch jobs is still pretty remarkable.
- bd 16y agoZynga can have absolutely crazy growth on launch. FarmVille was growing few million users a week. Also keep in mind interactive multiplayer games with persistent state are different beasts compared to usual web applications. Games are very write heavy and do not cache well.
- zalew 16y agoIs it all written in ColdFusion or something? http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ Zynga uses Apache PHP on the front end, memcached for active user play and MySQL on the back end. It uses memcached to store key value pairs to deal with active user play during sessions and then later writes it to disk.
- brown9-2 16y agoI'm obviously asking the wrong person, but if they use memcached as a temporary datastore until another process can come along later and write the data to disk, what do they do if a memcache node fails? Or would a node fail rare enough for the data loss to not matter much?
- zalew 16y agoI imagine they have other memcached nodes for failover. Have no idea how it's all managed though.
- cageface 16y agoAt this scale the performance advantages of a language like Scala start to seem pretty significant.
- philfreo 16y agoZynga has acquired a bunch of different game studios. While I'm sure they've tried to retrofit the acquired games to better architectures, they probably weren't designed with scalability or efficiency in mind from the beginning.
- portman 16y agoDownvoted for not thinking through the numbers. When Zynga launches a new game, they can get over 1M new DAILY players on that game within the first week. [1] For their games, it seems that 7% of daily players are online at any given time. [2] So within the first week of launch, you have, about 100,000 people playing, simultaneously. Let's say the client is sending 5 requests per second to the game server. And let's say that this week, all 1000 servers went to the new game. Now, of course that's just an AVERAGE across the day. In reality, there will be a cresting factor, like any distributed system, and sometimes the load will spike, say, 8x. Is it ridiculous for a single server to process 500 requests per second on average and 4000 requests per second peak? That seems pretty standard by my book. [1] http://techcrunch.com/2010/06/14/pincus-frontierville/ http://techcrunch.com/2010/06/14/pincus-frontierville/ [2] http://techcrunch.com/2010/08/17/zynga-launches-first-localized-game-debuts-chinese-version-of-texas-poker/ http://techcrunch.com/2010/08/17/zynga-launches-first-locali...
- Retric 16y agoAn average of 5 requests per second from a casual game's client is a sign of a poor design. Realistically you very close to 1 user action = 1 request. Now, a top tier starcraft micromanager may average pull 300 actions per (ed: minute), but an average user should be somewhere around 1 action per second to 6 actions a minute. PS: The key to pulling this off is running the same simulation on the client using the same random seed. User clicks attack blob with fork. > Client says: attacked blob at (timestamp, with fork) > Server says: blob took X damage at (timestamp) random seed at (value) > Client shows: A long string of actions that adds up to that value and or requests a new game state if something does not add up.
- jaxn 16y agoThis makes me feel a little better. We created a multi-player rpg for mobile phones at startup weekend a few year ago. It actually did ok and it was growing pretty quickly. I hosted it on AppEngine to start and the resources just started going sky high. Players were initiating an attack at least once per second (not to mention all of the other server calls like messaging, weapon purchasing, scanning to see who is in your area). I thought we were going to have to reengineer in order to try and make it as efficient as possible. The thing is, these addictive multi-player games that are all about throwing data back and forth end up taking huge amounts of resources. We eventually ditched the game, but the site is still up at http://killyourneighbor.com http://killyourneighbor.com
- iampims 16y agoThat'd be great if you could elaborate a bit on how appengine wasn't fit or too expensive for your use case? Was performance an issue, or just costs? Thanks.
- jaxn 16y agoJust cost. There were some issues with a high number of writes, but it has got better. The unscheduled downtime also killed us. But really, it was just the sheer amount of traffic. Real-time multi-player games that people have with them all the time can just create a ton of requests.
- iampims 16y agoThanks for the reply. Do you think another hosting environment (factoring in the administration costs and all) would have been cheaper?
- xtacy 16y agoDoes the 1000 servers a week also include new servers that replace old failed ones?
- dpcan 16y agoHere's how this adds up to me: - 215M users. - Up to 1000 servers per week. - Say, they've been doing this for 30 weeks. - So they need 1 server for every 7167 users? Is this normal for mmorpg type games? - Are 215M people playing around the clock? I'm just a little baffled by this number.
- jdk 16y agoCompare and contrast with World of Warcraft: http://www.crunchgear.com/2009/09/18/blizzard-reveals-some-technical-data-about-world-of-warcraft/ http://www.crunchgear.com/2009/09/18/blizzard-reveals-some-t... "...it takes roughly 20,000 computer systems, over a petabyte of storage, and over 4600 people. Using multiple data centers around the world, this works out to a total of 13,250 server blades, 75,000 CPU cores, and 112.5 terabytes of blade RAM." It doesn't say if that's total or just US (since WoW is hosted by different partners in Asia, last I paid any attention).
- _delirium 16y ago7000 users/server is actually pretty high for an MMO, though I don't know how Zynga's MMOs compare to something like World of Warcraft in terms of server resources needed. The numbers being floated around for World of Warcraft are around 75,000 cores for 10m subscribers, or about 133 subscribers per core. Obviously those aren't single-core servers, but it's still under 1k per server. Of course this and the Zynga numbers are also both subscriber numbers, not peak player numbers, and I'm not sure how play frequency/duration differs between the two companies' userbases. Some MMO designers actually really want to bring that number even lower, though lots of other people at game companies are looking at ways to pack more users on each core. But from a designer's perspective, if you typically get less than 1/50 of a CPU slice per user, maybe 1/100, and much of that is taken up by doing basic bookkeeping, it's hard to put in things that even 1990s RPGs took for granted, like NPCs with at least passably interesting AI.
- joshbuckley 16y agoOne key difference may be the sheer amount of metrics that Zynga tracks. I've heard they keep stats of every (however mundane) action each user performs. It racks up to a huge amount of data.
- robryan 16y agoYou'd think this stuff could be pretty efficient if done right, defer as much as possible to the flash application. I guess security is an issue if you do to much processing the the flash, probably a lot of servers to just to serve the flash.
- weaksauce 16y agoIf the flash is not changed very often why not a CDN for the flash delivery and then have the flash poll their servers with the farming actions, ect....
- owyn 16y agoI'm sure they use a CDN for delivery of the flash assets. The flash is still most likely updated several times a day. We used to batch commands in the client and submit every 6 seconds. It was still a crazy number of writes, because basically every click is a database update. MySQL is really bad at that sort of work load, but I migrated almost everything to Redis. Really fun problems to solve!
- kno 16y ago1000 servers per week number is not just realistic. I will like to believe the author misspoke otherwise it will take more that Zynga employees just to configure 1000 servers in a week time.
- lukev 16y ago1000 servers is not at all unrealistic if you've automated the process.
- d2viant 16y agoIt's likely fully automated since it's running on EC2
- davidu 16y agoUmm... With EC2, which is what they often use, powering up and configuring 1000 machines can take 10 minutes or so, give or take a ramped boot. :-)
- deleted 16y ago[deleted]
- nkassis 16y agoThat doesn't suprise me, I was running a network for 150 computers while in college and one of the computers was always doing a crazy amount of download and upload (order of gigabyte a day). Our IDS flagged it and I started investigating what was going on. I finally figured out that the users was using a flash game site. I couldn't believe a darn stupid game could cause that much traffic but we asked the person to stop for a day and it was exactly that.
- c1sc0 16y agoAnyone else think this is a sad use of both human and computing resources? Technically interesting, yes, but all of that for playing freaking farmville?
- portman 16y agoFarmville is a perfect example of the self-actualization tier of Maslow's Hierarchy of Needs. [1] The people playing Farmville are enjoying themselves and expressing themselves. Would it be better if they deployed that cognitive surplus and attention towards Wikipedia or a similar project? Of course. But it doesn't make me sad that there are 215 million people in the world who have a comfortable enough life that they can afford to dabble in some casual games. In fact, I hope that in 100 years we can have all 7+ billion humans striving for self-actualization instead of food and water. [1] http://en.wikipedia.org/wiki/Maslows_hierarchy_of_needs http://en.wikipedia.org/wiki/Maslows_hierarchy_of_needs
- antirez 16y agoApparently this are 1,000 ec2 instances not real servers. This changes a lot of things. Btw how is it possible that 10 or 20 years ago using megaflops or MIPS (or other sane measures of computer power) was normal and now we are here talking about "servers"? :) Edit: I'm searching around the internet if there is some new general measure of computer power that is able to reflect the real-world performance in non scientific applications. No luck so far.
- greenlblue 16y agoI read this and thought what a waste of electricity. Putting all those processors to work on folding proteins would have brought about so much more social good.
- ashitvora 16y agoWhen I first read this on TechCrunch, I thought it's a typo.