17 ms·
Postmortem of Service Outage at 3.4M Concurrent Users
- Gogogogirl 9y ago"318k database reads and 132k database writes per second" Impressive.
- warriormonk5 9y agoThey need to quit using mongodb.
- zuni 9y agoMongoDB? Really? Come on guys. Having used MongoDB at scale, I can tell you this won't be the last issue you have by any stretch.
- iBotPeaches 9y agoThat was an incredible fun read. Makes me curious of the other failures in this industry if they could be explained in this detail.
- Thaxll 9y agoVideo game industry doesn't reveal that much about their tech. You only see a glimpse of what they use during public talk like GDC ect...
- lazyjones 9y agoThe EVE developers used to post a lot of details about their setup, upgrades, software and about failures / congestion issues...
- robotmay 9y agoEVE's time dilation system is ingenious: https://www.eveonline.com/article/introducing-time-dilation-tidi/ https://www.eveonline.com/article/introducing-time-dilation-...
- plopz 9y agoIts an absolutely awful experience for a user. Especially when some things are being affected by tidi while others are not.
- always_good 9y agoWorse UX than failing completely any time a solar system gets more than X players in it, or having a player cap per solar system? Those would be "absolutely horrible" to me. Time dilation is a compromise, not just arbitrary lore that became a game mechanic.
- plopz 9y agoSure, its the least bad system, but its still a bad system.
- FLUX-YOU 9y agoIs there any game that can do 6k+ simultaneous players with a similar level of physics/combat system that EVE has at a 1s tick rate? I'm fairly sure there isn't.
- AlphaSite 9y agoEve doesn't really have much in the way of physics, which would be genuinely impressive.
- FLUX-YOU 9y agoBy physics, I mean the ship/object sphere bouncy collisions and the velocities that are calculated from a collision. Ship agility is also technically physics. You still have to calculate those every tick
- needusername 9y agoEve at 6k+ simultaneous players has a 10s tick rate due to TiDi. At 6k+ simultaneous players Eve isn't actually playable. I don't think there's an other game with a 1s tick rate, AFAIK Eve has the worst tick rate in the business.
- dmpress 9y agoRiot Games has an engineering blog that goes into some good details: https://engineering.riotgames.com/ https://engineering.riotgames.com/ Disclaimer: Riot Games Engineer :-)
- Tobba_ 9y agoIt's great to actually see a game developer that seems to have some grip on what they're doing. At least writing about issues like clocking, animation quantization, etc is far better than what everyone else does - screwing it up and never fixing it. It's not all perfect, but still(I don't play LoL though).
- eterm 9y agoThis is an interesting read, it's always interesting to hear why something that ought to be fairly heavily federated or sharded can nevertheless fall over centrally.
- pkilgore 9y agoLove this because it shows two things 1) competent people are handling problems and 2) they actually care. A whole lot better than spoon feeding customers bullshit for weeks while hamstringing your product rather than investing in it (looks at EA, mumbles about SimCity).
- Thaxll 9y agoEpic game is not a publicly traded company and there is a lot of things you can say when you're private.
- Grazester 9y agoI would have thought it was the opposite way around! Private companies always seem so tight lipped especially being that they dont have to answer to shareholders.
- Thaxll 9y agoYou're not going to say to shareholders "please give me your money and also we're looking for expert because we lack expertise in the product we're doing"
- gameswithgo 9y agoThat is an interesting state of affairs isn't it? Why do we, the world, expect that large, supposedly more professional companies, communicate less honestly?
- deadbunny 9y ago> Why do we, the world, expect that large, supposedly more professional companies, communicate less honestly? Because being honest can hurt the share price.
- hhmc 9y agoParticularly, it can hurt the share price on an horizon over which the C-level execs have their performance measured.
- aaossa 9y agoLoved the tone of the article. They know they have some problems to work on, they're being transparent about them and they're explicitly saying that they need help with it.
- stevenwoo 9y agoLook forward to seeing how they fix that MongoDB collection write stalling problem. Vaguely recall that was still a big problem the last time I was looking at MongoDB years ago.
- tweenagedream 9y agoDisclaimer: I work on Google Cloud so I will be speaking from the bias of knowing those products. They talk a lot about reducing operating complexity and scaling their infrastructure, I wonder what the cost of their current infrastructure + the staff to maintain it might be vs the managed solutions that cloud providers offer now. For example, using cloud datastore or spanner or big table as a persistent layer, these managed services can definitely scale to the current need and I've seen them go much higher as well. For logs ingestion and analysis, big query can be a very powerful tool as well, and with streaming inserts that data can be queried in near real time. For things that are less urgent, batch queries. For other things dataflow can help with streaming workloads. I think one of the problems they alluded to though was that at the moment they're on a single provider, and what they're looking for is a multi cloud strategy which totally makes sense. A lot of the above products create some kind of locking, with some exceptions, like using hbase as an interface to big table or beam as an interface to dataflow. Though I don't know what the other providers offer that may have these same interfaces. Another option is kubernetes, which I believe all providers are pretty strongly embracing. Having most of the supporting infrastructure be brought up with a few kubectl commands could help them scale across several cloud providers quickly.
- matt_s 9y agoI think they detailed in the article that the problem isn't their game servers which are AWS cloud based and can scale up, it is their login/setup/matchmaking (my term) server infrastructure that is the first thing users first encounter that is having issues. Usually there is a cost/scale threshold with managed providers where it is cheaper do DIY than to pay thousands upon thousands per month for say log ingestion.
- tweenagedream 9y agoI agree and my comment was mostly aimed at the supporting services rather than the actual game servers. However, they mentioned that they flew out Mongo experts to help on site and during weekend peaks. That sounds pretty expensive to me. Many times, the cost of the infrastructure is the cheap part, and the salaries of the humans managing and tuning it are the expensive part. I wonder if they really hit the point where doing this part in house rather than offloading it is a net win.
- fokinsean 9y agoAs an addicted Fortnite player this is a neat read. However as an application layer dev, the architecture specifics were slightly over my head. My biggest concern is shipping a working docker image, all of the architecture is mostly abstracted at our company. This gave me some inspiration to dive deeper into our architecture.
- AccountCreated 9y ago> The primary MCP database is comprised of 9 MongoDB shards enough said.
- victorqhong 9y agoReally surprised that they use XMPP. Since you don't really hear anything about XMPP anymore, I think most people assumed that it's dropped off in usage/popularity (or people have moved to some other proprietary solution). I've always thought that XMPP would be useful for games, just surprised to hear that people are actually doing it.
- solotronics 9y agoa ton of services use XMPP underneath the hood like Signal, WhatsApp, Jabber etc. it looks like a great business model to take something open source and hard to use and make a proprietary version thats easy to install and use https://en.wikipedia.org/wiki/Signal_Protocol https://en.wikipedia.org/wiki/Signal_Protocol https://en.wikipedia.org/wiki/WhatsApp https://en.wikipedia.org/wiki/WhatsApp https://www.cisco.com/c/en/us/about/corporate-strategy-office/acquisitions/jabberinc.html https://www.cisco.com/c/en/us/about/corporate-strategy-offic...
- e12e 9y agoSignal isn't xmpp based? Jabber is xmpp. Not sure about WhatsApp.
- merb 9y ago> Not sure about WhatsApp. whatsapp at least was in the "early" days. they had some extensions to it but you could connect to it pretty easily (once you had a valid account generated with a real mobile number) not sure if it is still the case, things changed a lot I guess, especially with encryption.
- gsich 9y agoit still is https://github.com/tgalal/yowsup https://github.com/tgalal/yowsup
- Dolores12 9y agoRiot Games is using XMPP to provide in-game chat for League of Legends.
- matt_s 9y agoA lot of players of massive online games tend to get hand-wavey when there are problems and act like "dude just get more servers" is the answer. This clearly shows how complex a system is needed that has to handle 3.4 million concurrent, connected users. I think the connected part compounds any scale problems you have since it is implied they are connected to each other.
- always_good 9y agoAgreed. Gamers also are among the most entitled users I've ever had to deal with. I'll never accept donations again for a game I've made available for free from the amount of problems someone has caused me after they gave me $5.
- trevyn 9y agoDid you quickly offer/send a refund? This is going to be my strategy for upset customers, and I’m curious about people’s experiences with this approach.
- always_good 9y agoNo, but it's not like they are privately emailing me with their complaints. They drum up threads in the subreddit and rally others to complain as if they are being personally wronged. I suppose I could offer a refund, but refunding a $5 donation is just another example of how a donation box is a waste of time. In the future I would at least consider a Patreon system where donations renew every month since server and development costs certainly do. Way too many people think they're buying an annuity against your free time with a one time donation/purchase.
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- 9y ago
- dom96 9y agoLooks like they are still having scaling issues. I just tried creating a new Epic account and was shown an error.
- evanosaurus 9y agoOur account service went down for a brief period this morning. Sorry about that.
- sgarman 9y agoSure, the game is not done growing in popularity. That's one of the difficult things - they don't just have to scale to 3.4m users because next week the target is moved and now its 4m then 6m etc. It's hard to hit a moving target like that.
- andrea_s 9y ago404 not found at the moment - does anyone has a snapshot? edit: nevermind, it is available again - weird
- deleted 9y ago[deleted]
- jasonjayr 9y ago> We run Fortnite’s dedicated game servers primarily on thousands of c4.8xlarge AWS instances, which scale up and down with our daily peak of players. That's between $572,000 (500 instances, 30 days) - $2,863,800 (2500 instances, 30 days), per month at current prices, and seems like it's only for one aspect of their infrastructure. That seems .... excessive? Is that a typical spend with a game server system like this? That does seem to suggest that once this becomes less than profitable, it's all going away ...
- eterm 9y agoPerhaps they're doing some creative counting and thousands refers to all the instances spun up over a day rather than the actual concurrent boxes.
- evanosaurus 9y agoNope. No creative counting. Those are our real numbers.
- ddorian43 9y agoI want bandwidth monthly pricing (maybe it adds up)! And what's minimum networking required on each server ?
- 013a 9y agoc4.8xlarges are the smallest c4 instance that guarantees 10G networking performance. Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that. No one at their scale actually pays the list price. Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game. Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.
- bullen 9y agoThis is why you use an async-to-async database: Why: https://github.com/tinspin/rupy/wiki/Fuse https://github.com/tinspin/rupy/wiki/Fuse How: https://github.com/tinspin/rupy/wiki/Storage https://github.com/tinspin/rupy/wiki/Storage
- orliesaurus 9y agoI never spent a dime on any of these free 2play games. I am in awe at how dedicated the team behind Fortnite seems to be when it comes to providing us data (real data?) of what's happening on their side, while I am sitting on my couch logging into one of the matches with my keyboard and mouse
- mmanfrin 9y agoWhen Bluehole take down PUBG for 5 hours, there's no communication outside of two tweets. When Epic see degraded performance for less than 2 hours, they give a postmortem. There's a difference in the level of respect each company gives for its customers. I play PUBG a lot, but I want to see Epic win in the long run.
- piyh 9y agoI want the gunplay of PUBG with the support of Epic.
- cjsawyer 9y agoThere isn’t even a custom message for “scheduled server maintenance” vs “our servers are busy” (which I can get multiple times a night anyway). You NEED to check the tweets to find out which it is. I love PUBG, I just wish it was made by more competent developers.
- mmanfrin 9y agoIt's astounding to me how the last couple patches have made performance worse and introduced new, gamebreaking bugs. The last patch took everything down for 5 hours. The patch before tanked FPS by like 40%. The patch before caused 2/3rds of games to spawn players below the match or simply crash on load. They don't do QA, and their 'deploy process' takes everything down for hours. It's embarrassing.
- smnscu 9y ago> They don't do QA, and their 'deploy process' takes everything down for hours. This reminded me of this classic: https://engineering.riotgames.com/news/automated-testing-league-legends https://engineering.riotgames.com/news/automated-testing-lea...
- reificator 9y agoI'm putting it in a to-read list after 2 minutes of skimming, but that looks like an excellent article. Sounds like Riot have a good head on their shoulders about how to handle testing games. If you come from a different domain, it's important to realize that testing games is much more difficult than testing a lot of other types of software. And much like other kinds of software, when it's done well it looks easy.
- NelsonMinar 9y agoLaunching this web page in Firefox on Windows causes my Oculus Rift software to start up. I am.. not happy about that. WTF?
- mevile 9y agoI'd be more likely to share this with my team if they didn't have a recruiting pitch in there. I have a (probably irrational) fear of people abandoning the team for some new shiny thing. Just a thought. It's a great writeup otherwise!
- sh_tinh_hair 9y agoSummary: Operating an interdependent service environment at scale is hard to do if you have multiple areas with heavy technical debt + known breakage and no testing past 'x'. IMO this is shit soup of the modern day. These guys explain it away better than most.
- tlynchpin 9y agoThis is a great article, lots of detail, props to Epic team for generally killing it and specifically putting this together.
- Kagerjay 9y agoI was playing fortnite on 2-04-18 22:00 UTC during the "Friends Service" outage. You couldn't see friends lists at all during that time period. So you couldn't queue up in a friends / people you knew at all in a match, the only options were either playing solo or using a "filled" team with random players. I've been playing fortnite as one of the early 60k concurrent users all the way to the 3.4M, so its been interesting seeing their load / server issues over time and then reading this (Granted, I don't understand everything discussed in their blog). They've done a outstanding job handling their growing traffic. One thing I've noticed with Fortnite, compared to PUBG or other MMOs, is how large their patch updates are. Its usually several GB large, and it comes fairly frequently about once a week.
- frenchie14 9y agoThat has to do more with how they package the game than how much they changed. Often times large parts of the game are bundled together so a single small change in one bundle causes you to re-download the whole thing
- Kagerjay 9y agoForgot to note that fortnite had an ongoing "friend list" problem before Most notably, whenever you wanted to add someone to your party. You had to do the following. 1. You Send user friend invite 2. User would have to accept invite 3. User would have to disconnect to refresh your friends list (due to friend-service issue mentioned in blog) 4. User would relog back on (took approximately 3 mins) 5. You could then see them on friends list 6. Send party invite
- einrealist 9y agoNice read. And nice to see Java running at the backend. I wonder whether Epic can solve its problems by rearchitecting more into a CQRS driven system with event sourcing: store events in a more write optimized DB (e.g. Cassandra) and then process the events for fast reads through whatever is required for the usecases. Maybe they touched the limits of MongoDB to handle both, reads and writes at their scale.
- skaplun 9y agoany thoughts by game production managers on the success of the battle royale concept? anything we should take away for other products?
- halflings 9y agoMeanwhile, the game is pretty much unplayable [1] on Mac OS while it was heralded as the first game to support Metal (even featured in Apple's keynote). [1] Getting ~16 FPS on medium settings, with the high-end late 2016 MBP 15".
- stats_n_trends 9y agoAs a counter point I could play quite well on my MBP 17 at medium settings. (45 FPS) Note: I did tweak some minor settings like AA to get better performance.
- halflings 9y agoInteresting. There shouldn't be any major difference in performance before the two MBPs. Did you get any AMD driver updates or something like that?
- je42 9y agoI wonder why they want to do the step: - Followed by removing Nginx + Memcached couple altogether out of equation.
- MuffinFlavored 9y agoI'm kind of surprised nobody in the comments is mentioning MongoDB "unknowingly" causing write delays. Albeit, it's after handling an amazing programming feat.
- SilverSurfer972 9y ago> "Along with a number of things mentioned, even small performance changes over N nodes collectively make large impacts for our services and player experience." I think this is where Stacktical helps with proactively detecting performance regressions at the CI level, before they hit production: https://stacktical.com https://stacktical.com Disclaimer: I am Stacktical's CTO
- aecorredor 9y agoHow do you get to this level of expertise? What are the resources people like these use to learn about this type of scalable systems? Any good books that start from the ground up on these topics?