10 ms·

> We run Fortnite’s dedicated game servers primarily on thousands of c4.8xlarge AWS instances, which scale up and down with our daily peak of players. That's b

> We run Fortnite’s dedicated game servers primarily on thousands of c4.8xlarge AWS instances, which scale up and down with our daily peak of players. That's between $572,000 (500 instances, 30 days) - $2,863,800 (2500 instances, 30 days), per month at current prices, and seems like it's only for one aspect of their infrastructure. That seems .... excessive? Is that a typical spend with a game server system like this? That does seem to suggest that once this becomes less than profitable, it's all going away ...

eterm 9y ago
Perhaps they're doing some creative counting and thousands refers to all the instances spun up over a day rather than the actual concurrent boxes.
- evanosaurus 9y ago
  Nope. No creative counting. Those are our real numbers.
  - ddorian43 9y ago
    I want bandwidth monthly pricing (maybe it adds up)! And what's minimum networking required on each server ?
013a 9y ago
c4.8xlarges are the smallest c4 instance that guarantees 10G networking performance. Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that. No one at their scale actually pays the list price. Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game. Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.
- bpicolo 9y ago
  > and it doesn't have as stringent real-time requirements Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc. Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away
  - kevingadd 9y ago
    Battle Royale games generally need to broadcast player state to all players within a large distance (iirc for PUBG, it's one km) because it's common to spot another player 500+ meters out and take some shots at them. If they're not moving much you might hit them.
    - bpicolo 9y ago
      Even if all 100 players are nearby, 100 isn't unusual in the context of MMOs. That's not to say it's trivial to do, it's just not a new feat unique to battle royale games.
- Thaxll 9y ago
  Since gameservers are mostly single threaded they probably run 35~~ of them on a single instance ( c4.8xlarges have 36 vcores ), so if you have a good matchmaking algorithm you can fit 3500~~ players per instance, to reach 3.5M you need between 1000 and 2000 instances. As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.
  - Flammy 9y ago
    Likely more than 35 games on a single instance, as ideally you wouldn't be starting all games at the same time on the same instance. Staggering the start of games on each instance would be ideal. For example you could have dozens of late stage games that combined have less players than a single, just started, 100 person game.
zemo 9y ago
I run game servers. Our intra-day swing gives us about 10x traffic during primetime hours compared to off hours. Autoscaling sheds a lot of overhead.
- pathseeker 9y ago
  Have you run the calculations to determine where that cutoff is? If you have a daily primetime of 10x traffic that's pretty consistently the same amount, then it's still cheaper (if we're talking the scale of hundreds of instances) to rack up the servers in a datacenter (COLO or your own) with the capacity for the standard 10x load. Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).
  - Thaxll 9y ago
    You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money, if you want to go this way you pay the minimum numbers of servers at any time and you use cloud for the rest. But again I think noways it doesn't make any sense to buy servers in DC. You pay for hardware that you need to mange and have to be sure that you will be using it after the game launch. ( most games player base decrease by 2x just after release. )
    - twic 9y ago
      > You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money No, it's cheaper once you factor in the real performance of physical vs virtual machines. Someone really needs to bring some actual numbers to this conversation.
      - Thaxll 9y ago
        Numbers were done it's cheaper to run in the cloud nowdays, also VM ( EC2 on AWS ) have the same perf that baremetal. Have a read: https://aws.amazon.com/ec2/instance-types/c5/ https://aws.amazon.com/ec2/instance-types/c5/
tbrock 9y ago
Shit. Switching to c5 would save them over 100k/month at a minimum and net better performance to boot. I’d get on that.
- evanosaurus 9y ago
  We’d love to, but have been told that our providers don’t yet have enough capacity for us to flip our entire fleet over to c5’s just yet.

eterm 9y ago

Perhaps they're doing some creative counting and thousands refers to all the instances spun up over a day rather than the actual concurrent boxes.

evanosaurus 9y ago
Nope. No creative counting. Those are our real numbers.
- ddorian43 9y ago
  I want bandwidth monthly pricing (maybe it adds up)! And what's minimum networking required on each server ?

013a 9y ago

c4.8xlarges are the smallest c4 instance that guarantees 10G networking performance. Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that. No one at their scale actually pays the list price. Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game. Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.

bpicolo 9y ago
> and it doesn't have as stringent real-time requirements Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc. Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away
- kevingadd 9y ago
  Battle Royale games generally need to broadcast player state to all players within a large distance (iirc for PUBG, it's one km) because it's common to spot another player 500+ meters out and take some shots at them. If they're not moving much you might hit them.
  - bpicolo 9y ago
    Even if all 100 players are nearby, 100 isn't unusual in the context of MMOs. That's not to say it's trivial to do, it's just not a new feat unique to battle royale games.
Thaxll 9y ago
Since gameservers are mostly single threaded they probably run 35~~ of them on a single instance ( c4.8xlarges have 36 vcores ), so if you have a good matchmaking algorithm you can fit 3500~~ players per instance, to reach 3.5M you need between 1000 and 2000 instances. As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.
- Flammy 9y ago
  Likely more than 35 games on a single instance, as ideally you wouldn't be starting all games at the same time on the same instance. Staggering the start of games on each instance would be ideal. For example you could have dozens of late stage games that combined have less players than a single, just started, 100 person game.

zemo 9y ago

I run game servers. Our intra-day swing gives us about 10x traffic during primetime hours compared to off hours. Autoscaling sheds a lot of overhead.

pathseeker 9y ago
Have you run the calculations to determine where that cutoff is? If you have a daily primetime of 10x traffic that's pretty consistently the same amount, then it's still cheaper (if we're talking the scale of hundreds of instances) to rack up the servers in a datacenter (COLO or your own) with the capacity for the standard 10x load. Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).
- Thaxll 9y ago
  You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money, if you want to go this way you pay the minimum numbers of servers at any time and you use cloud for the rest. But again I think noways it doesn't make any sense to buy servers in DC. You pay for hardware that you need to mange and have to be sure that you will be using it after the game launch. ( most games player base decrease by 2x just after release. )
  - twic 9y ago
    > You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money No, it's cheaper once you factor in the real performance of physical vs virtual machines. Someone really needs to bring some actual numbers to this conversation.
    - Thaxll 9y ago
      Numbers were done it's cheaper to run in the cloud nowdays, also VM ( EC2 on AWS ) have the same perf that baremetal. Have a read: https://aws.amazon.com/ec2/instance-types/c5/ https://aws.amazon.com/ec2/instance-types/c5/

tbrock 9y ago

Shit. Switching to c5 would save them over 100k/month at a minimum and net better performance to boot. I’d get on that.

evanosaurus 9y ago
We’d love to, but have been told that our providers don’t yet have enough capacity for us to flip our entire fleet over to c5’s just yet.