5 ms·

c4.8xlarges are the smallest c4 instance that guarantees 10G networking performance. Those costs can easily be cut in half with reservations. Its likely there'

by 013a 9y ago

c4.8xlarges are the smallest c4 instance that guarantees 10G networking performance. Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that. No one at their scale actually pays the list price. Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game. Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.

bpicolo 9y ago
> and it doesn't have as stringent real-time requirements Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc. Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away
- kevingadd 9y ago
  Battle Royale games generally need to broadcast player state to all players within a large distance (iirc for PUBG, it's one km) because it's common to spot another player 500+ meters out and take some shots at them. If they're not moving much you might hit them.
  - bpicolo 9y ago
    Even if all 100 players are nearby, 100 isn't unusual in the context of MMOs. That's not to say it's trivial to do, it's just not a new feat unique to battle royale games.
Thaxll 9y ago
Since gameservers are mostly single threaded they probably run 35~~ of them on a single instance ( c4.8xlarges have 36 vcores ), so if you have a good matchmaking algorithm you can fit 3500~~ players per instance, to reach 3.5M you need between 1000 and 2000 instances. As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.
- Flammy 9y ago
  Likely more than 35 games on a single instance, as ideally you wouldn't be starting all games at the same time on the same instance. Staggering the start of games on each instance would be ideal. For example you could have dozens of late stage games that combined have less players than a single, just started, 100 person game.
  - Thaxll 9y ago
    You usually don't run two or more game server on single a thread / core, it kills performance, and with VMs with many physical processor you have other issues like NUMA so you have to be very careful. I'm pretty sure they have a ratio of one gameserver per thread / core.
  - evanosaurus 9y ago
    We run 16 game servers per c4.8xl instance for Battle Royale. We run more than that (on a different instance type) for StW.
vvanders 9y ago
FYI most game network servers tick the network at 10-12Hz(although the internal sim may run higher). We generally didn't go higher than 10Hz since you'd start saturating lower end connections and any well written game will be good at reconciling state via dead-reckoning. You have to handle 100ms+ spikes anyway so it doesn't make sense to run the network super-fast.
- troncjb 9y ago
  What do you mean by "game network servers tick the network"? Like data going server -> client ?
- AlphaSite 9y ago
  Most competitive games go well above 10Hz ticks, OW does it, CSGO does it (I don't really know the tick-rates of other games but when it matters, so no comment there, but there is an important area for it). OW just has a restricted rate option for low bandwidth connections (added when they went from 23 -> 61 Hz).
  - vvanders 9y ago
    So that might be the case for LAN games, generally internet tends to stay lower. Valve for instance recommends 13Hz[1] for Source(CSGO): > // 60 for updaterate is LAN ONLY use 13 for internet > // 20 is default but will cut the maxplayers you can handle in 1/2 > // for SRCDS Servers use 30 - you might be able to use 20 > // sv_maxupdaterate 60 > sv_maxupdaterate 13 Like I said, depending on how good your potential-vis code is you may be able to go higher but you risk saturating your client links and most game netcode doesn't handle congestion nicely. My info is a little out of date(2-3 years) but back then we had to keep data rates below 20kb/sec(and ideally 10kb/sec) if you wanted low jitter across the link. [1] https://support.steampowered.com/kb_article.php?ref=5386-HMJI-5162 https://support.steampowered.com/kb_article.php?ref=5386-HMJ...
    - learc83 9y ago
      >So that might be the case for LAN games Overwatch has a 63Hz tick rate over the internet.
welder 9y ago
> No one at their scale actually pays the list price. I've always wondered what AWS rates do large users receive from negotiations?
AlphaSite 9y ago
MAG somehow managed to be a competitive FPS with 256 players back in 2008ish, its not an impossible battle.
- jsmeaton 9y ago
  I think MAG was using clever tricks. There were multiple “zones” per battle, and I think each of those zones was a separate instance with some coordination between.
  - xchaotic 9y ago
    to be fair you could apply the same tricks to any Battle Royale type game
rocqua 9y ago
> Additionally, Battle Royale style games are the most demanding multiplayer games ever created. I'd like to draw your attention to Planetside 2. A pretty recent FPS that was run a lot like WoW, with 3 'continents' and complete seamlessness on those continents. Firefights where 100 people were actively engaging each other were an almost constant experience back when I played it. I think there was a cap of 1000 players per continent. Those players could gather wherever they wanted. I'm pretty sure I was in some 300 player battles, and nothing but lag would prevent all 1000 people on a continent from going to the same base. It has to be said that in the big battles (at around a 100) lag became an issue. I'm not sure whether this was clientside, serverside or both. (There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)
- JCharante 9y ago
  > It has to be said that in the big battles (at around a 100) lag became an issue. I remember the crown frequently having 96+ per faction without any lag problems. Must have been clientside. > (There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded) It was 6000, and it was terrible to be in. It took me 10 minutes to enter a station (normally takes a second) and I had to relog because I couldn't get out of the station. Relogging took another hour of black screens. It was pretty neat when I finally got out, but there wasn't much of a fight compared to previous encounters.
  - wyager 9y ago
    Planetside 2 was horrendously single-threaded, so anything but recent Intel chips would leave you with really bad performance in large battles. I would often max out my AMD CPU (well, one core of it) at around 12 FPS without my graphics card breaking a sweat in 100-200 player battles. PS2 was for the most part an amazing game. The graphics were truly beautiful and the gameplay was very fun. Unfortunately, their shitty performance basically killed the game by making it unplayable for many users during the trademark huge battles.
mbid 9y ago
> Its an n^2 problem > actually increases resource utilization exponentially I think you'll have to choose there.
evanosaurus 9y ago
We don’t tick at 30Hz. Somewhat slower than that. Working on making that better, though. And we do indeed take advantage of reserved instances. We’re also looking at other options along these lines.
bufferoverflow 9y ago
> actually increases resource utilization exponentially Quadratically actually. Exponential would mean each user doubles the amount of calculations, for instance.

by 013a 9y ago

bpicolo 9y ago

> and it doesn't have as stringent real-time requirements Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc. Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away

kevingadd 9y ago
Battle Royale games generally need to broadcast player state to all players within a large distance (iirc for PUBG, it's one km) because it's common to spot another player 500+ meters out and take some shots at them. If they're not moving much you might hit them.
- bpicolo 9y ago
  Even if all 100 players are nearby, 100 isn't unusual in the context of MMOs. That's not to say it's trivial to do, it's just not a new feat unique to battle royale games.

Thaxll 9y ago

Since gameservers are mostly single threaded they probably run 35~~ of them on a single instance ( c4.8xlarges have 36 vcores ), so if you have a good matchmaking algorithm you can fit 3500~~ players per instance, to reach 3.5M you need between 1000 and 2000 instances. As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.

Flammy 9y ago
Likely more than 35 games on a single instance, as ideally you wouldn't be starting all games at the same time on the same instance. Staggering the start of games on each instance would be ideal. For example you could have dozens of late stage games that combined have less players than a single, just started, 100 person game.
- Thaxll 9y ago
  You usually don't run two or more game server on single a thread / core, it kills performance, and with VMs with many physical processor you have other issues like NUMA so you have to be very careful. I'm pretty sure they have a ratio of one gameserver per thread / core.
- evanosaurus 9y ago
  We run 16 game servers per c4.8xl instance for Battle Royale. We run more than that (on a different instance type) for StW.

vvanders 9y ago

FYI most game network servers tick the network at 10-12Hz(although the internal sim may run higher). We generally didn't go higher than 10Hz since you'd start saturating lower end connections and any well written game will be good at reconciling state via dead-reckoning. You have to handle 100ms+ spikes anyway so it doesn't make sense to run the network super-fast.

troncjb 9y ago
What do you mean by "game network servers tick the network"? Like data going server -> client ?
AlphaSite 9y ago
Most competitive games go well above 10Hz ticks, OW does it, CSGO does it (I don't really know the tick-rates of other games but when it matters, so no comment there, but there is an important area for it). OW just has a restricted rate option for low bandwidth connections (added when they went from 23 -> 61 Hz).
- vvanders 9y ago
  So that might be the case for LAN games, generally internet tends to stay lower. Valve for instance recommends 13Hz[1] for Source(CSGO): > // 60 for updaterate is LAN ONLY use 13 for internet > // 20 is default but will cut the maxplayers you can handle in 1/2 > // for SRCDS Servers use 30 - you might be able to use 20 > // sv_maxupdaterate 60 > sv_maxupdaterate 13 Like I said, depending on how good your potential-vis code is you may be able to go higher but you risk saturating your client links and most game netcode doesn't handle congestion nicely. My info is a little out of date(2-3 years) but back then we had to keep data rates below 20kb/sec(and ideally 10kb/sec) if you wanted low jitter across the link. [1] https://support.steampowered.com/kb_article.php?ref=5386-HMJI-5162 https://support.steampowered.com/kb_article.php?ref=5386-HMJ...
  - learc83 9y ago
    >So that might be the case for LAN games Overwatch has a 63Hz tick rate over the internet.

welder 9y ago

> No one at their scale actually pays the list price. I've always wondered what AWS rates do large users receive from negotiations?

AlphaSite 9y ago

MAG somehow managed to be a competitive FPS with 256 players back in 2008ish, its not an impossible battle.

jsmeaton 9y ago
I think MAG was using clever tricks. There were multiple “zones” per battle, and I think each of those zones was a separate instance with some coordination between.
- xchaotic 9y ago
  to be fair you could apply the same tricks to any Battle Royale type game

rocqua 9y ago

> Additionally, Battle Royale style games are the most demanding multiplayer games ever created. I'd like to draw your attention to Planetside 2. A pretty recent FPS that was run a lot like WoW, with 3 'continents' and complete seamlessness on those continents. Firefights where 100 people were actively engaging each other were an almost constant experience back when I played it. I think there was a cap of 1000 players per continent. Those players could gather wherever they wanted. I'm pretty sure I was in some 300 player battles, and nothing but lag would prevent all 1000 people on a continent from going to the same base. It has to be said that in the big battles (at around a 100) lag became an issue. I'm not sure whether this was clientside, serverside or both. (There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)

JCharante 9y ago
> It has to be said that in the big battles (at around a 100) lag became an issue. I remember the crown frequently having 96+ per faction without any lag problems. Must have been clientside. > (There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded) It was 6000, and it was terrible to be in. It took me 10 minutes to enter a station (normally takes a second) and I had to relog because I couldn't get out of the station. Relogging took another hour of black screens. It was pretty neat when I finally got out, but there wasn't much of a fight compared to previous encounters.
- wyager 9y ago
  Planetside 2 was horrendously single-threaded, so anything but recent Intel chips would leave you with really bad performance in large battles. I would often max out my AMD CPU (well, one core of it) at around 12 FPS without my graphics card breaking a sweat in 100-200 player battles. PS2 was for the most part an amazing game. The graphics were truly beautiful and the gameplay was very fun. Unfortunately, their shitty performance basically killed the game by making it unplayable for many users during the trademark huge battles.

mbid 9y ago

> Its an n^2 problem > actually increases resource utilization exponentially I think you'll have to choose there.

evanosaurus 9y ago

We don’t tick at 30Hz. Somewhat slower than that. Working on making that better, though. And we do indeed take advantage of reserved instances. We’re also looking at other options along these lines.

bufferoverflow 9y ago

> actually increases resource utilization exponentially Quadratically actually. Exponential would mean each user doubles the amount of calculations, for instance.