7 ms·

Valorant's 128-Tick Servers (2020)

220 points by nairadithya 1y ago

whalesalad 1y ago
I wonder if any game servers are implemented in Erlang?
- seivan 1y ago
  Network connection, lobby, matchmaking, leaderboards or even chats, yes. But the actual simulation, probably not for fast paced twitchy shooter. Also not just for performance reasons, I wouldn’t call BeamVM hard realtime, but also for code. Your game server would usually be the client but headless (without rendering). Helps with reuse and architecture.
  - bogwog 1y ago
    IIRC, Activision/Blizzard uses Erlang for their matchmaking systems (or used to... I saw it in a very old talk)
  - mikhmha 1y ago
    In the case of Call of Duty: Black Ops 1. Thee matchmaking + leaderboards system was implemented by DemonWare (3rd party) in Erlang. Erlang actually has good enough performance for many types of multiplayer games. Though you are correct that it may not cut it for fast paced twitch shooters. Well...I'm not exactly sure about that. You can offload lots of expensive physics computations to NIF's. In my game the most expensive computation is AI path-finding. Though this never occurs on the main simulation tick. Other processes run this on their own time.
    - array_key_first 1y ago
      The biggest hurdle to a game server written entire on the BEAM is the GC. GC pauses just take too much time, and when you need to get out (for example) 120 updates per second, you can't afford it. Even offloading stuff to C or C++ does not save you, because you either have to use the GC, do a copy, or both. Game servers typically use very cheap memory allocation techniques like arenas and utilize DOD. It's not uncommon for a game server simulation to be just a bunch of arrays that you grow, never shrink, and then reset at the end of the game.
      - mikhmha 1y ago
        Good point. Yeah I guess it wouldn't cut it for any fast-paced twitch shooter. Especially with a 120 update per second deadline. A non-deterministic GC pause could have disastorous effects, especially in a tense shootout. I don't know much about GC theory but the GC in BEAM is per process and heap-based? I'm not sure exactly what that entails, but can you not structure the main simulation process to take advantage of this fact? I find myself interested in developing multi-player simulations with more flexible deadlines. My MMO runs at 10 ticks. And its not twitch-based. So the main simulation process can have pauses and it wouldn't have a big impact on gameplay. Though this has never occurred. As long as: (tick process time) + (send update to clients) + (gc pause) < 100ms, everything is fine?. (My assumption). Btw what does DOD mean? Is it Data on Demand? Since my game is persistent I can't reset arrays at some match end state. So I store things either in maps on the main server process or I store it in the dedicated client process state (can only be updated via server process).
- echelon 1y ago
  It sounds like they're so heavily invested in Unreal Engine that it's become the entire stack. I was imagining some blindingly fast C or Rust on bare metal. That UE4 code snippet is brutal on the eyes.
- andrewflnr 1y ago
  I distinctly remembered that Eve Online was in Erlang, went to go find sources and found out I was 100% wrong. But I did find this thread about a game called "Vendetta Online" that has Erlang... involved, though the blog post with details seems to be gone. Anyway, enjoy! http://lambda-the-ultimate.org/node/2102 http://lambda-the-ultimate.org/node/2102
  - Cyph0n 1y ago
    Eve used Stackless Python. CoD Black Ops used/uses Erlang for most of its backend afaik. https://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf https://www.erlang-factory.com/upload/presentations/395/Erla...
    - andrewflnr 1y ago
      There's the real answer. This should be a reply to the original question!
- mikhmha 1y ago
  I am currently doing this! Working on an MMO game server implemented in Elixir. It works AMAZING and you get so much extra observability and reliability features for FREE. I don't know why its not more popular. Before I started the project, some people said that BeamVM would not cut it for performance. But this was not true. For many types of games, we are not doing expensive computation on each tick. Rather its just checking rules for interactions between clients and some quick AABB + visibility checks.
- Thaxll 1y ago
  You'll never get a modern FPS gameserver with good performance written in a GC language. Erlang is also pretty slow, it's Python like performance. Very far from C#, Go and Java. The other reason is that the client and the server have to be written in the same language.
  - Sohcahtoa82 1y ago
    > The other reason is that the client and the server have to be written in the same language. This isn't true at all. Sure, it can help to have both client and server built using the same engine or framework, but it's not a hard requirement. Heck, the fact that you can have browser-based games when the server is written in Python is proof enough that they don't need to be the same language.
    - Thaxll 1y ago
      Browser game don't need performance, I'm talking about AAA online games here, which 99% are built in c++ and the rest in c#.
      - Sohcahtoa82 1y ago
        > Browser game don't need performance Mobile users will hate you when your game drains their battery much faster than it should. > I'm talking about AAA online games here, which 99% are built in c++ and the rest in c#. It still doesn't apply. There's absolutely nothing stopping you from having a server written in Java with a game client written in C#, C++, or whatever. I'm really curious why you think client and server must be written in the same language. A TCP socket is a TCP socket. It doesn't matter what language opens the connection. You can send bytes down from one side and decode them on the other. I mean, sure, if you're writing the server in Java and use the language's object serialization functions to encode them, you might have a hard time decoding them on the other side if the client is in C, but the answer then is to not use Java's object serialization functions. You'll roll your own method of sending updates between client and server.
        1y ago
- aftbit 1y ago
  I've been told that Erlang is somewhat popular for matchmaking servers. It ran the Call of Duty matchmaking at one point. Not the actual game servers though - those are almost certainly C++ for perf reasons.
mmmeff 1y ago
Take notes, Valve.
- Linkd 1y ago
  As a lifelong Valve/CS fan, I've been so disappointed with subtick. It was pitched as generational evolution to the games netcode. Yet years later they're still playing catchup to what CS:GO provided.. Hopefully competition from Valorant and others puts more pressure to make things happen at Valve.
  - koakuma-chan 1y ago
    I didn’t notice any difference between 64 and subtick.
    - switz 1y ago
      There’s a long history of subtick bugs that have been identified and patched over the years. CS2 still isn’t quite as stable as 128-tick CS:GO (perhaps benefitting from a decade of patches and simpler architecture)
- Thev00d00 1y ago
  Sub tick is probably more accurate overall but I do think the cs2 animation netcode is crap and hides a lot of the positives. Hopefully moving to Animgraph 2 will help that, who knows
  - deleted 1y ago
    [deleted]
  - ZeWaka 1y ago
    The partially-implemented animgraph2 heroes in Deadlock are looking pretty good and hits feel accurate in-game.
  - charlie90 1y ago
    random fun fact but the animgraph in source 2 is based on code from here: https://github.com/BobbyAnguelov/Esoterica https://github.com/BobbyAnguelov/Esoterica, thought it was interesting when I saw common symbol names
deleted 1y ago
[deleted]
jauntywundrkind 1y ago
> In VALORANT’s case, .5ms is a meaningful chunk of our 2.34ms budget. You could process nearly a 1/4th of a frame in that time! There’s 0% chance that any of the game server’s memory is still going to be hot in cache. This feels like an unideal architectural choice, if this is the case!? Sounds like each game server is independent. I wonder if anyone has more shared state multi-hosting? Warm up a service process, then fork it as needed, so there's some share i-cache? Have things like levels and hit boxes in immutable memfd, shared with each service instance, so that the d-cache can maybe share across instances? With heartbleed et al, a context switch probably has to totally burn down the caches now a days? So maybe this wouldn't be enough to keep data hot, that you might need a multi-threaded not multi-process architecture to see shared caching wins. Obviously I dunno, but it feels like caches are shorter lived than they used to be! I remember being super hopeful that maybe something like Google Stadia could open up some interesting game architecture wins, by trying to render multiple different clients cooperatively rather than as individual client processes. Afaik nothing like that ever emerged, but it feels like there's some cool architecture wins out there & possible.
- Ellipsis753 1y ago
  It does sound like each server is its own process. I think you're correct that it would be a little faster if all games shared a single process. That said, then if one crashed it'd bring the rest down. This is one of those things that might take weeks just to _test_. Personally I suspect the speedup by merging them would be pretty minor, so I think they've made the right choice just keeping them separate. I've found context switching to be surprisingly cheap when you only have a few hundred threads. But ultimately, no way to know for sure without testing it. A lot of optimization is just vibes and hypothesize.
deathanatos 1y ago
128 ticks per second servers. (And lo, suddenly the article's thesis is inherently clear.) A "tick", or an update, is a single step forward in the game's state. UPS (as I'll call it from here) or tick rate is the frequency of those. So, 128 ticks/s == 128 updates per sec. That's a high number. For comparison, Factorio is 60 UPS, and Minecraft is 20 UPS. At first I imagined an FPS's state would be considerably smaller, which should support a higher tick rate. But I also forgot about fog of war & visibility (Factorio for example just trusts the clients), and needing to animate for hitbox detection. (Though I was curious if they're always animating players? I assume there'd be a big single rectangular bounding box or sphere, and only once a projectile is in that range, then animations occur. I assume they've thought of this & it just isn't in there. But then there was the note about not animating the "buy" portion, too…)
- actionfromafar 1y ago
  And Fortnite is allegedly 30 ticks per second.
  - Hikikomori 1y ago
    And a few more players.
  - NekkoDroid 1y ago
    Apex Legends is at 20 IIRC (My memory is from 2-3 years back on this one tho) CSGO was at 64 for the standard servers and 128 for Faceit (IIRC CS2 is doing some dynamic tick schenanigans unless they changed back on that) Overwatch is I think at 60
    - shaokind 1y ago
      CS2 is 64 tick under the hood, with interpolation between the ticks. In the beta, server operators could modify the tick rate by patching the server binary, but when that revealed inconsistencies (which was meant to be avoided with the "subtick" system), they hard coded the client side tick rate to 64 [0]. [0]: https://twitter.com/thexpaw/status/1702277004656050220 https://twitter.com/thexpaw/status/1702277004656050220
holoduke 1y ago
But animations are now lerped after each 4 frames. Do tickrate is 32 with interpolation. Not sure if sudden direction changes now might result in ghost hits. Some hardcore quake fans probably know the answer.
xmprt 1y ago
We should add 2020 to this. I read this article earlier and thought there had been some updates to the architecture.
deleted 1y ago
[deleted]
mmanfrin 1y ago
Great irony of finishing a league game just now where the whole game lagged (for everyone in the game) to find this at the top of HN.
- ajkjk 1y ago
  not very ironic at all really, since they're different games?
  - dankwizard 1y ago
    Same company, different focus
    - ajkjk 1y ago
      And massively legacy codebase vs fairly new codebase
  - deleted 1y ago
    [deleted]
greatgib 1y ago
This deep dive article is very nice. At any given time, ~50 of those games are going to be in the buy phase. Players will be purchasing equipment safely behind their spawn barriers and no shots can hurt them. We realized we don’t even need to do any server-side animation during the buy phase, we could just turn it off. That explains the current trend of "online" video game that is so annoying: For 10 minutes of play, you have to wait for 10 minutes of lobby time and forced animations, like end game animations. On BO6 it kills me, you just want to play, sometimes you don't have more than 30 minutes for a quick video game session, and with the current games, you always have to wait a very very long time. Painfully annoying.
- nemothekid 1y ago
  This is not equivalent to "lobby time or end game animations" in other games. In Valorant (similar to Counter Strike), at the start of the game you have 60 seconds to buy your weapons and abilities for the round. Valorant/CS is typically a best-of-13, and before each round is a 60 second "buy" period.
  - typewithrhythm 1y ago
    It's the idea that if they leave more players idling in a lobby, but period, or animation, that it costs them less. It's a deceptive way to sell people less game.
    - gopher2000 1y ago
      > It's a deceptive way to sell people less game. That's a dumb take. The buying phase is an integral part of the game mode. And the game is free.
  - dgunay 1y ago
    In CS you can leave the buy zone immediately. I don't necessarily believe that Valorant's decision to fence players in their spawn for the first minute during the buy period is simply to save on server costs, especially because they realized that optimization possibility after the fact. Being able to buy your weapons quickly may have an element of skill, but it doesn't make for particularly interesting gameplay. They may have just decided that they'd rather level this part of the playing field so people can focus on the core tactical FPS gameplay.
Liquix 1y ago
very interesting read, it seems like management/engineering/vendors were all willing to get on the same page to hit the frame budget. especially the bit about profiling every line of game code into an appropriate bucket - sounds like a lot of work which paid off handsomely. If you just make a list of “performance tweaks” you might learn about in, say, a game dev blog post on the internet, and execute them without considering your application’s specific needs and considerations, you might hurt performance more than you help it. nice.
syspec 1y ago
This post reads less like an engineering deep dive and more like a Xeon product brochure that wandered into a video game blog. They casually name-drop every Intel optimization short of tattooing "Hyperthreaded" on their foreheads.
- mtoner23 1y ago
  well of course they would. they bought all intel hardware. and they are making one of the most perfromant multiplayer servers ever. they should be mentioning every optimization possible. if they had amd thread ripper servers they would mention all those features too.
Havoc 1y ago
You can mess with the code all day long, but you're not getting away from raw latency. The modern matchmaking approach groups people by skill not latency, so you get a pretty wild mix of latency. It feels nothing like the old regional servers. Sure the skill mix was varied, but at least you got your ass handed to you in crisp <10ms by actual skill. Now it's all getting knife noscoped around a corner by a guy that rubberbanded 200ms into the next sector of the map already while insulting your mom and wearing a unicorn skin
- bob1029 1y ago
  I miss playing on consistent <10ms servers in the CS 1.6 days. The Houston/Dallas/Austin/San Antonio region was like a mini universe of highly competitive FPS action. My 2mbps roadrunner cable modem could achieve single digit ping from Houston to Dallas. Back in those days we plugged the modem directly into the gaming PC.
- ROBLOX_MOMENTS 1y ago
  > The modern matchmaking approach groups people by skill not latency I work at a game studio and something I have seen is that nobody is on wired anymore. You are poweruser if you are on wired. Significantly the 99% of users will be on mobile or wifi and be 10ms to first hop or two hop.
- wyldberry 1y ago
  Good thing they thought of that. Disclaimer: I was at Riot During some of the Valorant dev cycle and the stated goal in this tech blog [0] was a huge goal (keeping latency < 35ms). This was only really doable because Riot has invested significantly in buying dark fiber and peering at major locations worldwide [1][2] [0] - https://technology.riotgames.com/news/peeking-valorants-netcode https://technology.riotgames.com/news/peeking-valorants-netc... [1] - https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-i https://technology.riotgames.com/news/fixing-internet-real-t... [2] - https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-ii https://technology.riotgames.com/news/fixing-internet-real-t...
  - 1y ago
uyzstvqs 1y ago
Counter-Strike: Global Offensive was also able to handle 128 TPS just fine. They just chose to never implement it in official matchmaking (64 TPS). It did work very smoothly on community servers. Counter-Strike 2 implements a controversial "sub tick" system on top of 64 TPS. It is not comparable to actual 128 TPS, and often worse than standard 64 TPS in practice.
- deleted 1y ago
  [deleted]
- forrestthewoods 1y ago
  Lots of things work fine when you throw twice as many dollars at them. It’s not a matter of it working or not. It’s a matter of economics. Most game servers are single threaded because the goal is to support the maximum number of players per dollar. A community server doesn’t mind throwing more compute dollars to support more players or higher tick rate. When you have one million concurrent players - as CounterStrike sometimes does - the choice may be different.
  - Dylan16807 1y ago
    Sure, twice as many dollars in the immediate term. In the context of a particular tick rate being decade(s) old, it's more like deciding whether you can host 50x or 100x as many players per dollar of infrastructure. The upside of a higher tick rate grows as computers and connections improve and the downside shrinks semi-exponentially.
    - forrestthewoods 1y ago
      Also twice as many dollars in the future term! The downside very much does not decrease “exponentially”. It would be interesting to know what Valve’s server costs look like over time. They definitely spend the pretty penny. And any business would prefer to spend one penny rather than two.
      - Dylan16807 1y ago
aftbit 1y ago
I vaguely remember Counter-Strike Source servers running at 33, 66, or 100 tick. My high school gaming clan was called "10tik", poking fun at the ancient Pentium box that I ran the CSS server on.
ghshephard 1y ago
They bury/obscure a quite important detail in this article: | We were still running on the older Intel Xeon E5 processors, ... | Moving to the more modern Xeon Scalable processors showed major performance gains for our server application But - I was unable to find any mention in the article as to what processors they were actually comparing in their before/after.
Gravityloss 1y ago
Quakeworld runs on 72 fps since 1996, though clients probably could reach that reliably only years later...

220 points by nairadithya 1y ago

whalesalad 1y ago

I wonder if any game servers are implemented in Erlang?

seivan 1y ago
Network connection, lobby, matchmaking, leaderboards or even chats, yes. But the actual simulation, probably not for fast paced twitchy shooter. Also not just for performance reasons, I wouldn’t call BeamVM hard realtime, but also for code. Your game server would usually be the client but headless (without rendering). Helps with reuse and architecture.
- bogwog 1y ago
  IIRC, Activision/Blizzard uses Erlang for their matchmaking systems (or used to... I saw it in a very old talk)
- mikhmha 1y ago
  In the case of Call of Duty: Black Ops 1. Thee matchmaking + leaderboards system was implemented by DemonWare (3rd party) in Erlang. Erlang actually has good enough performance for many types of multiplayer games. Though you are correct that it may not cut it for fast paced twitch shooters. Well...I'm not exactly sure about that. You can offload lots of expensive physics computations to NIF's. In my game the most expensive computation is AI path-finding. Though this never occurs on the main simulation tick. Other processes run this on their own time.
  - array_key_first 1y ago
    The biggest hurdle to a game server written entire on the BEAM is the GC. GC pauses just take too much time, and when you need to get out (for example) 120 updates per second, you can't afford it. Even offloading stuff to C or C++ does not save you, because you either have to use the GC, do a copy, or both. Game servers typically use very cheap memory allocation techniques like arenas and utilize DOD. It's not uncommon for a game server simulation to be just a bunch of arrays that you grow, never shrink, and then reset at the end of the game.
    - mikhmha 1y ago
      Good point. Yeah I guess it wouldn't cut it for any fast-paced twitch shooter. Especially with a 120 update per second deadline. A non-deterministic GC pause could have disastorous effects, especially in a tense shootout. I don't know much about GC theory but the GC in BEAM is per process and heap-based? I'm not sure exactly what that entails, but can you not structure the main simulation process to take advantage of this fact? I find myself interested in developing multi-player simulations with more flexible deadlines. My MMO runs at 10 ticks. And its not twitch-based. So the main simulation process can have pauses and it wouldn't have a big impact on gameplay. Though this has never occurred. As long as: (tick process time) + (send update to clients) + (gc pause) < 100ms, everything is fine?. (My assumption). Btw what does DOD mean? Is it Data on Demand? Since my game is persistent I can't reset arrays at some match end state. So I store things either in maps on the main server process or I store it in the dedicated client process state (can only be updated via server process).
echelon 1y ago
It sounds like they're so heavily invested in Unreal Engine that it's become the entire stack. I was imagining some blindingly fast C or Rust on bare metal. That UE4 code snippet is brutal on the eyes.
andrewflnr 1y ago
I distinctly remembered that Eve Online was in Erlang, went to go find sources and found out I was 100% wrong. But I did find this thread about a game called "Vendetta Online" that has Erlang... involved, though the blog post with details seems to be gone. Anyway, enjoy! http://lambda-the-ultimate.org/node/2102 http://lambda-the-ultimate.org/node/2102
- Cyph0n 1y ago
  Eve used Stackless Python. CoD Black Ops used/uses Erlang for most of its backend afaik. https://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf https://www.erlang-factory.com/upload/presentations/395/Erla...
  - andrewflnr 1y ago
    There's the real answer. This should be a reply to the original question!
mikhmha 1y ago
I am currently doing this! Working on an MMO game server implemented in Elixir. It works AMAZING and you get so much extra observability and reliability features for FREE. I don't know why its not more popular. Before I started the project, some people said that BeamVM would not cut it for performance. But this was not true. For many types of games, we are not doing expensive computation on each tick. Rather its just checking rules for interactions between clients and some quick AABB + visibility checks.
Thaxll 1y ago
You'll never get a modern FPS gameserver with good performance written in a GC language. Erlang is also pretty slow, it's Python like performance. Very far from C#, Go and Java. The other reason is that the client and the server have to be written in the same language.
- Sohcahtoa82 1y ago
  > The other reason is that the client and the server have to be written in the same language. This isn't true at all. Sure, it can help to have both client and server built using the same engine or framework, but it's not a hard requirement. Heck, the fact that you can have browser-based games when the server is written in Python is proof enough that they don't need to be the same language.
  - Thaxll 1y ago
    Browser game don't need performance, I'm talking about AAA online games here, which 99% are built in c++ and the rest in c#.
    - Sohcahtoa82 1y ago
      > Browser game don't need performance Mobile users will hate you when your game drains their battery much faster than it should. > I'm talking about AAA online games here, which 99% are built in c++ and the rest in c#. It still doesn't apply. There's absolutely nothing stopping you from having a server written in Java with a game client written in C#, C++, or whatever. I'm really curious why you think client and server must be written in the same language. A TCP socket is a TCP socket. It doesn't matter what language opens the connection. You can send bytes down from one side and decode them on the other. I mean, sure, if you're writing the server in Java and use the language's object serialization functions to encode them, you might have a hard time decoding them on the other side if the client is in C, but the answer then is to not use Java's object serialization functions. You'll roll your own method of sending updates between client and server.
      - 1y ago
aftbit 1y ago
I've been told that Erlang is somewhat popular for matchmaking servers. It ran the Call of Duty matchmaking at one point. Not the actual game servers though - those are almost certainly C++ for perf reasons.

mmmeff 1y ago

Take notes, Valve.

Linkd 1y ago
As a lifelong Valve/CS fan, I've been so disappointed with subtick. It was pitched as generational evolution to the games netcode. Yet years later they're still playing catchup to what CS:GO provided.. Hopefully competition from Valorant and others puts more pressure to make things happen at Valve.
- koakuma-chan 1y ago
  I didn’t notice any difference between 64 and subtick.
  - switz 1y ago
    There’s a long history of subtick bugs that have been identified and patched over the years. CS2 still isn’t quite as stable as 128-tick CS:GO (perhaps benefitting from a decade of patches and simpler architecture)
Thev00d00 1y ago
Sub tick is probably more accurate overall but I do think the cs2 animation netcode is crap and hides a lot of the positives. Hopefully moving to Animgraph 2 will help that, who knows
- deleted 1y ago
  [deleted]
- ZeWaka 1y ago
  The partially-implemented animgraph2 heroes in Deadlock are looking pretty good and hits feel accurate in-game.
- charlie90 1y ago
  random fun fact but the animgraph in source 2 is based on code from here: https://github.com/BobbyAnguelov/Esoterica https://github.com/BobbyAnguelov/Esoterica, thought it was interesting when I saw common symbol names

deleted 1y ago

[deleted]

jauntywundrkind 1y ago

> In VALORANT’s case, .5ms is a meaningful chunk of our 2.34ms budget. You could process nearly a 1/4th of a frame in that time! There’s 0% chance that any of the game server’s memory is still going to be hot in cache. This feels like an unideal architectural choice, if this is the case!? Sounds like each game server is independent. I wonder if anyone has more shared state multi-hosting? Warm up a service process, then fork it as needed, so there's some share i-cache? Have things like levels and hit boxes in immutable memfd, shared with each service instance, so that the d-cache can maybe share across instances? With heartbleed et al, a context switch probably has to totally burn down the caches now a days? So maybe this wouldn't be enough to keep data hot, that you might need a multi-threaded not multi-process architecture to see shared caching wins. Obviously I dunno, but it feels like caches are shorter lived than they used to be! I remember being super hopeful that maybe something like Google Stadia could open up some interesting game architecture wins, by trying to render multiple different clients cooperatively rather than as individual client processes. Afaik nothing like that ever emerged, but it feels like there's some cool architecture wins out there & possible.

Ellipsis753 1y ago
It does sound like each server is its own process. I think you're correct that it would be a little faster if all games shared a single process. That said, then if one crashed it'd bring the rest down. This is one of those things that might take weeks just to _test_. Personally I suspect the speedup by merging them would be pretty minor, so I think they've made the right choice just keeping them separate. I've found context switching to be surprisingly cheap when you only have a few hundred threads. But ultimately, no way to know for sure without testing it. A lot of optimization is just vibes and hypothesize.

deathanatos 1y ago

128 ticks per second servers. (And lo, suddenly the article's thesis is inherently clear.) A "tick", or an update, is a single step forward in the game's state. UPS (as I'll call it from here) or tick rate is the frequency of those. So, 128 ticks/s == 128 updates per sec. That's a high number. For comparison, Factorio is 60 UPS, and Minecraft is 20 UPS. At first I imagined an FPS's state would be considerably smaller, which should support a higher tick rate. But I also forgot about fog of war & visibility (Factorio for example just trusts the clients), and needing to animate for hitbox detection. (Though I was curious if they're always animating players? I assume there'd be a big single rectangular bounding box or sphere, and only once a projectile is in that range, then animations occur. I assume they've thought of this & it just isn't in there. But then there was the note about not animating the "buy" portion, too…)

actionfromafar 1y ago
And Fortnite is allegedly 30 ticks per second.
- Hikikomori 1y ago
  And a few more players.
- NekkoDroid 1y ago
  Apex Legends is at 20 IIRC (My memory is from 2-3 years back on this one tho) CSGO was at 64 for the standard servers and 128 for Faceit (IIRC CS2 is doing some dynamic tick schenanigans unless they changed back on that) Overwatch is I think at 60
  - shaokind 1y ago
    CS2 is 64 tick under the hood, with interpolation between the ticks. In the beta, server operators could modify the tick rate by patching the server binary, but when that revealed inconsistencies (which was meant to be avoided with the "subtick" system), they hard coded the client side tick rate to 64 [0]. [0]: https://twitter.com/thexpaw/status/1702277004656050220 https://twitter.com/thexpaw/status/1702277004656050220

holoduke 1y ago

But animations are now lerped after each 4 frames. Do tickrate is 32 with interpolation. Not sure if sudden direction changes now might result in ghost hits. Some hardcore quake fans probably know the answer.

xmprt 1y ago

We should add 2020 to this. I read this article earlier and thought there had been some updates to the architecture.

deleted 1y ago

[deleted]

mmanfrin 1y ago

Great irony of finishing a league game just now where the whole game lagged (for everyone in the game) to find this at the top of HN.

ajkjk 1y ago
not very ironic at all really, since they're different games?
- dankwizard 1y ago
  Same company, different focus
  - ajkjk 1y ago
    And massively legacy codebase vs fairly new codebase
- deleted 1y ago
  [deleted]

greatgib 1y ago

This deep dive article is very nice. At any given time, ~50 of those games are going to be in the buy phase. Players will be purchasing equipment safely behind their spawn barriers and no shots can hurt them. We realized we don’t even need to do any server-side animation during the buy phase, we could just turn it off. That explains the current trend of "online" video game that is so annoying: For 10 minutes of play, you have to wait for 10 minutes of lobby time and forced animations, like end game animations. On BO6 it kills me, you just want to play, sometimes you don't have more than 30 minutes for a quick video game session, and with the current games, you always have to wait a very very long time. Painfully annoying.

nemothekid 1y ago
This is not equivalent to "lobby time or end game animations" in other games. In Valorant (similar to Counter Strike), at the start of the game you have 60 seconds to buy your weapons and abilities for the round. Valorant/CS is typically a best-of-13, and before each round is a 60 second "buy" period.
- typewithrhythm 1y ago
  It's the idea that if they leave more players idling in a lobby, but period, or animation, that it costs them less. It's a deceptive way to sell people less game.
  - gopher2000 1y ago
    > It's a deceptive way to sell people less game. That's a dumb take. The buying phase is an integral part of the game mode. And the game is free.
- dgunay 1y ago
  In CS you can leave the buy zone immediately. I don't necessarily believe that Valorant's decision to fence players in their spawn for the first minute during the buy period is simply to save on server costs, especially because they realized that optimization possibility after the fact. Being able to buy your weapons quickly may have an element of skill, but it doesn't make for particularly interesting gameplay. They may have just decided that they'd rather level this part of the playing field so people can focus on the core tactical FPS gameplay.

Liquix 1y ago

very interesting read, it seems like management/engineering/vendors were all willing to get on the same page to hit the frame budget. especially the bit about profiling every line of game code into an appropriate bucket - sounds like a lot of work which paid off handsomely. If you just make a list of “performance tweaks” you might learn about in, say, a game dev blog post on the internet, and execute them without considering your application’s specific needs and considerations, you might hurt performance more than you help it. nice.

syspec 1y ago

This post reads less like an engineering deep dive and more like a Xeon product brochure that wandered into a video game blog. They casually name-drop every Intel optimization short of tattooing "Hyperthreaded" on their foreheads.

mtoner23 1y ago
well of course they would. they bought all intel hardware. and they are making one of the most perfromant multiplayer servers ever. they should be mentioning every optimization possible. if they had amd thread ripper servers they would mention all those features too.

Havoc 1y ago

You can mess with the code all day long, but you're not getting away from raw latency. The modern matchmaking approach groups people by skill not latency, so you get a pretty wild mix of latency. It feels nothing like the old regional servers. Sure the skill mix was varied, but at least you got your ass handed to you in crisp <10ms by actual skill. Now it's all getting knife noscoped around a corner by a guy that rubberbanded 200ms into the next sector of the map already while insulting your mom and wearing a unicorn skin

bob1029 1y ago
I miss playing on consistent <10ms servers in the CS 1.6 days. The Houston/Dallas/Austin/San Antonio region was like a mini universe of highly competitive FPS action. My 2mbps roadrunner cable modem could achieve single digit ping from Houston to Dallas. Back in those days we plugged the modem directly into the gaming PC.
ROBLOX_MOMENTS 1y ago
> The modern matchmaking approach groups people by skill not latency I work at a game studio and something I have seen is that nobody is on wired anymore. You are poweruser if you are on wired. Significantly the 99% of users will be on mobile or wifi and be 10ms to first hop or two hop.
wyldberry 1y ago
Good thing they thought of that. Disclaimer: I was at Riot During some of the Valorant dev cycle and the stated goal in this tech blog [0] was a huge goal (keeping latency < 35ms). This was only really doable because Riot has invested significantly in buying dark fiber and peering at major locations worldwide [1][2] [0] - https://technology.riotgames.com/news/peeking-valorants-netcode https://technology.riotgames.com/news/peeking-valorants-netc... [1] - https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-i https://technology.riotgames.com/news/fixing-internet-real-t... [2] - https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-ii https://technology.riotgames.com/news/fixing-internet-real-t...
- 1y ago

uyzstvqs 1y ago

Counter-Strike: Global Offensive was also able to handle 128 TPS just fine. They just chose to never implement it in official matchmaking (64 TPS). It did work very smoothly on community servers. Counter-Strike 2 implements a controversial "sub tick" system on top of 64 TPS. It is not comparable to actual 128 TPS, and often worse than standard 64 TPS in practice.

deleted 1y ago
[deleted]
forrestthewoods 1y ago
Lots of things work fine when you throw twice as many dollars at them. It’s not a matter of it working or not. It’s a matter of economics. Most game servers are single threaded because the goal is to support the maximum number of players per dollar. A community server doesn’t mind throwing more compute dollars to support more players or higher tick rate. When you have one million concurrent players - as CounterStrike sometimes does - the choice may be different.
- Dylan16807 1y ago
  Sure, twice as many dollars in the immediate term. In the context of a particular tick rate being decade(s) old, it's more like deciding whether you can host 50x or 100x as many players per dollar of infrastructure. The upside of a higher tick rate grows as computers and connections improve and the downside shrinks semi-exponentially.
  - forrestthewoods 1y ago
    Also twice as many dollars in the future term! The downside very much does not decrease “exponentially”. It would be interesting to know what Valve’s server costs look like over time. They definitely spend the pretty penny. And any business would prefer to spend one penny rather than two.
    - Dylan16807 1y ago

aftbit 1y ago

I vaguely remember Counter-Strike Source servers running at 33, 66, or 100 tick. My high school gaming clan was called "10tik", poking fun at the ancient Pentium box that I ran the CSS server on.

ghshephard 1y ago

They bury/obscure a quite important detail in this article: | We were still running on the older Intel Xeon E5 processors, ... | Moving to the more modern Xeon Scalable processors showed major performance gains for our server application But - I was unable to find any mention in the article as to what processors they were actually comparing in their before/after.

Gravityloss 1y ago

Quakeworld runs on 72 fps since 1996, though clients probably could reach that reliably only years later...