3 ms·
This question sounds like it's pointed directly at me. However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in
by dijit 7y ago
This question sounds like it's pointed directly at me.
However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company.
My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2).
Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++
Things like: matchmaking will happen ideally on a single machine with no shared state, everything will happen in memory, which is much faster and can be more reliable than any distributed state or failover.
(it's less reliable if you're in matchmaking and the server or service dies; But then everyone's client will reconnect to the newly scheduled matchmaking instance and populate in memory state.)
We use a lot of windows, nearly every machine that doesn't handle state is a windows server. This has pros and cons, from my ops perspective I try to treat windows like cattle, but windows doesn't like that. they have their own way of operating fleets of machines which include using SCCM and packaging things. There's nice GUI's, but we use saltstack and we removed AD, because it was a huge burden to: create a machine, link it to AD, reboot it, finally get a machine worth using.
From a dev perspective, Windows is good, IO Completion Ports is superior to the linux epoll in terms of interface and performance, so we can have machines that take 200,000+ connections.
How you decide which dedicated server you connect to is up to your client, it does a naive check as it's logging in where it will do a TLS handshake with a random gameserver in a region, for each region. (during the login phase we send your client a list of all currently active gameservers and an int to represent the region).
This works fine until there's packet loss on a particular path because your single ping might be fine but overall your experience could be better elsewhere; if you're not able to ping anything then we fall back on geoip.
That said, if you have friends on another datacenter, we try to put you on the same server. So that if you join groups or whatever then it's just a phase transition rather than a full server sync.
Everything is orchestrated with a "core" set of servers which handle player authentication and matchmaking centrally, then each of the gameserver datacenters (geographically distributed to be closer to players) is attached via ipsec VPN.
In Division 2 we spread out into GCP as well as having physical machines, so we developed a custom auto-scaler. The autoscaler checks the capacity of a region and how many players are currently in a region, keeps a record over 10 minutes and makes a prediction. If the prediction goes over the current capacity in 20 minutes or less, it will create a new instance (since making windows servers on GCP takes longer than linux servers)
If the prediction goes lower than the capacity of a server, it will send a decomission request to the machine, which takes up to 3hrs to complete (to give people time to leave the server naturally).
Idk, I've been doing this for 5 years now so I can talk at length about how we do it, but ultimately out biggest challenges are the fact that we can't use cool shit or new trends because latency matters a lot and we use windows everywhere.
--
As an aside; the overwhelming majority of other ubisoft games (excption: For Honor) use something very similar to what we released open source in collaboration with google to do matchmaking: https://agones.dev/site/ https://agones.dev/site/
- Slartie 7y agoWhat is the rationale behind using Windows on the servers? I'm wondering because the argument I usually hear is that someone high up in the ops team hierarchy declares "but we need everything in the AD, including all the servers, otherwise I can't sleep well". In your case however this apparently doesn't apply, as you do not register them in the AD. Is it so the game devs don't need to write cross-platform server code?
- dijit 7y agoReal reasons for: * One platform to develop on; back-end coders tend to go back and forth between client and server programming. * Faster iterations. (just hit F5 in visual studio) * IOCP Stupid reasons for: * Old IT Director denied the use of virtualisation software. (mac address randomisation wasn't great and it caused a switch crash if two people had the same mac) * Windows licenses are really cheap compared to developer time. (until we went to cloud, where Microsoft charges insane amounts for licensing)
- speedplane 7y agoI no longer do windows development, but I miss it. It's strengths are greater than you portray. The C# / .NET libraries are really beautiful and well thought out. I most often program in Python, and while the Python language is more concise and elegant than C#, the Python libraries are not nearly as well organized and consistent as C# / .NET. Java libraries are better than Python but worse than C#, and the Java language is worse than both C# and Python. As far as dev enivronments, Visual Studio is far better than Eclipse and roughly equal to Python's PyCharm. Because C# is compiled, it's far more performant than Python and a bit better than Java. The downside of course, is that once you start down that road, you are pretty locked into the MS world. If I was starting a new project, and I was a scrappy startup, I'd probably go with Python. But if I had resources (ie $$$) to support MS licenses and performance actually mattered, I'd go with MS. Side projects for experimentation may lead to node or rust. But sorry Java, can't really imagine a scenario where I'd start with you.
- 7y ago