11 ms·

MMO Architecture: clients, sockets, threads and connection-oriented servers

185 points by cpeterso 3y ago

bullen 3y ago
I made a MMO in 2001 and came to the conclusions that one thread per socket was not going to scale and thought I would mothball the project until that was solved in Java. 2023 and now we have virtual threads that replace blocking IO with non-blocking automatically! Haven't tested it yet though! But I am reviving the 2001 MMO now! In between I made my own MMO backend that uses NIO, conclusion is event-based network and validation is the only way to scale action MMOs past 100 players. Also you need to share memory atomically between cores, so Java or C.
- nvarsj 3y ago
  I feel like the Erlang/OTP would be perfect for the MMO use case. It’s what I’d try first if building from scratch.
  - bullen 3y ago
    No erlang cannot share memory between cores, it has to copy it.
    - klibertp 3y ago
      I think binaries are refcounted and shared in Erlang? I'm not sure if this happens between schedulers/cores, though. They're also obviously immutable, so might not be what you mean by "sharing memory atomically". Other than Erlang, Pony[1] might be an interesting choice. It allows sharing memory, even mutable memory, but it tracks the sharing in the type system. It's been a long time since I looked at it, and it definitely had a bunch of rough edges, but I really liked where the things were going. I hope it got even better since. [1] https://www.ponylang.io/ https://www.ponylang.io/
      - bullen 3y ago
        For me C (1970) and Java (1990) are the only options for eternity. I use some C++ features and dabble in js when I make HTML but really since Applet and Flash has been removed the browser is just a bloated waste of time. Make your apps fast by allowing them to share memory without latency! I have also coded Perl, VisualBasic, php and C# and I'm pretty sure this is it. What we need now are VMs that can take any bytecode/instructionset and translate it to all others. For true crossplatform development. But that requires us to dump dynamic allocation. Arrays of 64 byte structures FTW!
        ReactiveJelly 3y ago
        > VMs that can take any bytecode/instructionset and translate it to all others You're gonna love this: https://en.wikipedia.org/wiki/WebAssembly https://en.wikipedia.org/wiki/WebAssembly
        bullen 3y ago
        I would if anything worked. Still waiting for a Windows release of this: https://github.com/bytecodealliance/wasm-micro-runtime https://github.com/bytecodealliance/wasm-micro-runtime Don't hold you breath. Also needs Risc-V and ARM 32/64.
  - Thaxll 3y ago
    Erlang performance would be a big problem. It's subpar compared to C++ by an order of magnitude.
    - cmdrk 3y ago
      depends on the kind of game you are making. People made MMOs in the year 2000 when CPUs themselves were an order of magnitude slower.
    - nvarsj 3y ago
      Would an MMO backend be CPU bound though? I feel the use case of having a large numbers of active connections with the very distributed nature of passing events around between them all would be perfect for Erlang/OTP. Anything CPU intensive (like dungeon generation or similar) could be offloaded to another service if needed.
      - bullen 3y ago
        You want low latency action, and for that you need one CPU with all cores talking to all the memory without locking. Erlang can't do that, and neither can any other language; except C (Arrays of 64 byte Structures) and Java (Concurrency complete rewrite of the JVM in 1.5 that was adopted and failed for C++11 because you need a VM + GC for that model to be meaningful).
        nvarsj 3y ago
        Does it really need to be low latency? It's not an FPS - so I don't think high tick rate is important. I'd rather have a wider synchronized global state and acceptable server side latency (<500ms) than instant server updates. The client can interpolate everything.
        bullen 3y ago
        Then you are making a slideshow, not a game. The whole point of games is real-time action. It does not need to be violent, but it needs physics that give you the feeling of being there. Multiplayer > VR
  - hmmokidk 3y ago
    I have an MMO demo in Elixir! I haven’t updated it in some time because busy from work but take a look: http://rotb.io http://rotb.io
  - deleted 3y ago
    [deleted]
  - bcrosby95 3y ago
    Erlang/OTP seems perfect for a diablo-esque game where you have lots of instances with a handful of players each. But a giant shared world with thousands of players doesn't seem to play into its wheelhouse as much.
firechickenbird 3y ago
I think your last chart is slightly incorrect: the game server is sending a message directly to the user, whereas it should theoretically only talk to the intermediate message queue (or frontend server)
bob1029 3y ago
I feel like some HFT/fintech concepts are ~ideal MMO architectures. With a 2 tier setup, you could theoretically have a central server processing batches of events that are aggregated across multiple players/regions/etc. A single thread can service upwards of half a billion events per second, and if each of those events covers multiple potential players, then I'd argue we are in a pretty good position. For me, having truly 1 consistent global universe is the only thing that would get me to consider an MMO in 2023+. The technical limitations forcing sharded worlds were excusable when WoW was released, but I don't have patience for those arguments anymore. Not if you want me to expressly burn my time and get paid for it on a recurring basis.
- meheleventyone 3y ago
  > I feel like some HFT/fintech concepts are ~ideal MMO architectures. Can you explain/link to some? I suspect MMO servers, particularly when server authoritative end up having to do a lot more processing per event and tick. Distributing world state updates is also an extremely thorny issue with lots of inter-dependencies.
  - bob1029 3y ago
    See concepts like LMAX Disruptor and https://aeron.io https://aeron.io Whatever is good for low-latency trading systems (i.e. where you are paying contractually for microsecond-level guarantees), is also maybe good for gaming. The whole concept with these patterns is that there is only ever a single writer at any given moment. Single writer principle w/ batching is what gives you enormous uplift in throughput, even when you are necessarily constrained to fully serialized business semantics. I'd argue that an MMO does not need to be as strictly serialized as whatever CBOE, et. al. are doing.
    - meheleventyone 3y ago
      Interesting, thanks! Skimming the Disruptor paper its very similar to how you keep audio context fed using a ring buffer to write ahead as the audio context chews through behind you so the entire thing can remain lock-free and low latency. Typically in a separate thread as the latency and timing requirements are very strict. Aeron on the face of it looks very similar to the architecture diagrams you'd get popping up designing a networking layer in a game already. I suspect the cross-pollination in both directions could be interesting. Neither seem to particularly help with the intensive/interesting bit of an MMO which is running the actual simulation. For example big fleet fights in EVE as far as I remember have never been IO bound rather single-core performance bound. It's common for the bigger Corporations to contact CCP ahead of big fights so the node it's likely to happen on can be moved to a bigger machine. It's actually quite interesting because a lot of it is written in Stackless Python so not the most performant of languages and trapped in a single-threaded context so there is (nominally at least) quite a lot of headroom. For big seamless worlds how you distribute the simulation is the tough bit. Hence companies like Improbable trying to solve that generally.
jongjong 3y ago
I didn't realize that one-thread-per-socket was still something people considered; especially given that many programming languages nowadays have excellent support for async/await which does a great job at managing multiple concurrent connections within a single process/thread and doesn't have the overhead of context-switching.
- deleted 3y ago
  [deleted]
- flohofwoe 3y ago
  In an MMO backend architecture, the way client connections are handled is really just a tiny implemention detail at the edge of the architecture diagram, only affecting a very small part of the overall architecture. Ideally you'd move that stuff out into proxy servers sitting between the clients and the actual game servers with "one connection per client" on the public-network side and "one connection per game server" on the internal-network side. The proxy servers would also deal with validating the incoming traffic (on the message protocol level at least), encryption and of course routing messages between clients and game servers. PS: now that I actually RTFA, the same concept is already mentioned there, it's just called Frontend Server.
- otabdeveloper4 3y ago
  Context switching will happen regardless of where your thread implementation lives. Reimplementing scheduling in a bespoke fashion in usermode will not give you the gains that you hope for. Basically, learn how an OS works first before trying to reimplement it badly.
  - flohofwoe 3y ago
    I guess the idea is that an async/await context switch would always be cheaper than an OS thread context switch. Whether that is actually true probably depends a lot on the specific operating system and async/await runtime (I think there's a lot of mysticism involved seeing async/await as some sort of silver bullet instead of relying on cold hard performance numbers alone).
VoidWhisperer 3y ago
Question with regards to the FIFO queue and processing it: Say a user does two actions in quick succession - action A that is something that inherently takes a bit longer to process on the game server (ie a complicated transaction of some kind) and then right after does action B, one that requires little to no processing. How is it best handled to make sure that the order of actions by the player is maintained with processing? With this scenario, it is possible action B will be processed before action A, leading to the game server considering them in the wrong order. I'm assuming the solution is to have some additional logic to make sure you are only processing one action per player at any given time, but I was wondering if anyone knows any better ways?
- flohofwoe 3y ago
  One way to solve this is to have per-player task queues in the game server which processes tasks sequentially (but concurrently with other player task queues, this is actually a perfect use case for async/await style code). Critical actions should also have a sequence counter to make sure they are enqueued in order and to detect whether actions got lost somewhere (especially when using an unordered networking protocol like UDP) - message ordering usually already happens on a lower level than the game code though in the code layer that sits on top of UDP sockets, but a higher level per-player action counter can't hurt either I guess, if only for debugging)
  - gryn 3y ago
    Only if player tasks are independent. For example, If both are trying to mine a ressource(assuming a shared world) and you do it parallel you might end up with the wrong person winning the mining race condition.
    - flohofwoe 3y ago
      Yeah, synchronization between task queues is left as an excercise to the reader :D But I guess in that specific case it doesn't really matter, the server could just as well roll a dice to decide who wins the race (or rather: access to shared resources between task queues need to be handled like a regular data race in multithreading, what's important is that the server is the only source of truth). Latency-sensitive games also might want to speculate ahead by running the local player's task queue on the client, and roll back when the authorative state coming in from the server disagrees with the speculated client state.
monlockandkey 3y ago
I wonder what a mmo using Golang and goroutines look like?
- flohofwoe 3y ago
  Anectodal, but from what I've seen spending a couple of weeks at a Chinese mobile games company in 2017, Golang game servers are quite popular in China but not elsewhere (in combination with Unity for the game client and Protobuf for the message protocol). I have no idea why that specific tech combo is "big in China" but not outside. AFAIK in the rest of the world, Photon Server is the most popular off-the-shelf solution (not necessarily for true MMOs though). Maybe that's good enough for most games to not wanting to write your own server backend (https://www.photonengine.com/ https://www.photonengine.com/)
  - Daegalus 3y ago
    Well I am building an MMO Server in Go, and I am in the US. Probably because I am a backend/DevOps engineer and i do a lot of Go as it is. I did a small rundown elsewhere in this thread: https://news.ycombinator.com/item?id=37972638 https://news.ycombinator.com/item?id=37972638
- Daegalus 3y ago
  I actually am building an MMO server for an old MMO. And I am using Go and Goroutines for it. I have a small rundown posted here in this thread: https://news.ycombinator.com/item?id=37972638 https://news.ycombinator.com/item?id=37972638
andai 3y ago
See also: Patrick Wyatt interviewed by Casey Muratori at HandmadeCon: https://youtu.be/1faaOrtHJ-A https://youtu.be/1faaOrtHJ-A He was the 2nd employee at Blizzard, did the netcode for their games and also for Guild Wars. His blog is great https://www.codeofhonor.com/blog/tough-times-on-the-road-to-starcraft https://www.codeofhonor.com/blog/tough-times-on-the-road-to-...
- martinohansen 3y ago
  Unrelated to the post here, but I think both Pat and this guy[1] say that they build the multiplayer for Diablo by themselves… Anyhow, both talks are fascinating and surely worth a watch. [1] https://youtu.be/Mlrrc_vy79E?si=Ga50gQyyUXjHwZz0 https://youtu.be/Mlrrc_vy79E?si=Ga50gQyyUXjHwZz0
  - instagib 3y ago
    Thanks for the link but I think David Brevik said a couple guys from blizzard came up to help for 6 months and he had zero multiplayer code built as well as it was his first C program.
  - vitus 3y ago
    Right around 37 minutes in, Brevik mentions that Mike O'Brien (along with Pat, one of the three co-founders of ArenaNet) was the brains behind Battle.net, and then "a few of the guys from [Blizzard South] moved up north during the last six months of development and started making Diablo into multiplayer and integrating Battle.net into the entire thing." That lines up with Pat's assessment: > Initially Collin Murray, a programmer on StarCraft, and I flew to Redwood City to help, while other developers at Blizzard “HQ” in Irvine California worked on network “providers” for battle.net, modem and LAN games as well as the user-interface screens (known as “glue screens” at Blizzard) that performed character creation, game joining, and other meta-game functions.
- DrammBA 3y ago
  That interview was exhausting, I couldn't finish it. I have no idea why the interviewer treated this like a systems design interview at a FAANG, with constant interruptions and unnecesarily rephrasing what the interviewee had already explained. The GDC talk linked in another reply, while only tangetially related, was much more enjoyable.
yobid20 3y ago
Amazed to see connection oriented protocols still being used here. As someone whose been in the real time media space for decades, we switched everything to UDP about 15 years ago and achieved much higher throughput and lower latency. Our server software architectures use a single dedicated thread on a UDP port, which reads packets and distributes them to lockless queues each owned by a dedicated single thread handler exclusively bound to separate physical cores. Similarly for the outbound side. This allows us to scale vertically to maximize a single machine's resources, while allowing us to essentially cap the upper bounds on latency in high load scenarios. Our architecture enables us to have hundreds of thousands of users PER MACHINE. We've been running this architecture for over 10 years now in production successfully. While we do have logic to scale horizontally, we minimize this and use it only when its required for scaling further or traversing specific network clouds, depending on where a user is able to connect. Worth mentioning we use general linux distros on physical hardware. No virtual machines. We also use kernal bypass techniques. Any time you add more servers to spread load, you're increasing latency because for each hop you're traversing the entire software network stack twice plus hops through hardware switches. Nobody uses dedicate thread per client anymore (if they do, its a poor design).
- meheleventyone 3y ago
  Games use UDP quite a lot as well.
- toast0 3y ago
  > Our server software architectures use a single dedicated thread on a UDP port, which reads packets and distributes them to lockless queues each owned by a dedicated single thread handler exclusively bound to separate physical cores. As an internet backseat network performance person... Have you considered one thread per core/NIC queue receiving packets, with RSS (receive side scaling)? If your bottleneck is network I/O, that should avoid some cross-cpu communications. Otoh, if you can't align client processing to the core its NIC queue is handled on, then my suggestion just adds contention on the processing queues; although maybe receive packet steering would help in that case. But, I'd also imagine game state processing is a bigger bottleneck than network I/O?
brobdingnagians 3y ago
I had this crazy idea to write an MMO server using Seastar with ScyllaDB and try to fit a huge isometric RPG shard in one server. It seems that modern servers could handle that load and have some interesting possibilities with a unified server. Kind of makes you wonder what the possibilities with text MUds would be if you dedicate more server time to player experience and game systems. Things people would have liked to do back in the day with open worlds might be more possible now.
mentos 3y ago
One thing I was hoping from cloud gaming was the ability to trust the client and offload all of the authentication from the server. Shame stadia died I feel like there was innovation possible in that direction that could have created a killer app for the service.
- candiddevmike 3y ago
  Has there ever been a case where trusting the client ends well?
  - meheleventyone 3y ago
    With cloud gaming there is no information leaked as the client is rendering video sent from a server and sending back inputs. You can still run an aimbot using computer vision techniques but the surface area for the client to be untrustworthy is much lower. I also think there is a compelling design space where cheating doesn’t really matter or make sense where trusting the client is fine.
Animats 3y ago
Interesting, but talking to the clients isn't the bottleneck for big-world systems. It's deciding who needs to know about what, and how often. If you tell everybody about everything that's happening, you have O(N^2) traffic. For large N, that dominates the O(N) load imposed by clients doing things. So there needs to be something that only tells some users what other users are doing, based on where they are and maybe which direction they are looking. Second Life has 50,552 users connected right now. Typical Saturday. They're all in the same world. But they're not near each other. The biggest crowds are about 130 users. There are a lot of filters. Each client is connected to the server for the region it is in, plus a few nearby regions, so it can see past region boundaries. Within a region, the server sends updates to each user as objects move. Updates are frequent, up to 45Hz, for nearby objects in the viewing frustrum. Lower for distant objects, and much lower outside the viewing frustrum. That avoids the O(N^2) load problem. The clients overload before the servers do, incidentally. The classic clients are single-thread and do too much CPU work per visible avatar.
Daegalus 3y ago
I have been working on a MMO resurrection project of an old MMO for a while now. It's nice to know my design is actually decent. I have no MMO or game backend dev experience. I am a backend and DevOps engineer though. I'm building the server in Go. I use goroutines a lot. I use a shrinking/growing ring buffer for requests and responses. Uses UDP and I just receive a request and pop it onto the queue. I keep the address of the client along with the packet request. The queue is read by another goroutine and gets processed and sent to a packet handler. Which does game state work, and queues a response, or passes it off to another handler. Then the go routine for the responses picks it up, reads the address and port of the client and sends it the packet. All I have maybe 5-6 threads running in parallel and I already can log into the game and walk around with others. There's tons of work still needed. But it's nice to know my underlying design is sufficient to maybe handle the dozens of people playing for nostalgia.
- cpeterso 3y ago
  Very cool! What game are you resurrecting? Are you using the original client applications? Is the client/server protocol documented or are you having to reverse engineer both the protocol and game server state?
  - Daegalus 3y ago
    It's Black Moon Chronicles Online: Winds of War. I have client sources as I am working with the person that owns the rights. Protocol is not documented, so reverse engineering, but having the clientside helps. The server software was lost to time and accidents. But we have dumps of quest data, times, mobs, etc and all assets in original form. So as soon as I get the server in a working state. We have the full game. We plan to open source/creative commons everything when we get something playable. I've done 4 massive redactors as the code grows, to accommodate changes in understanding and design flaws.
    - mrklol 3y ago
      What about this project here? https://github.com/jeanbmar/black-moon-rewind https://github.com/jeanbmar/black-moon-rewind Is this far behind?

185 points by cpeterso 3y ago

I made a MMO in 2001 and came to the conclusions that one thread per socket was not going to scale and thought I would mothball the project until that was solved in Java. 2023 and now we have virtual threads that replace blocking IO with non-blocking automatically! Haven't tested it yet though! But I am reviving the 2001 MMO now! In between I made my own MMO backend that uses NIO, conclusion is event-based network and validation is the only way to scale action MMOs past 100 players. Also you need to share memory atomically between cores, so Java or C.

nvarsj 3y ago
I feel like the Erlang/OTP would be perfect for the MMO use case. It’s what I’d try first if building from scratch.
- bullen 3y ago
  No erlang cannot share memory between cores, it has to copy it.
  - klibertp 3y ago
    I think binaries are refcounted and shared in Erlang? I'm not sure if this happens between schedulers/cores, though. They're also obviously immutable, so might not be what you mean by "sharing memory atomically". Other than Erlang, Pony[1] might be an interesting choice. It allows sharing memory, even mutable memory, but it tracks the sharing in the type system. It's been a long time since I looked at it, and it definitely had a bunch of rough edges, but I really liked where the things were going. I hope it got even better since. [1] https://www.ponylang.io/ https://www.ponylang.io/
    - bullen 3y ago
      For me C (1970) and Java (1990) are the only options for eternity. I use some C++ features and dabble in js when I make HTML but really since Applet and Flash has been removed the browser is just a bloated waste of time. Make your apps fast by allowing them to share memory without latency! I have also coded Perl, VisualBasic, php and C# and I'm pretty sure this is it. What we need now are VMs that can take any bytecode/instructionset and translate it to all others. For true crossplatform development. But that requires us to dump dynamic allocation. Arrays of 64 byte structures FTW!
      - ReactiveJelly 3y ago
        > VMs that can take any bytecode/instructionset and translate it to all others You're gonna love this: https://en.wikipedia.org/wiki/WebAssembly https://en.wikipedia.org/wiki/WebAssembly
        bullen 3y ago
        I would if anything worked. Still waiting for a Windows release of this: https://github.com/bytecodealliance/wasm-micro-runtime https://github.com/bytecodealliance/wasm-micro-runtime Don't hold you breath. Also needs Risc-V and ARM 32/64.
- Thaxll 3y ago
  Erlang performance would be a big problem. It's subpar compared to C++ by an order of magnitude.
  - cmdrk 3y ago
    depends on the kind of game you are making. People made MMOs in the year 2000 when CPUs themselves were an order of magnitude slower.
  - nvarsj 3y ago
    Would an MMO backend be CPU bound though? I feel the use case of having a large numbers of active connections with the very distributed nature of passing events around between them all would be perfect for Erlang/OTP. Anything CPU intensive (like dungeon generation or similar) could be offloaded to another service if needed.
    - bullen 3y ago
      You want low latency action, and for that you need one CPU with all cores talking to all the memory without locking. Erlang can't do that, and neither can any other language; except C (Arrays of 64 byte Structures) and Java (Concurrency complete rewrite of the JVM in 1.5 that was adopted and failed for C++11 because you need a VM + GC for that model to be meaningful).
      - nvarsj 3y ago
        Does it really need to be low latency? It's not an FPS - so I don't think high tick rate is important. I'd rather have a wider synchronized global state and acceptable server side latency (<500ms) than instant server updates. The client can interpolate everything.
        bullen 3y ago
        Then you are making a slideshow, not a game. The whole point of games is real-time action. It does not need to be violent, but it needs physics that give you the feeling of being there. Multiplayer > VR
- hmmokidk 3y ago
  I have an MMO demo in Elixir! I haven’t updated it in some time because busy from work but take a look: http://rotb.io http://rotb.io
- deleted 3y ago
  [deleted]
- bcrosby95 3y ago
  Erlang/OTP seems perfect for a diablo-esque game where you have lots of instances with a handful of players each. But a giant shared world with thousands of players doesn't seem to play into its wheelhouse as much.

firechickenbird 3y ago

I think your last chart is slightly incorrect: the game server is sending a message directly to the user, whereas it should theoretically only talk to the intermediate message queue (or frontend server)

bob1029 3y ago

I feel like some HFT/fintech concepts are ~ideal MMO architectures. With a 2 tier setup, you could theoretically have a central server processing batches of events that are aggregated across multiple players/regions/etc. A single thread can service upwards of half a billion events per second, and if each of those events covers multiple potential players, then I'd argue we are in a pretty good position. For me, having truly 1 consistent global universe is the only thing that would get me to consider an MMO in 2023+. The technical limitations forcing sharded worlds were excusable when WoW was released, but I don't have patience for those arguments anymore. Not if you want me to expressly burn my time and get paid for it on a recurring basis.

meheleventyone 3y ago
> I feel like some HFT/fintech concepts are ~ideal MMO architectures. Can you explain/link to some? I suspect MMO servers, particularly when server authoritative end up having to do a lot more processing per event and tick. Distributing world state updates is also an extremely thorny issue with lots of inter-dependencies.
- bob1029 3y ago
  See concepts like LMAX Disruptor and https://aeron.io https://aeron.io Whatever is good for low-latency trading systems (i.e. where you are paying contractually for microsecond-level guarantees), is also maybe good for gaming. The whole concept with these patterns is that there is only ever a single writer at any given moment. Single writer principle w/ batching is what gives you enormous uplift in throughput, even when you are necessarily constrained to fully serialized business semantics. I'd argue that an MMO does not need to be as strictly serialized as whatever CBOE, et. al. are doing.
  - meheleventyone 3y ago
    Interesting, thanks! Skimming the Disruptor paper its very similar to how you keep audio context fed using a ring buffer to write ahead as the audio context chews through behind you so the entire thing can remain lock-free and low latency. Typically in a separate thread as the latency and timing requirements are very strict. Aeron on the face of it looks very similar to the architecture diagrams you'd get popping up designing a networking layer in a game already. I suspect the cross-pollination in both directions could be interesting. Neither seem to particularly help with the intensive/interesting bit of an MMO which is running the actual simulation. For example big fleet fights in EVE as far as I remember have never been IO bound rather single-core performance bound. It's common for the bigger Corporations to contact CCP ahead of big fights so the node it's likely to happen on can be moved to a bigger machine. It's actually quite interesting because a lot of it is written in Stackless Python so not the most performant of languages and trapped in a single-threaded context so there is (nominally at least) quite a lot of headroom. For big seamless worlds how you distribute the simulation is the tough bit. Hence companies like Improbable trying to solve that generally.

jongjong 3y ago

I didn't realize that one-thread-per-socket was still something people considered; especially given that many programming languages nowadays have excellent support for async/await which does a great job at managing multiple concurrent connections within a single process/thread and doesn't have the overhead of context-switching.

deleted 3y ago
[deleted]
flohofwoe 3y ago
In an MMO backend architecture, the way client connections are handled is really just a tiny implemention detail at the edge of the architecture diagram, only affecting a very small part of the overall architecture. Ideally you'd move that stuff out into proxy servers sitting between the clients and the actual game servers with "one connection per client" on the public-network side and "one connection per game server" on the internal-network side. The proxy servers would also deal with validating the incoming traffic (on the message protocol level at least), encryption and of course routing messages between clients and game servers. PS: now that I actually RTFA, the same concept is already mentioned there, it's just called Frontend Server.
otabdeveloper4 3y ago
Context switching will happen regardless of where your thread implementation lives. Reimplementing scheduling in a bespoke fashion in usermode will not give you the gains that you hope for. Basically, learn how an OS works first before trying to reimplement it badly.
- flohofwoe 3y ago
  I guess the idea is that an async/await context switch would always be cheaper than an OS thread context switch. Whether that is actually true probably depends a lot on the specific operating system and async/await runtime (I think there's a lot of mysticism involved seeing async/await as some sort of silver bullet instead of relying on cold hard performance numbers alone).

VoidWhisperer 3y ago

Question with regards to the FIFO queue and processing it: Say a user does two actions in quick succession - action A that is something that inherently takes a bit longer to process on the game server (ie a complicated transaction of some kind) and then right after does action B, one that requires little to no processing. How is it best handled to make sure that the order of actions by the player is maintained with processing? With this scenario, it is possible action B will be processed before action A, leading to the game server considering them in the wrong order. I'm assuming the solution is to have some additional logic to make sure you are only processing one action per player at any given time, but I was wondering if anyone knows any better ways?

flohofwoe 3y ago
One way to solve this is to have per-player task queues in the game server which processes tasks sequentially (but concurrently with other player task queues, this is actually a perfect use case for async/await style code). Critical actions should also have a sequence counter to make sure they are enqueued in order and to detect whether actions got lost somewhere (especially when using an unordered networking protocol like UDP) - message ordering usually already happens on a lower level than the game code though in the code layer that sits on top of UDP sockets, but a higher level per-player action counter can't hurt either I guess, if only for debugging)
- gryn 3y ago
  Only if player tasks are independent. For example, If both are trying to mine a ressource(assuming a shared world) and you do it parallel you might end up with the wrong person winning the mining race condition.
  - flohofwoe 3y ago
    Yeah, synchronization between task queues is left as an excercise to the reader :D But I guess in that specific case it doesn't really matter, the server could just as well roll a dice to decide who wins the race (or rather: access to shared resources between task queues need to be handled like a regular data race in multithreading, what's important is that the server is the only source of truth). Latency-sensitive games also might want to speculate ahead by running the local player's task queue on the client, and roll back when the authorative state coming in from the server disagrees with the speculated client state.

monlockandkey 3y ago

I wonder what a mmo using Golang and goroutines look like?

flohofwoe 3y ago
Anectodal, but from what I've seen spending a couple of weeks at a Chinese mobile games company in 2017, Golang game servers are quite popular in China but not elsewhere (in combination with Unity for the game client and Protobuf for the message protocol). I have no idea why that specific tech combo is "big in China" but not outside. AFAIK in the rest of the world, Photon Server is the most popular off-the-shelf solution (not necessarily for true MMOs though). Maybe that's good enough for most games to not wanting to write your own server backend (https://www.photonengine.com/ https://www.photonengine.com/)
- Daegalus 3y ago
  Well I am building an MMO Server in Go, and I am in the US. Probably because I am a backend/DevOps engineer and i do a lot of Go as it is. I did a small rundown elsewhere in this thread: https://news.ycombinator.com/item?id=37972638 https://news.ycombinator.com/item?id=37972638
Daegalus 3y ago
I actually am building an MMO server for an old MMO. And I am using Go and Goroutines for it. I have a small rundown posted here in this thread: https://news.ycombinator.com/item?id=37972638 https://news.ycombinator.com/item?id=37972638

andai 3y ago

See also: Patrick Wyatt interviewed by Casey Muratori at HandmadeCon: https://youtu.be/1faaOrtHJ-A https://youtu.be/1faaOrtHJ-A He was the 2nd employee at Blizzard, did the netcode for their games and also for Guild Wars. His blog is great https://www.codeofhonor.com/blog/tough-times-on-the-road-to-starcraft https://www.codeofhonor.com/blog/tough-times-on-the-road-to-...

martinohansen 3y ago
Unrelated to the post here, but I think both Pat and this guy[1] say that they build the multiplayer for Diablo by themselves… Anyhow, both talks are fascinating and surely worth a watch. [1] https://youtu.be/Mlrrc_vy79E?si=Ga50gQyyUXjHwZz0 https://youtu.be/Mlrrc_vy79E?si=Ga50gQyyUXjHwZz0
- instagib 3y ago
  Thanks for the link but I think David Brevik said a couple guys from blizzard came up to help for 6 months and he had zero multiplayer code built as well as it was his first C program.
- vitus 3y ago
  Right around 37 minutes in, Brevik mentions that Mike O'Brien (along with Pat, one of the three co-founders of ArenaNet) was the brains behind Battle.net, and then "a few of the guys from [Blizzard South] moved up north during the last six months of development and started making Diablo into multiplayer and integrating Battle.net into the entire thing." That lines up with Pat's assessment: > Initially Collin Murray, a programmer on StarCraft, and I flew to Redwood City to help, while other developers at Blizzard “HQ” in Irvine California worked on network “providers” for battle.net, modem and LAN games as well as the user-interface screens (known as “glue screens” at Blizzard) that performed character creation, game joining, and other meta-game functions.
DrammBA 3y ago
That interview was exhausting, I couldn't finish it. I have no idea why the interviewer treated this like a systems design interview at a FAANG, with constant interruptions and unnecesarily rephrasing what the interviewee had already explained. The GDC talk linked in another reply, while only tangetially related, was much more enjoyable.

yobid20 3y ago

Amazed to see connection oriented protocols still being used here. As someone whose been in the real time media space for decades, we switched everything to UDP about 15 years ago and achieved much higher throughput and lower latency. Our server software architectures use a single dedicated thread on a UDP port, which reads packets and distributes them to lockless queues each owned by a dedicated single thread handler exclusively bound to separate physical cores. Similarly for the outbound side. This allows us to scale vertically to maximize a single machine's resources, while allowing us to essentially cap the upper bounds on latency in high load scenarios. Our architecture enables us to have hundreds of thousands of users PER MACHINE. We've been running this architecture for over 10 years now in production successfully. While we do have logic to scale horizontally, we minimize this and use it only when its required for scaling further or traversing specific network clouds, depending on where a user is able to connect. Worth mentioning we use general linux distros on physical hardware. No virtual machines. We also use kernal bypass techniques. Any time you add more servers to spread load, you're increasing latency because for each hop you're traversing the entire software network stack twice plus hops through hardware switches. Nobody uses dedicate thread per client anymore (if they do, its a poor design).

meheleventyone 3y ago
Games use UDP quite a lot as well.
toast0 3y ago
> Our server software architectures use a single dedicated thread on a UDP port, which reads packets and distributes them to lockless queues each owned by a dedicated single thread handler exclusively bound to separate physical cores. As an internet backseat network performance person... Have you considered one thread per core/NIC queue receiving packets, with RSS (receive side scaling)? If your bottleneck is network I/O, that should avoid some cross-cpu communications. Otoh, if you can't align client processing to the core its NIC queue is handled on, then my suggestion just adds contention on the processing queues; although maybe receive packet steering would help in that case. But, I'd also imagine game state processing is a bigger bottleneck than network I/O?

brobdingnagians 3y ago

I had this crazy idea to write an MMO server using Seastar with ScyllaDB and try to fit a huge isometric RPG shard in one server. It seems that modern servers could handle that load and have some interesting possibilities with a unified server. Kind of makes you wonder what the possibilities with text MUds would be if you dedicate more server time to player experience and game systems. Things people would have liked to do back in the day with open worlds might be more possible now.

mentos 3y ago

One thing I was hoping from cloud gaming was the ability to trust the client and offload all of the authentication from the server. Shame stadia died I feel like there was innovation possible in that direction that could have created a killer app for the service.

candiddevmike 3y ago
Has there ever been a case where trusting the client ends well?
- meheleventyone 3y ago
  With cloud gaming there is no information leaked as the client is rendering video sent from a server and sending back inputs. You can still run an aimbot using computer vision techniques but the surface area for the client to be untrustworthy is much lower. I also think there is a compelling design space where cheating doesn’t really matter or make sense where trusting the client is fine.

Animats 3y ago

Interesting, but talking to the clients isn't the bottleneck for big-world systems. It's deciding who needs to know about what, and how often. If you tell everybody about everything that's happening, you have O(N^2) traffic. For large N, that dominates the O(N) load imposed by clients doing things. So there needs to be something that only tells some users what other users are doing, based on where they are and maybe which direction they are looking. Second Life has 50,552 users connected right now. Typical Saturday. They're all in the same world. But they're not near each other. The biggest crowds are about 130 users. There are a lot of filters. Each client is connected to the server for the region it is in, plus a few nearby regions, so it can see past region boundaries. Within a region, the server sends updates to each user as objects move. Updates are frequent, up to 45Hz, for nearby objects in the viewing frustrum. Lower for distant objects, and much lower outside the viewing frustrum. That avoids the O(N^2) load problem. The clients overload before the servers do, incidentally. The classic clients are single-thread and do too much CPU work per visible avatar.

Daegalus 3y ago

I have been working on a MMO resurrection project of an old MMO for a while now. It's nice to know my design is actually decent. I have no MMO or game backend dev experience. I am a backend and DevOps engineer though. I'm building the server in Go. I use goroutines a lot. I use a shrinking/growing ring buffer for requests and responses. Uses UDP and I just receive a request and pop it onto the queue. I keep the address of the client along with the packet request. The queue is read by another goroutine and gets processed and sent to a packet handler. Which does game state work, and queues a response, or passes it off to another handler. Then the go routine for the responses picks it up, reads the address and port of the client and sends it the packet. All I have maybe 5-6 threads running in parallel and I already can log into the game and walk around with others. There's tons of work still needed. But it's nice to know my underlying design is sufficient to maybe handle the dozens of people playing for nostalgia.

cpeterso 3y ago
Very cool! What game are you resurrecting? Are you using the original client applications? Is the client/server protocol documented or are you having to reverse engineer both the protocol and game server state?
- Daegalus 3y ago
  It's Black Moon Chronicles Online: Winds of War. I have client sources as I am working with the person that owns the rights. Protocol is not documented, so reverse engineering, but having the clientside helps. The server software was lost to time and accidents. But we have dumps of quest data, times, mobs, etc and all assets in original form. So as soon as I get the server in a working state. We have the full game. We plan to open source/creative commons everything when we get something playable. I've done 4 massive redactors as the code grows, to accommodate changes in understanding and design flaws.
  - mrklol 3y ago
    What about this project here? https://github.com/jeanbmar/black-moon-rewind https://github.com/jeanbmar/black-moon-rewind Is this far behind?