9 ms·

Hobbyist game dev here with random systemd thoughts. I’ve recently started to lean on systemd more as my ‘local game server process manager’ process. At first I

Hobbyist game dev here with random systemd thoughts. I’ve recently started to lean on systemd more as my ‘local game server process manager’ process. At first I thought I’d have to write this up myself as a whole slew of custom code, but then I realized the linux distros I use have systemd. That + cgroups and profiling my game server’s performance lets me pack an OS with as many game servers dynamically (target 80% resource utilization, funny things happen after that — things I don’t quite understand). In this way I’m able to set up AWS EC2 instances or digital ocean droplets, a bunch of game servers spin up and report back their existence to a backend game services API. So far it’s working but this part of my project is still in development. I used to target containerizing my apps, which adds complexity, but often in AWS I have to care about VMs as resources anyways (e.g. AWS gamelift requires me to spin up VMs, same with AWS EKS). I’m still going back and forth between containerizing and using systemd; having a local stack easily spun up via docker compose is nice, but with systemd what I write locally is basically what runs in prod environment, and there’s less waiting for container builds and such. I share all of this in case there’s a gray beard wizard out there who can offer opinions. I have a tendency to explore and research (it’s fuuun!) so I’m not sure if I’m on a “this is cool and a great idea” path or on a “nobody does this because <reasons>” path.

baggy_trough 10mo ago
Did you try systemd's containers (nspawn)?
- sovietmudkipz 10mo ago
  …no. TIL.
  - panick21_ 10mo ago
    Portable services are another option.
    - open-paren 10mo ago
      And podman systemd quadlets yet another https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
      - sovietmudkipz 10mo ago
        Wow systemd can do more than I thought to imagine it could
        bonzini 10mo ago
        Technically that's part of podman, not systemd. But it's the same architecture that was used to support sysvinit scripts. (In fact, nothing prevents anyone from extracting and repackaging the sysvinit generator, now that I think of it).
  - nszceta 10mo ago
    I wrote a blog post about using nspawn from an Arch Linux host. The Arch Wiki shows more information about how to get a Debian base if you want that instead. Link to the wiki is at the bottom of the blog post along with more references. https://adamgradzki.com/lightweight-development-sandboxes-with-systemd-nspawn-on-linux.html https://adamgradzki.com/lightweight-development-sandboxes-wi...
dijit 10mo ago
This is sort of how I designed Accelbytes managed gameserver system (previously called: Armada). You provide us a docker image, and we unpack it, turn it into a VM image and run as many instances as you want side-by-side with CPU affinity and NUMA awareness. Obviating the docker network stack for latency/throughput reasons - since you can They had tried nomad, agones and raw k8s before that.
- sovietmudkipz 10mo ago
  Checking out the website now. Looks enticing. Would a user of accelbyte multiplayer services still be in the business of knowing about underlying VMs? I caught some copy on the website that led me to question. As a hobbyist part of me wants the VM abstracted completely (which may not be realistic). I want to say “here’s my game server process, it needs this much cpu/mem/network per unit, and I need 100 processes” and not really care about the underlying VM(s), at least until later. The closest thing I’ve found to this is AWS fargate. Also holy smokes if you were a part of the team that architected this solution I’d love to pick your brain.
  - dijit 10mo ago
    That was was actually the original intent. If we scale to bare metal providers we can get much more performance. m By making it an “us” problem to run the infrastructure at a good cost, and be cheaper then than AWS for us to run, meaning we could take no profit on cloud vms. making us cost competitive as hell.
    - sovietmudkipz 10mo ago
      If I understand correctly you're saying you manage hardware yourself (colocate in a data center? Dedicated hosting?) and that gives you an edge in pricing. That's pretty cool, and I think I can see how it could be less expensive to purchase hardware & maintain it rather than renting that compute from a third party. There is obviously the tradeoff of then being responsible for capacity planning for the workloads supported among other downsides and maintaining hardware lifecycle but I wouldn't be surprised to hear this downside is overstated compared to benefits reaped.
esseph 10mo ago
If you use podman quadlets, you get containers and systemd together as a first class citizen, in a config that is easily portable to kubernetes if you need more complex features.
- sovietmudkipz 10mo ago
  O.O this may be the feature that gets me into podman over docker.
  - esseph 10mo ago
    The shift from docker to podman was originally quite painful at first, but it's much better, very usable, and quite stable now. Still, I can see the draw for independent devs to use docker compose. Teams and orgs though makes sense to use podman and systemd for the smaller stuff or dev, and then literally export the config as a kubernetes yaml.
    - ziml77 10mo ago
      How is podman managed in larger environments? It's designed around running rootless, but it seemed like the nature of that is there wasn't a proper way to centrally manage the containers even on just a single machine. Like even seeing the logs required using machinectl to get a shell as the user who owns the service (sudo/su do not retain the necessary environment variables for journalctl to work). Trying to get the logs as root seems to let you filter down to the user (rather than service) level at best. Meanwhile, with Docker (or the not recommended rootful podman), you can have centralized management of multiple machines with a tool like Portainer. I like the idea of podman, but this has been a head-scratcher for me.
      - esseph 10mo ago
        The way you manage podman in large environments is called Kubernetes :) podman is really only suitable for a single node, but they may have added things I have missed.
  - 10mo ago
rbjorklin 10mo ago
You sound like you've explored at least a few options in this space. Have you looked at https://agones.dev/ https://agones.dev/ ?
- sovietmudkipz 10mo ago
  Yes! It’s a great project. I’m super happy they have a coherent local development story. I kinda abandoned using it though when I said “keeeep it simple” and stopped using containers/k8s. I think I needed to journey through understanding why multiplayer game services like Agones/gamelift/photon were set up like they were. I read through Multiplayer Game Programming: Architecting Networked Games by Joshua Glazer and Sanjay Madhav really helped (not to mention allowed me to better understand GDC talks over multiplayer topics much better). This all probably speaks to my odd prioritization: I want to understand and use. I’ve had to step back and realize part of the fun I have in pursuing these projects is the research.
  - dontlaugh 10mo ago
    I’ve also found docker / k8s to mostly just get in the way. Even VMs are often a problem, depending on the details of the game. Bare metal is the only actually good option, but you often have to do a lot yourself. Multiplay did offer it last time I looked, but I don’t know what’s going on with them now.
madjam002 10mo ago
Definitely don't recommend going down this path if you're not already familiar with Nix, but if you are, a strategy that I find works really well is to package your software with Nix, then you can run it easily via systemd but also create super lightweight containers using nix-snapshotter[0] so you don't have to "build" container images if you still want the flexibility of containers. You can then run the containers on Docker or Kubernetes without having to build heavy images. [0] https://github.com/pdtpartners/nix-snapshotter https://github.com/pdtpartners/nix-snapshotter
- throwaway091025 10mo ago
  [dead]
- frantathefranta 10mo ago
  I don't recommend getting familiar with Nix because your chances of getting nerd sniped by random HN comments increase exponentially.
  - sovietmudkipz 10mo ago
    Funny. I probably will dive into Nix some day but I've been content letting it sit waiting for me to check it out.
colechristensen 10mo ago
> (target 80% resource utilization, funny things happen after that — things I don’t quite understand). The closer you get to 100% resource utilization the more regular your workload has to become. If you can queue requests and latency isn't a problem, no problem, but then you have a batch process and not a live one (obviously not for games). The reason is because live work doesn't come in regular beats, it comes in clusters that scale in a fractal way. If your long term mean is one request per second what actually happens is you get five requests in one second, three seconds with one request each, one second with two requests, and five seconds with 0 requests (you get my point). "fractal burstiness" You have to have free resources to handle the spikes at all scales. Also very many systems suffer from the processing time for a single request increasing as overall system loads increase. "queuing latency blowup" So what happens? You get a spike, get behind, and never ever catch up. https://en.wikipedia.org/wiki/Network_congestion#Congestive_collapse https://en.wikipedia.org/wiki/Network_congestion#Congestive_...
- sovietmudkipz 10mo ago
  Yea. I realize I ought to dig into things more to understand how to push past into 90%-95% utilization territory. Thanks for the resource to read through.
  - colechristensen 10mo ago
    One way to think about it is 80% IS full utilization. The engineering time, the risks of decreased performance, and the fragility of pushing the limit at some point become not worth the benefits of reaching some higher metric of utilization. If it's not where you are, that optimum trade off point is somewhere.
  - mpyne 10mo ago
    You absolutely do not want 90-95% utilization. At that level of utilitization random variability alone is enough to cause massive whiplash in average queue lengths. The cycle time impact of variability of a single-server/single-queue system at 95% load is nearly 25x the impact on the same system at 75% load, and there are similar measures for other process queues. As the other comment notes, you should really work from an assumption that 80% is max loading, just as you'd never aim to have a swap file or swap partition of exactly the amount of memory overcommit you expect.
reactordev 10mo ago
This actually works really well with custom user scripts to do the initial setup. It’s also trivial to do this with docker/podman if you don’t want it to take over the machine. Batching/Matchmaking is the hard part of this, setting up a fleet is the fun part of this. I’ve also done Microsoft Orleans clusters and still recommend the single pid, multiple containers/processes approach. If you can avoid Orleans and kubernetes and all that, the better. It just adds complexity to this setup.
- sovietmudkipz 10mo ago
  > If you can avoid Orleans and kubernetes and all that, the better. It just adds complexity to this setup. I’m starting to appreciate simplicity away from containers that’s why I’m even exploring systemd. I bet big on containers and developed plenty of skills, especially with k8s. I never stopped to appreciate that I’m partly in the business of making processes run on OSes, and it kinda doesn’t matter if the pid is a container or running ‘directly’ on the hardware. I’ll probably layer it back in but for now I’m kinda avoiding it as an exercise. E.g. if I’m testing a debug ready build locally and want to attach my debugger, I can do that in k8s but there’s a ceremony of opening relevant ports and properly pointing to the file system of the container. Not a show stopper since I mostly debug while writing testing/production code in dev… But occasionally the built artifact demands inspection.
miladyincontrol 10mo ago
> I’m still going back and forth between containerizing and using systemd Why not both? Systemd allows you to make containers via nspawn, which are defined just about the exact same as you do a regular systemd service. Best of both worlds.
- Levitating 10mo ago
  > Best of both worlds. That would be portable[1] services. [1]: https://systemd.io/PORTABLE_SERVICES/ https://systemd.io/PORTABLE_SERVICES/

baggy_trough 10mo ago

Did you try systemd's containers (nspawn)?

sovietmudkipz 10mo ago
…no. TIL.
- panick21_ 10mo ago
  Portable services are another option.
  - open-paren 10mo ago
    And podman systemd quadlets yet another https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
    - sovietmudkipz 10mo ago
      Wow systemd can do more than I thought to imagine it could
      - bonzini 10mo ago
        Technically that's part of podman, not systemd. But it's the same architecture that was used to support sysvinit scripts. (In fact, nothing prevents anyone from extracting and repackaging the sysvinit generator, now that I think of it).
- nszceta 10mo ago
  I wrote a blog post about using nspawn from an Arch Linux host. The Arch Wiki shows more information about how to get a Debian base if you want that instead. Link to the wiki is at the bottom of the blog post along with more references. https://adamgradzki.com/lightweight-development-sandboxes-with-systemd-nspawn-on-linux.html https://adamgradzki.com/lightweight-development-sandboxes-wi...

dijit 10mo ago

This is sort of how I designed Accelbytes managed gameserver system (previously called: Armada). You provide us a docker image, and we unpack it, turn it into a VM image and run as many instances as you want side-by-side with CPU affinity and NUMA awareness. Obviating the docker network stack for latency/throughput reasons - since you can They had tried nomad, agones and raw k8s before that.

sovietmudkipz 10mo ago
Checking out the website now. Looks enticing. Would a user of accelbyte multiplayer services still be in the business of knowing about underlying VMs? I caught some copy on the website that led me to question. As a hobbyist part of me wants the VM abstracted completely (which may not be realistic). I want to say “here’s my game server process, it needs this much cpu/mem/network per unit, and I need 100 processes” and not really care about the underlying VM(s), at least until later. The closest thing I’ve found to this is AWS fargate. Also holy smokes if you were a part of the team that architected this solution I’d love to pick your brain.
- dijit 10mo ago
  That was was actually the original intent. If we scale to bare metal providers we can get much more performance. m By making it an “us” problem to run the infrastructure at a good cost, and be cheaper then than AWS for us to run, meaning we could take no profit on cloud vms. making us cost competitive as hell.
  - sovietmudkipz 10mo ago
    If I understand correctly you're saying you manage hardware yourself (colocate in a data center? Dedicated hosting?) and that gives you an edge in pricing. That's pretty cool, and I think I can see how it could be less expensive to purchase hardware & maintain it rather than renting that compute from a third party. There is obviously the tradeoff of then being responsible for capacity planning for the workloads supported among other downsides and maintaining hardware lifecycle but I wouldn't be surprised to hear this downside is overstated compared to benefits reaped.

esseph 10mo ago

If you use podman quadlets, you get containers and systemd together as a first class citizen, in a config that is easily portable to kubernetes if you need more complex features.

sovietmudkipz 10mo ago
O.O this may be the feature that gets me into podman over docker.
- esseph 10mo ago
  The shift from docker to podman was originally quite painful at first, but it's much better, very usable, and quite stable now. Still, I can see the draw for independent devs to use docker compose. Teams and orgs though makes sense to use podman and systemd for the smaller stuff or dev, and then literally export the config as a kubernetes yaml.
  - ziml77 10mo ago
    How is podman managed in larger environments? It's designed around running rootless, but it seemed like the nature of that is there wasn't a proper way to centrally manage the containers even on just a single machine. Like even seeing the logs required using machinectl to get a shell as the user who owns the service (sudo/su do not retain the necessary environment variables for journalctl to work). Trying to get the logs as root seems to let you filter down to the user (rather than service) level at best. Meanwhile, with Docker (or the not recommended rootful podman), you can have centralized management of multiple machines with a tool like Portainer. I like the idea of podman, but this has been a head-scratcher for me.
    - esseph 10mo ago
      The way you manage podman in large environments is called Kubernetes :) podman is really only suitable for a single node, but they may have added things I have missed.
- 10mo ago

rbjorklin 10mo ago

You sound like you've explored at least a few options in this space. Have you looked at https://agones.dev/ https://agones.dev/ ?

sovietmudkipz 10mo ago
Yes! It’s a great project. I’m super happy they have a coherent local development story. I kinda abandoned using it though when I said “keeeep it simple” and stopped using containers/k8s. I think I needed to journey through understanding why multiplayer game services like Agones/gamelift/photon were set up like they were. I read through Multiplayer Game Programming: Architecting Networked Games by Joshua Glazer and Sanjay Madhav really helped (not to mention allowed me to better understand GDC talks over multiplayer topics much better). This all probably speaks to my odd prioritization: I want to understand and use. I’ve had to step back and realize part of the fun I have in pursuing these projects is the research.
- dontlaugh 10mo ago
  I’ve also found docker / k8s to mostly just get in the way. Even VMs are often a problem, depending on the details of the game. Bare metal is the only actually good option, but you often have to do a lot yourself. Multiplay did offer it last time I looked, but I don’t know what’s going on with them now.

madjam002 10mo ago

Definitely don't recommend going down this path if you're not already familiar with Nix, but if you are, a strategy that I find works really well is to package your software with Nix, then you can run it easily via systemd but also create super lightweight containers using nix-snapshotter[0] so you don't have to "build" container images if you still want the flexibility of containers. You can then run the containers on Docker or Kubernetes without having to build heavy images. [0] https://github.com/pdtpartners/nix-snapshotter https://github.com/pdtpartners/nix-snapshotter

throwaway091025 10mo ago
[dead]
frantathefranta 10mo ago
I don't recommend getting familiar with Nix because your chances of getting nerd sniped by random HN comments increase exponentially.
- sovietmudkipz 10mo ago
  Funny. I probably will dive into Nix some day but I've been content letting it sit waiting for me to check it out.

colechristensen 10mo ago

> (target 80% resource utilization, funny things happen after that — things I don’t quite understand). The closer you get to 100% resource utilization the more regular your workload has to become. If you can queue requests and latency isn't a problem, no problem, but then you have a batch process and not a live one (obviously not for games). The reason is because live work doesn't come in regular beats, it comes in clusters that scale in a fractal way. If your long term mean is one request per second what actually happens is you get five requests in one second, three seconds with one request each, one second with two requests, and five seconds with 0 requests (you get my point). "fractal burstiness" You have to have free resources to handle the spikes at all scales. Also very many systems suffer from the processing time for a single request increasing as overall system loads increase. "queuing latency blowup" So what happens? You get a spike, get behind, and never ever catch up. https://en.wikipedia.org/wiki/Network_congestion#Congestive_collapse https://en.wikipedia.org/wiki/Network_congestion#Congestive_...

sovietmudkipz 10mo ago
Yea. I realize I ought to dig into things more to understand how to push past into 90%-95% utilization territory. Thanks for the resource to read through.
- colechristensen 10mo ago
  One way to think about it is 80% IS full utilization. The engineering time, the risks of decreased performance, and the fragility of pushing the limit at some point become not worth the benefits of reaching some higher metric of utilization. If it's not where you are, that optimum trade off point is somewhere.
- mpyne 10mo ago
  You absolutely do not want 90-95% utilization. At that level of utilitization random variability alone is enough to cause massive whiplash in average queue lengths. The cycle time impact of variability of a single-server/single-queue system at 95% load is nearly 25x the impact on the same system at 75% load, and there are similar measures for other process queues. As the other comment notes, you should really work from an assumption that 80% is max loading, just as you'd never aim to have a swap file or swap partition of exactly the amount of memory overcommit you expect.

reactordev 10mo ago

This actually works really well with custom user scripts to do the initial setup. It’s also trivial to do this with docker/podman if you don’t want it to take over the machine. Batching/Matchmaking is the hard part of this, setting up a fleet is the fun part of this. I’ve also done Microsoft Orleans clusters and still recommend the single pid, multiple containers/processes approach. If you can avoid Orleans and kubernetes and all that, the better. It just adds complexity to this setup.

sovietmudkipz 10mo ago
> If you can avoid Orleans and kubernetes and all that, the better. It just adds complexity to this setup. I’m starting to appreciate simplicity away from containers that’s why I’m even exploring systemd. I bet big on containers and developed plenty of skills, especially with k8s. I never stopped to appreciate that I’m partly in the business of making processes run on OSes, and it kinda doesn’t matter if the pid is a container or running ‘directly’ on the hardware. I’ll probably layer it back in but for now I’m kinda avoiding it as an exercise. E.g. if I’m testing a debug ready build locally and want to attach my debugger, I can do that in k8s but there’s a ceremony of opening relevant ports and properly pointing to the file system of the container. Not a show stopper since I mostly debug while writing testing/production code in dev… But occasionally the built artifact demands inspection.

miladyincontrol 10mo ago

> I’m still going back and forth between containerizing and using systemd Why not both? Systemd allows you to make containers via nspawn, which are defined just about the exact same as you do a regular systemd service. Best of both worlds.

Levitating 10mo ago
> Best of both worlds. That would be portable[1] services. [1]: https://systemd.io/PORTABLE_SERVICES/ https://systemd.io/PORTABLE_SERVICES/