6 ms·

Oh hey, you don't see a lot of Docker Swarm nowadays, though in my experience it's still a wonderful solution for getting started with container orchestration,

Oh hey, you don't see a lot of Docker Swarm nowadays, though in my experience it's still a wonderful solution for getting started with container orchestration, that will take a lot of the smaller/medium scale projects pretty far, before you need to look at something else (e.g. Nomad or Kubernetes). There's a lot of benefit in being able to hit the ground running even when you're self-hosting your clusters and administering them yourself. It comes available with an install of Docker, is easy to setup and operate, has great optional UI solutions like Portainer (analogue to Rancher for Kubernetes), has one of the lower resource usages for the orchestrator itself, as well as supports the Docker Compose specification, which in my opinion is far more usable than the Kubernetes manifests (though less powerful than Helm charts) and far more common than Nomad's HCL. For my Master's Degree, I explored a comparison where I ran the same workloads across a Docker Swarm cluster and a K3s cluster (a great Kubernetes distro that's low on resource usage as well) and even then Swarm used less memory (~2x less than Kubernetes for the leader node both under load and when idle) and used a bit less CPU (~30% less for the leader nodes under load) as well. That said, K3s still performed admirably, at least in comparison to RKE which wouldn't even run in a stable fashion on the limited hardware that I had at the time. Maybe one of these days I should run Proxmox in my homelab as well, instead of just something like Debian or Ubuntu directly on the hardware. Also, while Podman is great, Docker still seems like a dependable option just because of how common it is and given how it's gotten more stable over time (despite the arguable architecture disadvantages). I think the only actual issues I've had since when using Docker Swarm have been using a network that ran out of addresses to assign to the containers (probably some default), some Oracle Linux bug where kswapd would top out the CPU when the swap got full, as well as some Debian bug years ago on an old version of Docker that caused networking to fail and the cluster needed to be re-created to fix it.

ndsipa_pomu 4y ago
We've been using docker swarm for running some basic services as kubernetes seems way over-engineered for simple stuff. Our basic requirements is that a service will be auto-restarted as soon as possible if/when something breaks. However, though swarm is easy to use and configure, I've found that having a small swarm (e.g. three nodes) means that you end up having each node be a manager as the swarm fails if half the manager nodes aren't up. I've also found that "docker compose config" doesn't work in a compatible manner and have to use "docker-compose config" instead.
CoolCold 4y ago
> comparison where I ran the same workloads across a Docker Swarm cluster and a K3s cluster (a great Kubernetes distro that's low on resource usage as well) and even then Swarm used less memory (~2x less than Kubernetes for the leader node both under load and when idle) and used a bit less CPU (~30% less for the leader nodes under load) as well. have you found why it's so? I'm curious - in theory all the processes inside are the same, except, what it should be slight overhead, from Swarm/K3S itself.
- KronisLV 4y ago
  > I'm curious - in theory all the processes inside are the same, except, what it should be slight overhead, from Swarm/K3S itself. What I saw during benchmarking was that the load on the worker nodes was almost the same for both K3s and Swarm, presumably because most of the resources were used by the actual container runtimes and containers themselves, the overhead for communication with the orchestrator not being too notable. However, when I tested the whole setup under load (containers serving lots of web requests, databases inside of containers etc., some containers eventually getting OOM killed and restarted), I found that the leader nodes had more pronounced resource usage differences, as described above. My guess is that Kubernetes (even K3s) is just a bigger and more complex system than Swarm, which probably also explains the comparatively higher requirements/overhead (for even single node Kubernetes as well). That's also why I'd be careful about running anything apart from the orchestrator on the leader nodes.

ndsipa_pomu 4y ago

We've been using docker swarm for running some basic services as kubernetes seems way over-engineered for simple stuff. Our basic requirements is that a service will be auto-restarted as soon as possible if/when something breaks. However, though swarm is easy to use and configure, I've found that having a small swarm (e.g. three nodes) means that you end up having each node be a manager as the swarm fails if half the manager nodes aren't up. I've also found that "docker compose config" doesn't work in a compatible manner and have to use "docker-compose config" instead.

CoolCold 4y ago

> comparison where I ran the same workloads across a Docker Swarm cluster and a K3s cluster (a great Kubernetes distro that's low on resource usage as well) and even then Swarm used less memory (~2x less than Kubernetes for the leader node both under load and when idle) and used a bit less CPU (~30% less for the leader nodes under load) as well. have you found why it's so? I'm curious - in theory all the processes inside are the same, except, what it should be slight overhead, from Swarm/K3S itself.

KronisLV 4y ago
> I'm curious - in theory all the processes inside are the same, except, what it should be slight overhead, from Swarm/K3S itself. What I saw during benchmarking was that the load on the worker nodes was almost the same for both K3s and Swarm, presumably because most of the resources were used by the actual container runtimes and containers themselves, the overhead for communication with the orchestrator not being too notable. However, when I tested the whole setup under load (containers serving lots of web requests, databases inside of containers etc., some containers eventually getting OOM killed and restarted), I found that the leader nodes had more pronounced resource usage differences, as described above. My guess is that Kubernetes (even K3s) is just a bigger and more complex system than Swarm, which probably also explains the comparatively higher requirements/overhead (for even single node Kubernetes as well). That's also why I'd be careful about running anything apart from the orchestrator on the leader nodes.