4 ms·

Isn’t that limited to a single node? How would you configure a cluster? I’m trying to explore lightweight alternatives to kubernetes, such as docker swarm, but

by iTokio 1y ago

Isn’t that limited to a single node? How would you configure a cluster? I’m trying to explore lightweight alternatives to kubernetes, such as docker swarm, but I think that the options are limited if you must support clusters with equivalent of pods and services at least.

gf000 1y ago
In the off chance your search didn't expand to k3s, I can semi-recommend it. My setup is a bit clunky (having a Hetzner cloud instance as controller and a local server as a node throught Tailscale), from which I get an occasional strange error that k3s pods fail to resolve another pod's domain without me having to re-create the DNS resolver system pod, and that I so far failed at getting Velero backups to work with k3s's local storage providers, but otherwise it is pretty decent.
- iTokio 1y ago
  K3s is light in terms of resources, but heavy in operational complexity, I’m not looking for a smaller version of kubernetes but for a simple way to run container backed services when you’re not google but a small company, something that has few moving parts but is very reliable and low maintenance.
  - esseph 1y ago
    I've been back and forth on this for a long time, but I've just decided at this point that I either settle for podman or docker on a single host, or go to Talos / k3s / k8s. There's a lot of tools there, a lot of inertia, and eventually it's likely that I will need to solve the problems that k8s does. YMMV
  - tracker1 1y ago
    It that's what you want to do, just get something running... what's wrong with docker swarm?
    - osn9363739 1y ago
      I recall seeing a couple of blog posts lately about docker swarm and how its better now. I can see a few references to it in the latest release notes so I guess it's still getting some love.
zelphirkalt 1y ago
I once tried Nomad for a very brief moment. Not sure if it fits your bill.
- 0x457 1y ago
  Nomad is weird. Its OSS version is like a very limited trial of paid version. At least last time I tried it. To a point that it was more productive for me to install k3s instead.
jabl 1y ago
> Isn’t that limited to a single node? Yes. Though unless you have a very dynamic environment maybe statically assigning containers to hosts isn't an insurmountable burden?
sc68cal 1y ago
> How would you configure a cluster? So, unless you have a service that requires a fixed number of running instances that is not the same count as the number of servers, I would argue that maybe you don't need Kubernetes. For example, I built up a Django web application and a set of Celery workers, and just have the same pod running on 8 servers, and I just use an Ansible playbook that creates the podman pod and runs the containers in the pod.
MrDrMcCoy 1y ago
HashiCorp Nomad is probably the only real alternative. It's what in using, and I like it better than the overcomplexity of k8s.
mands 1y ago
I've found you can get pretty far with a couple of fixed nodes and scaling vertically before bringing in k8s these days. Right now I'm running, - podman, with quadlet to orchestrate both single containers and `pods` using their k8s-compatible yaml definition - systemd for other services - you can control and harden services via systemd pretty well (see https://news.ycombinator.com/item?id=44937550 https://news.ycombinator.com/item?id=44937550 from the other day). I prefer using systemd directly for Java services over containers, seems to work better imo - Pyinfra (https://pyinfra.com/ https://pyinfra.com/) to manage and provision the VMs and services - Fedora CoreOS as an immutable base OS with regular automatic updates All seems to be working really well.
pianopatrick 1y ago
I've been reading and watching videos about how you can use Ansible with Podman as a simpler alternative to Kubernetes. Basically Ansible just SSHs into each server and uses podman to start up the various pods / containers etc. that you specify. I have not tried this yet though so take this idea with a grain of salt.
- mdaniel 1y ago
  whew, "alternative" is doing a lot of work there. Contrast: ansible -i server1,server2,server3 deploy_fake_pods.yaml ssh server1 sudo shutdown -h now # aww, too bad, now your pods on server1 are no longer With kubectl apply -f deployment.yaml for i in $(kubectl get nodes -o jsonpath='{.status.hostIP}'); do ssh $i sudo shutdown -h now sleep 120 done # nothing has changed except you have fresh Nodes If you don't happen to have a cluster autoscaler available, feel free to replace the for loop with |head -1 or a break, but I mean to point out that the overall health and availability of the system is managed by kubernetes, but ansible is not that
osigurdson 1y ago
>> lightweight alternatives to kubernetes microk8s seems exceedingly simple to setup and use. k3s is easy as well.

by iTokio 1y ago

gf000 1y ago

In the off chance your search didn't expand to k3s, I can semi-recommend it. My setup is a bit clunky (having a Hetzner cloud instance as controller and a local server as a node throught Tailscale), from which I get an occasional strange error that k3s pods fail to resolve another pod's domain without me having to re-create the DNS resolver system pod, and that I so far failed at getting Velero backups to work with k3s's local storage providers, but otherwise it is pretty decent.

iTokio 1y ago
K3s is light in terms of resources, but heavy in operational complexity, I’m not looking for a smaller version of kubernetes but for a simple way to run container backed services when you’re not google but a small company, something that has few moving parts but is very reliable and low maintenance.
- esseph 1y ago
  I've been back and forth on this for a long time, but I've just decided at this point that I either settle for podman or docker on a single host, or go to Talos / k3s / k8s. There's a lot of tools there, a lot of inertia, and eventually it's likely that I will need to solve the problems that k8s does. YMMV
- tracker1 1y ago
  It that's what you want to do, just get something running... what's wrong with docker swarm?
  - osn9363739 1y ago
    I recall seeing a couple of blog posts lately about docker swarm and how its better now. I can see a few references to it in the latest release notes so I guess it's still getting some love.

zelphirkalt 1y ago

I once tried Nomad for a very brief moment. Not sure if it fits your bill.

0x457 1y ago
Nomad is weird. Its OSS version is like a very limited trial of paid version. At least last time I tried it. To a point that it was more productive for me to install k3s instead.

jabl 1y ago

> Isn’t that limited to a single node? Yes. Though unless you have a very dynamic environment maybe statically assigning containers to hosts isn't an insurmountable burden?

sc68cal 1y ago

> How would you configure a cluster? So, unless you have a service that requires a fixed number of running instances that is not the same count as the number of servers, I would argue that maybe you don't need Kubernetes. For example, I built up a Django web application and a set of Celery workers, and just have the same pod running on 8 servers, and I just use an Ansible playbook that creates the podman pod and runs the containers in the pod.

MrDrMcCoy 1y ago

HashiCorp Nomad is probably the only real alternative. It's what in using, and I like it better than the overcomplexity of k8s.

mands 1y ago

I've found you can get pretty far with a couple of fixed nodes and scaling vertically before bringing in k8s these days. Right now I'm running, - podman, with quadlet to orchestrate both single containers and `pods` using their k8s-compatible yaml definition - systemd for other services - you can control and harden services via systemd pretty well (see https://news.ycombinator.com/item?id=44937550 https://news.ycombinator.com/item?id=44937550 from the other day). I prefer using systemd directly for Java services over containers, seems to work better imo - Pyinfra (https://pyinfra.com/ https://pyinfra.com/) to manage and provision the VMs and services - Fedora CoreOS as an immutable base OS with regular automatic updates All seems to be working really well.

pianopatrick 1y ago

I've been reading and watching videos about how you can use Ansible with Podman as a simpler alternative to Kubernetes. Basically Ansible just SSHs into each server and uses podman to start up the various pods / containers etc. that you specify. I have not tried this yet though so take this idea with a grain of salt.

mdaniel 1y ago
whew, "alternative" is doing a lot of work there. Contrast: ansible -i server1,server2,server3 deploy_fake_pods.yaml ssh server1 sudo shutdown -h now # aww, too bad, now your pods on server1 are no longer With kubectl apply -f deployment.yaml for i in $(kubectl get nodes -o jsonpath='{.status.hostIP}'); do ssh $i sudo shutdown -h now sleep 120 done # nothing has changed except you have fresh Nodes If you don't happen to have a cluster autoscaler available, feel free to replace the for loop with |head -1 or a break, but I mean to point out that the overall health and availability of the system is managed by kubernetes, but ansible is not that

osigurdson 1y ago

>> lightweight alternatives to kubernetes microk8s seems exceedingly simple to setup and use. k3s is easy as well.