13 ms·

Replacing Kubernetes with systemd (2024)

436 points by birdculture 2y ago

klooney 2y ago
https://github.com/coreos/fleet https://github.com/coreos/fleet I feel like fleet deserved more of a shot than it ultimately got.
- deleted 2y ago
  [deleted]
- pa7ch 2y ago
  agreed, coreos pivoted to k8s almost immediately after releaseing fleet and didn't really get the chance to use and develop it much.
  - llarsson 2y ago
    That's also where etcd came from. It really felt, to me, like the precursor to Kubernetes.
- thebeardisred 2y ago
  Well, it's funny you mention that because I started working on a PoC of running vLLM atop Fleet this morning. :grin:
  - klooney 2y ago
    I'm glad to hear it still works!
byrnedo 2y ago
I created skate (https://github.com/skateco/skate https://github.com/skateco/skate) to be basically this but multihost and support k8s manifests. Under the hood it’s podman and systemd
- woile 2y ago
  This looks awesome!
- nemofoo 2y ago
  Thank you for building this. I appreciate you.
- psviderski 2y ago
  This is a great approach which resonates with me a lot. It's really frustrating that there is no simple way to run a multi-host Docker/Podman (Docker Swarm is abandonware since 2019 unfortunately). However, in my opinion K8s has the worst API and UX possible. I find Docker Compose spec much more user friendly. So I'm experimenting with a multi-host docker-compose at the moment: https://github.com/psviderski/uncloud https://github.com/psviderski/uncloud
  - byrnedo 2y ago
    Uncloud looks awesome and seems to have a great feature set!! Nice work!
    - psviderski 2y ago
      Thanks! I'm more than happy to catch up to discuss the challenges. Feel free to reach out to me on twitter @psviderski or email 'me at psviderski.name'
  - byrnedo 2y ago
    Wouldn’t argue with you abut the k8s ux. Since it has all the ground concepts ( service, cronjob etc ) it required less effort than rolling yet another syntax.
zokier 2y ago
Why on earth would you run services on a server with --user and then fiddle with lingering logins instead of just using the system service manager?
- phoronixrly 2y ago
  I'll answer this for you. You want rootless podman because docker is the defacto standard way of packaging non-legacy software now including autoupdates. I know, sad... Podman still does not offer convenient and mature way for systemd to run it with an unprivileged user. It is the only gripe I've had with this approach...
  - ggpsv 2y ago
    This is no longer true as of Podman 5 and Quadlet? You can define rootless containers to run under systemd services as unprivileged users. You can use machinectl to login as said user and interact with systemctl.
    - jcgl 2y ago
      Quadlet debuted with Podman 4.4 iirc.
      - ggpsv 2y ago
        Oh yes, correct!
    - phoronixrly 2y ago
      Can you please link the docs for this?
      - ggpsv 2y ago
        This is a good intro [0], courtesy of Redhat. This is a good example [1], cited elsewhere in this post. Documentation for quadlet systemd units [2]. [0]: https://www.redhat.com/en/blog/quadlet-podman https://www.redhat.com/en/blog/quadlet-podman [1]: https://mo8it.com/blog/quadlet/ https://mo8it.com/blog/quadlet/ [2]: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
maxclark 2y ago
I know some large AWS environments that run a variation of this Autoscaling fleet - image starts, downloads container from registry and starts on instance 1:1 relationship between instance and container - and they’re running 4XLs When you get past the initial horror it’s actually beautiful
- immibis 1y ago
  doesn't Amazon already provide a service which does this for you?
masneyb 2y ago
The next step to simplify this even further is to use Quadlet within systemd to manage the containers. More details are at https://www.redhat.com/en/blog/quadlet-podman https://www.redhat.com/en/blog/quadlet-podman
- rsolva 2y ago
  This us the way! Quadlets is such a nice way to run containers, really a set and forget experience. No need to install extra packages, at least on Fedora or Rocky Linux. I should do a write up of this some time...
  - aorth 2y ago
    Yep! My experience on Ubuntu 24.04 LTS was that I needed to create a system user to reserve the subuids / subgids for Podman (defaults to looking for a `containers` user): useradd --comment "Helper user to reserve subuids and subgids for Podman" \ --no-create-home \ --shell /usr/sbin/nologin \ containers I also found this blog post about the different `UserNS` options https://www.redhat.com/en/blog/rootless-podman-user-namespace-modes https://www.redhat.com/en/blog/rootless-podman-user-namespac... very helpful. In the end it seems that using `UserNS=auto` for rootful containers (with appropriate system security settings like private devices, etc) is easier and more secure than trying to get rootless containers running in a systemd user slice (Dan Walsh said it on a GitHub issue but I can't find it now).
    - aorth 2y ago
      I found Dan's recommendation to use rootful with `userns=auto`: > User= causes lots of issues with running podman and rootless support is fairly easy. I also recomend that people look at using rootful with --userns=auto, which will run your containers each in a unique user namespace. ― https://github.com/containers/podman/issues/12778#issuecomment-1791562753 https://github.com/containers/podman/issues/12778#issuecomme...
- overtone1000 2y ago
  I came to the comments to make sure someone mentioned quadlets. Just last week, I migrated my home server from docker compose to rootless podman quadlets. The transition was challenging, but I am very happy with the result.
VWWHFSfQ 2y ago
We went back to just packaging debs and running them directly on ec2 instances with systemd. no more containers. Put the instances in an autoscaling group with an ALB. A simple ansible-pull installs the debs on-boot. really raw-dogging it here but I got tired of endless json-inside-yaml-inside-hcl. ansible yaml is about all I want to deal with at this point.
- SvenL 2y ago
  Yes, I also have gone this route for a very simple application. Systemd was actually delightful, using a system assigned user account to run the service with the least amount of privileges is pretty cool. Also cgroup support does really make it nice to run many different services on one vps.
- secabeen 2y ago
  I also really like in this approach that if there is a bug in a common library that I use, all I have to do is `apt full-upgrade` and restart my running processes, and I am protected. No rebuilding anything, or figuring out how to update some library buried deep a container that I may (or may not) have created.
- r3trohack3r 2y ago
  The number of human lifetimes wasted on the problem domain of "managing YAML at scale"...
drivenextfunc 2y ago
I share the author's sentiment completely. At my day job, I manage multiple Kubernetes clusters running dozens of microservices with relative ease. However, for my hobby projects—which generate no revenue and thus have minimal budgets—I find myself in a frustrating position: desperately wanting to use Kubernetes but unable to due to its resource requirements. Kubernetes is simply too resource-intensive to run on a $10/month VPS with just 1 shared vCPU and 2GB of RAM. This limitation creates numerous headaches. Instead of Deployments, I'm stuck with manual docker compose up/down commands over SSH. Rather than using Ingress, I have to rely on Traefik's container discovery functionality. Recently, I even wrote a small script to manage crontab idempotently because I can't use CronJobs. I'm constantly reinventing solutions to problems that Kubernetes already solves—just less efficiently. What I really wish for is a lightweight alternative offering a Kubernetes-compatible API that runs well on inexpensive VPS instances. The gap between enterprise-grade container orchestration and affordable hobby hosting remains frustratingly wide.
- thenewwazoo 2y ago
  > I'm constantly reinventing solutions to problems that Kubernetes already solves—just less efficiently. But you've already said yourself that the cost of using K8s is too high. In one sense, you're solving those solutions more efficiently, it just depends on the axis you use to measure things.
  - randallsquared 2y ago
    The original statement is ambiguous. I read it as "problems that k8s already solves -- but k8s is less efficient, so can't be used".
  - AdrianB1 2y ago
    That picture with the almost-empty truck seems to be the situation that he describes. He wants the 18 wheeler truck, but it is too expensive for just a suitcase.
- sweettea 2y ago
  Have you seen k0s or k3s? Lots of stories about folks using these to great success on a tiny scale, e.g. https://news.ycombinator.com/item?id=43593269 https://news.ycombinator.com/item?id=43593269
dullcrisp 2y ago
I’m here just doing docker compose pull && docker compose down && docker compose up -d and it’s basically fine.
- Aluminum0643 2y ago
  I believe you can skip the "down" :)
  - dullcrisp 2y ago
    Good to know
    - jdsleppy 2y ago
      You also skip the docker compose pull if you configure it to always pull in the compose file or in the up command.
  - chupasaurus 2y ago
    docker-compose had very unfun mechanism of detecting image updates so it depends (I haven't dug deep into V2).
arjie 2y ago
I also use systemd+podman. I manage the access into the machine via an nginx that reverse proxies the services. With quadlets things will probably be even better but right now I have a manual flow with `podman run` etc. because sometimes I just want to run on the host instead and this allows for me to incrementally move in.
- sureglymop 2y ago
  I do this with traefik as the reverse proxy. To host something new, all I need to do is add a label to the new container for traefik to recognize. It's neat with a wildcard cert that traefik automatically renews. I've also heard good things about caddy, a similar alternative.
  - arjie 2y ago
    Yeah, I've heard that these new reverse proxies are great like that. I have to run certbot (which I do) and I should have created wildcard certs but I didn't. I use traefik on k3s and it's good there for other stuff.
lrvick 2y ago
Or drop all brittle c code entirely and replace systemd with kubernetes: https://www.talos.dev/ https://www.talos.dev/
- weikju 2y ago
  Just replace the battle tested brittle C code with a brittle bespoke yaml mess!
  - sepositus 2y ago
    To suggest that Kubernetes doesn't fall under the scope of "battle tested" is a bit misleading. As much as systemd? Of course not. But it's not 2016 anymore and we have billions of collective operational hours with Kubernetes. Just the other day I found a cluster I had inadvertently left running on my macBook using Kind. It literally had three weeks of uptime (running a full stack of services) and everything was still working, even with the system getting suspended repeatedly.
    - deleted 2y ago
      [deleted]
    - djhn 2y ago
      I once discovered that I had left a toy project postgres instance running on my macbook for two and a half years. Everything was working perfectly, and this was more than a decade ago, on an intel mac.
      - sepositus 2y ago
        Ha, I figured someone was going to come with a retort like that. Fair, but it would be better to compare it to running a distributed postgres cluster on your local mac.
- nullpoint420 2y ago
  +1, easiest bare-metal installation of k8s ever.
iAm25626 2y ago
on the resource part - try running k8s on Talos linux. much less overhead.
arevno 2y ago
This is cool, but it doesn't address the redundancy/availability aspect of k8s, specifically, being able to re-schedule dead services when a node (inevitably) dies.
- pphysch 2y ago
  I like to look at Kubernetes as "distributed systemd". "What if I could define a systemd unit that managed a service across multiple nodes" leads naturally to something like k8s.
- psviderski 2y ago
  Generally speaking, redundancy/availability could also be achieved through replication rather than automatic rescheduling, where you deploy multiple replicas of the service across multiple machines. If one of them dies, the other one still continues service traffic. Like in good old days when we didn't have k8s and dynamic infra. This trades off some automation for simplicity. Although, this approach may requires manual intervention when a machine fails permanently.
mdeeks 2y ago
From what I read, I think you can replace this all with a docker compose command and something like Caddy to automatically get certs. It's basically just this command once you have compose.yaml: `docker compose up -d --pull always` And then the CI setup is this: scp compose.yaml user@remote-host:~/ ssh user@remote-host 'docker compose up -d --pull always' The benefit here is that it is simple and also works on your development machine. Of course if the side goal is to also do something fun and cool and learn, then Quadlet/k8s/systemd are great options too!
- rollcat 2y ago
  Do this (once): docker context create --docker 'host=ssh://user@remote-host' remote-host Then try this instead: docker -c remote-host compose -f compose.yaml up -d --pull always ^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ No need to copy files around. Also, another pro tip: set up your ~/.ssh/config so that you don't need the user@ part in any ssh invocations. It's quite practical when working in a team, you can just copy-paste commands between docs and each other. Host *.example.com User myusername
- jdsleppy 2y ago
  Do what the sibling comment says or set DOCKER_HOST environment variable. Watch out, your local environment will be used in compose file interpolation!
thebeardisred 2y ago
Funny, because when we built Fleet (https://github.com/coreos/fleet/ https://github.com/coreos/fleet/) Kubernetes didn't exist.
mattbillenstein 2y ago
I never moved to containers and seeing the churn the community has went through with all of this complicated container tooling, I'm happy orchestrating small-scale systems with supervisord and saltstack-like chatops deployments - it's just stupid simple by comparison and provides parity between dev and prod environments that's nice.
- baggy_trough 2y ago
  try the built-in systemd containers - via nspawn. Underrated tool!
  - candiddevmike 2y ago
    Too many gaps around image management. It seems like an unfinished feature that wasn't completely thought out IMO. Podman is what systemd-nspawns OCI interface should've become.
    - MrDrMcCoy 2y ago
      The incomplete resource controls compared with other units is also annoying. Probably the biggest reason for me that I haven't used nspawn much.
      - baggy_trough 2y ago
        What's an example of that?
- cydmax 2y ago
  It looks like supervisord had it last release in December 2022. GitHub issue for a new release are not answered: https://github.com/Supervisor/supervisor/issues/1635#issue-2198571175 https://github.com/Supervisor/supervisor/issues/1635#issue-2... The original author seems to have moved on to NixOS.
- mardifoufs 2y ago
  What churn? For 95% of users, the way to use containers hasn't changed in the past decade. It's just a combination of docker CLI, maybe some docker compose for local testing and then pushing that image somewhere.
TZubiri 2y ago
Looking forward to the follow up "replacing systemd with init and bash scripts."
- egorfine 2y ago
  That will cut down memory usage by half, again. No, just kidding. As much as I'd like to, reverting back to a sane initsystem is not possible and not an option anymore.
IAmNotACellist 2y ago
That's the most disgusting sentence fragment I've ever heard. I wish it could be sent back in a localized wormhole and float across the table when Systemd was being voted on to become the next so-called init system. Edit: Nevermind, I misunderstood the article from just the headline. But I'm keeping the comment as I find the reference funny
dorongrinstein 2y ago
At https://controlplane.com https://controlplane.com we give you the power of Kubernetes without the toil of k8s. A line of code gets you a tls terminated endpoint that is geo routed to any cloud region and on-prem location. We created the Global Virtual Cloud that let's you run compute on any cloud, on premises hardware or vm's and any combination. I left vmware to start the company because the cognitive load on engineers was becoming ridiculous. Logs, metrics, tracing, service discovery, TLS, DNS, service mesh, network tunnels and much more - we made it easy. We do to the cloud what vmware did to hardware - you don't care what underlying cloud you're on. Yet you can use ANY backing service of AWS, GCP and Azure - as if they merged and your workloads are portable - they run unmodified anywhere and can consume any combination of services like RDS, Big Query, Cosmos db and any other. It is as if the cloud providers decided to merge and then lower your cost by 60-80%. Check it out. Doron
- Iwan-Zotow 2y ago
  Interesting, thanks
- nwellinghoff 2y ago
  Why / how is it 60% cheaper?
mattrighetti 2y ago
Do you actually need the container at that point? I host all of my hobby projects on a couple of raspi zeros using systemd alone, zero containers. Haven’t had a problem since when I started using it. Single binaries are super easy to setup and things rarely break, you have auto restart and launch at startup. All of the binaries get generated on GitHub using Actions and when I need to update stuff I login using ssh and execute a script that uses a GitHub token to download and replace the binary, if something is not okay I also have a rollback script that switches things back to its previous setup. It’s as simple as it gets and it’s been my go-to for 2 years now.
- bbkane 2y ago
  I do this too; but I'm looking to try the container approach when I can't use a single binary (i.e. someone else's Python project).
- mrbluecoat 2y ago
  How does your rollback work without containers?
  - eitland 2y ago
    If it is a single binary, replace the current with the previous. If it is deployed as folders, install new versions as whatever.versionnumber and upgrade by changing the symlink that points to the current version to point to the new one.
- stackskipton 2y ago
  Sure, if you are using single binary output language like Golang, Rust, C or .Net/Java self contained, containers are overkill if you are not using container management system. However, Ruby, Python, JS/TS, Java/.Net are all easier inside a container then outside. Not to say it's not doable, just hair pulling.
kgeist 2y ago
When a service is updated like this: - is there downtime? (old service down, new service hasn't started yet) - does it do health checks before directing traffic? (the process is up, but its HTTP service hasn't initialized yet) - what if the new process fails to start, how do you rollback? Or it's solved with nginx which sits in front of the containers? Or systemd has a builtin solution? Articles like this often omit such details. Or no one cares about occasional downtimes?
- LinuxAmbulance 2y ago
  Hobbyists care not for such things. Sure does matter at the enterprise level though.
  - kgeist 2y ago
    If your site has almost no users, the approach outlined in the article is viable. In all other cases, if my site were greeting visitors with random errors throughout the day, I'd consider that a pretty poor job.
teleforce 2y ago
The article is more than one year old, systemd now even has specialized officially supported OS distro for immutable workflow namely ParticleOS [1],[2]. [1] ParticleOS: https://github.com/systemd/particleos https://github.com/systemd/particleos [2] Systemd ParticleOS: https://news.ycombinator.com/item?id=43649088 https://news.ycombinator.com/item?id=43649088
- egorfine 2y ago
  Nice. The next logical step is to replace the Linux kernel with "systemd-kernel" and that will make it complete.
  - noisy_boy 2y ago
    Not deep enough; we need systemd to replace the BIOS and preferably the CPU microcodes too.
lstodd 2y ago
was actually fun running crypto traders without all this container bullshit. one service is one service. and so much simpler. it helped of course that people writing them knew what they were doing.
kaylynb 2y ago
I've run my homelab with podman-systemd (quadlet) for awhile and every time I investigate a new k8s variant it just isn't worth the extra hassle. As part of my ancient Ansible playbook I just pre-pull images and drop unit files in the right place. I even run my entire Voron 3D printer stack with podman-systemd so I can update and rollback all the components at once, although I'm looking at switching to mkosi and systemd-sysupdate and just update/rollback the entire disk image at once. The main issues are: 1. A lot of people just distribute docker-compose files, so you have to convert it to systemd units. 2. A lot of docker images have a variety of complexities around user/privilege setup that you don't need with podman. Sometimes you need to do annoying userns idmapping, especially if a container refuses to run as root and/or switches to another user. Overall, though, it's way less complicated than any k8s (or k8s variant) setup. It's also nice to have everything integrated into systemd and journald instead of being split in two places.
- Touche 2y ago
  You can use podlet to convert compose files to quadlet files. https://github.com/containers/podlet https://github.com/containers/podlet
  - kaylynb 2y ago
    It works pretty well. I've also found that some AI models are pretty decent at it too. Obviously need to fix up some of the output but the tooling for conversion is much better than when I started.
- mati365 2y ago
  Nice! I’ve been using a similar approach for years with my own setup: https://github.com/Mati365/hetzner-podman-bunjs-deploy https://github.com/Mati365/hetzner-podman-bunjs-deploy. It’s built around Podman and systemd, and honestly, nothing has broken in all that time. Super stable, super simple. Just drop your units and go. Rock solid.
  - kaylynb 2y ago
    Neat. I like to see other takes on this. Any reason to use rootless vs `userns=auto`? I haven't really seen any discussion of it other than this issue: https://github.com/containers/podman/discussions/13728 https://github.com/containers/podman/discussions/13728
MartinGAugustin 2y ago
[dead]
TacticalCoder 2y ago
[dead]
stavros 2y ago
I am of the opinion that deploying stuff to a single server shouldn't be this complicated, and I wrote a tool to deploy the way I wanted: https://harbormaster.readthedocs.io/ https://harbormaster.readthedocs.io/ Harbormaster uses a YAML file to discover repositories, clones and updates them every so often, and runs the Docker Compose files they contain. It also keeps all state in a single directory, so you can easily back everything up. That's it. It's by far the easiest and best tool for container orchestration I've come across, if all you need is a single server. I love how the entire config is declared in a repo, I love how all the state is in one directory, and I love how everything is just Compose files, nothing more complicated. I know I'm tooting my own horn, I just love it so much.
malteg 2y ago
What about using nomad? https://developer.hashicorp.com/nomad https://developer.hashicorp.com/nomad
- tgz 2y ago
  ... or (maybe) incus.
  - iotapi322 2y ago
    it's what I'm deploying everything on right now... it's still early and the update restarts everything... eeek.
- jcgl 2y ago
  I ran it for a couple years. While it had some quirks at the time, it (and the rest of the Hashi stack) were lightweight, nicely integrated, and quite pleasant. However, it’s no longer open source. Like the rest of Hashicorp’s stuff.
zhex-hn 2y ago
[dead]
xwowsersx 2y ago
FYI the link to Flux in > Particularly with GitOps and [Flux](https://www.weave.works/oss/flux/?ref=blog.yaakov.online https://www.weave.works/oss/flux/?ref=blog.yaakov.online), making changes was a breeze. appears to be broken. EDIT: oh, I hadn't realized the article was a year old.
strangus 2y ago
I just deployed a couple containers this way, was pretty easy to port the docker-compose. However, I then tried to get them to run rootless, and well, that turned out to be headache after headache. Went back to rootful, other than I'm pretty happy with the deployment.
godelski 2y ago
Systemd gets a lot of hate but it really solves a lot of problems. People really shouldn't dismiss it. I think it really happened because when systemd started appearing on distros by default people were upset they had to change Here's some cool stuff: - containers - machinectl: used for controlling: - nspawn: a more powerful chroot. This is often a better solution than docker. Super lightweight. Shares kernel - vmspawn: when nspawn isn't enough and you need full virtualization - importctl: download, import, export your machines. Get the download features in {vm,n}spawn like we have with docker. There's a hub, but it's not very active - homed/homectl: extends user management to make it easier to do things like encryption home directories (different mounts), better control of permissions, and more - mounts: forget fstab. Make it easy to auto mount and dismount drives or partitions. Can be access based, time, triggered by another unit (eg a spawn), sockets, or whatever - boot: you can not only control boot but this is really what gives you access to starting and stopping services in the boot sequence. - timers: forget cron. Cron can't wake your machine. Cron can't tell a service didn't run because your machine was off. Cron won't give you fuzzy timing, do more complicated things like wait for X minutes after boot if it's the third Sunday of the month and only if Y.service is running. Idk why you'd do that, but you can! - service units: these are your jobs. You can really control them in their capabilities. Lock them down so they can only do what they are meant to do. - overrides: use `systemctl edit` to edit your configs. Creates an override config and you don't need to destroy the original. No longer that annoying task of finding the original config and for some reason you can't get it back even if reinstalling! Same with when the original config changes in an install, your override doesn't get touched!! It's got a lot of stuff and it's (almost) all there already on your system! It's a bit annoying to learn, but it really isn't too bad if you really don't want to do anything too complicated. But in that case, it's not like there's a tool that doesn't require docs but allows you to do super complicated things.
- holuponemoment 2y ago
  Nice list, I'd add run0 as the sudo replacement. My only bugbear with it is that there's no equivalent to the old timeout default you could set (note that doas explicitly said they won't implement this too). The workaround is to run it in `sudo -i` fashion and not put a command afterwards which is reasonable enough even though it worked hard against my muscle memory + copypaste commands when switching over. > Systemd gets a lot of hate I'd argue it doesn't and is simply another victim of loud internet minority syndrome. It's just a generic name at this point, basically all associated with init and service units and none of the other stuff. https://man.archlinux.org/man/run0.1.en https://man.archlinux.org/man/run0.1.en
LelouBil 2y ago
I don't know if someone knows a better stack for my fleet of self hosted applications, maybe moving to quadlet would simplify stuff ? Right now I have an Ansible playbook responsible for updating my services, in a git repo. The playbook stops changed services, backups their configs and volumes, applies the new docker-compose.yml and other files, and restarts them. If any of them fail to start, or aren't reachable after 3 minutes, it rolls back everything *including the volumes* (using buttervolume, docker volumes as btrfs subvolumes to make snapshots free). I am looking into Kubernetes, but I didn't find a single stack/solution that would do all that this system does. For example I found nothing that can auto rollback on failure *including persistent volumes*. I found Argo Rollback but it doesn't seem to have hooks that would allow me to add the functionality.
- rollcat 2y ago
  YMMV, no warranty, IMHO, etc. Disclaimer: I haven't used k8s in a long while, mostly because I don't really have a good use case for it. You'd need to slightly rethink rollbacks, express them in terms of always rolling forward. K8s supports snapshots directly (you'd need a CSI driver; https://github.com/topolvm/topolvm https://github.com/topolvm/topolvm, https://github.com/openebs/zfs-localpv https://github.com/openebs/zfs-localpv, or similar). Restores happen by creating a new PVC (dataSource from a VolumeSnapshot). So in case rollout to version N+1 fails, instead of a rollback to N you'd roll forward to N+2 (which itself would be a copy of N, but referencing the new PVC/PV). You'd still have to script that sequence of actions somehow - perhaps back to Ansible for that? Considering there might be some YAML parsing and templating involved. Of course this looks (and likely is) much more complicated, so if your use case doesn't justify k8s in the first place, I'd stick to what already works ;)
  - LelouBil 2y ago
    Thanks for the recommendation, I'll look into it. I'm interested in moving to Kubernetes to make use of the templating languages available that are better than plain Ansible jinja2, and also offer features like schema checking. Because my services are pretty integrated together and to avoid having hardcoded values in multiple places my Ansible files are a pain to maintain
lovelysoni03 2y ago
[dead]
OptionOfT 2y ago
Wrt to the updates: Does it mean it maintains the original environment variables you passed in separate from the ones set by the container? E.g. when updating a container with watchtower: You deploy a container 'python-something'. The container has PYTHON=3.11. Now, the container has an update which sets PYTHON=3.13. Watchtower will take the current settings, and use them as the settings to be preserved. So the next version will deploy with PYTHON=3.11, even though you haven't set those settings.
sph 2y ago
I remember seeing a project in development that built k8s-like orchestration on top of systemd a couple years ago, letting you control applications across nodes and the nodes themselves with regular systemd config files and I have been unable to find it again. IIRC it was either a Redhat project or hosted under github.com/containers and looked semi-official. Anyone knows what I’m talking about? Is it still alive? EDIT: it’s not CoreOS/Fleet, it’s something much more recent, but was still in early alpha state when I found it.
- deleted 2y ago
  [deleted]
- stryan 2y ago
  Maybe you're thinking of BlueChi[0]? It used to be Red Hat project called 'hirte' and was renamed[1] and moved to the Eclipse foundation for whatever reason. It lets you control systemd units across nodes and works with the normal systemd tools (D-bus etc). [0] https://github.com/eclipse-bluechi/bluechi https://github.com/eclipse-bluechi/bluechi [1] https://www.redhat.com/en/blog/hirte-renamed-eclipse-bluechi https://www.redhat.com/en/blog/hirte-renamed-eclipse-bluechi
  - sph 2y ago
    Oh that might be it! Of course I couldn't find it, what with the rename and now it's part of Eclipse (?) Seemed quite promising to me at the time, though it seem they've changed the scope of the project a little bit. Here's an old post about it: https://www.redhat.com/en/blog/introducing-hirte-deterministic-multi-node-service-controller https://www.redhat.com/en/blog/introducing-hirte-determinist...
abhisek 2y ago
I think you are only looking at Kubernetes for running and updating container images. If that’s the use-case then I guess it’s overkill. But Kubernetes does much more in terms of providing the resources required for these containers to share state, connect to each other, get access to config or secrets etc. That’s where comes the CPU and memory cost. The cost of managing your containers and providing them the resources they need. > basically acts as a giant while loop Yep. That’s the idea of convergence of states I guess. In a distributed system you can’t always have all the participating systems behave in the desired way. So the manager (or orchestrator) of the system continuously tries to achieve the desired state.
- jpc0 2y ago
  > But Kubernetes does much more in terms of providing the resources required for these containers to share state, connect to each other, get access to config or secrets etc. This was OPs argument, and mine as well. My side project which is counting requests per minute or hour really doesn’t need that, however I need to eat the overhead of K8s just to have the nice dx of being able to push a container to a registry and it gets deployed automatically with no downtime. I don’t want to pay to host even a K3s node when my workload doesn’t even tickle a 1vCPU 256mb ram instance, but I also don’t want to build some custom scaffold to so the work. So I end up with SSH and SCP… quadlets and podman-systemd solves those problems I have reasonably well and OPs post is very valuable because it builds awareness of a solution that solves my problems.
fpoling 2y ago
At some point I tried to run a few small websites dedicated for activism (couple of Wordpress instances, a forum and custom PHP code) using docker. It was time sink as updating and testing the images turned out to be highly non-trivial. Eventually I replaced everything with a script that generated systemd units and restarted the services on changes under Debian using the Wordpress that comes with it. Then I have a test VM on my laptop and just rsync changes to the deployment host and run the deployment script there. It reduced my chores very significantly. The whole system runs on 2GB VPS. It could be reduced to 1GB if Wordpress would officially support SQLite. But I prefer to pay few more euros per month and stick to Mariadb to minimize support requirements.
forty 2y ago
I do that too, I run everything in rootless podman managed by systemd units it's quite nice. With systemd network activation I could even save the cost of user space networking, though for my single user use case, it's not really needed and for now I could not bother. I also have Quadlet on my backlog, I'm waiting the release of next stable version of Debian (which I think should be released sometimes this year) as the current version of Debian has a podman slightly too old which doesn't include Quadlet
grimblee 2y ago
For anyone interested in quadlet, I've discovered a very usefull tool to convert compose files and manifests into quadlets: podlet. It dramatically speeds up the process of converting the usual provided files into quadlets. https://github.com/containers/podlet https://github.com/containers/podlet
dgan 2y ago
why does it require so much RAM, can someone more knowledgeable explain? what does it store un such quantities ?
Aldipower 2y ago
Whoever thought running your personal blog on Kubernetes is a good idea? Kubernetes is good for large scale applications. Nothing else. I do not get why even mid-sized companies take the burden of Kubernetes although they have low infrastructur needs.
- JamesSwift 2y ago
  Its a tool that lets you punch way above your weight resource wise. As long as you dont manage the control plane, if you already know k8s there is very little reason not to use it. Otherwise you end up replicating what it does piecemeal.
  - Aldipower 2y ago
    There are a lot of reasons not to use a complex additional layer, like k8s, even if _you_ know it inside out. Long-term maintainability for example. This is especially important for low traffic sites, where it does not pay out to maintain them every 1-2 years. And it adds an additional point of failure. Trust me, I've maintained some applications running on k8s. They did always fail due to the k8s setup, not the application itself.
    - JamesSwift 2y ago
      Why do I need to trust you when I can trust myself and the clusters Ive maintained : D Until something better comes I will start with k8s 100% of the time for production systems. The minor one time pains getting it stood up are worth it compared to the migration later and everything is in place waiting to be leveraged.
dpbriggs 2y ago
I've had a really great experience with docker compose and systemd units. I use a generic systemd unit and it's just as easy as setting up users then `systemctl enable --now docker-compose@service-name`.
Tabular-Iceberg 2y ago
Couldn’t it be argued that this is a bug in Kubernetes? Most of what it does is run programs with various cgroups and namespaces, which is what systemd does, so should it really be any more resource intensive?
zoobab 2y ago
Soon back to chroot with a busybox cgi shell script as an API. I am working on proot-docker: https://github.com/mtseet/proot-docker https://github.com/mtseet/proot-docker OK it does not need process namespaces, but that's good enough for most containers.
goku12 2y ago
Well, if you're planning to run a single-node container server, then K8s is probably an overkill compared to Podman Quadlets. You just choose the lightest solution that meets your requirement. However, there was a noteworthy project named Aurae [1]. It was an ambitious project intended to replace systemd and kubelets on a server. Besides running containerized and baremetal loads, it was meant to take authenticated commands over an API and had everything that was expected on K8s worker nodes. It could work like K8s and like Docker with appropriate control planes. Unfortunately, the project came to an abrupt end when its main author Kris Nova passed away in an unfortunate accident. [1] https://aurae.io/ https://aurae.io/
NTARelix 2y ago
A couple years ago I upgraded my desktop hardware, which meant it was time to upgrade my homelab. I had gone through various operating systems and methods of managing my services: systemd on Ubuntu Server, Docker Compose on CentOS, and Podman on NixOS. I was learning about Kubernetes at work and it seemed like such a powerful tool, so I had this grand vision of building a little cluster in my laundry room with nodes net booting into Flatcar and running services via k3s. When I started building this, I was horrified by the complexity, so I went the complete opposite direction. I didn't need a cluster, net booting, blue-green deployments, or containers. I landed on NixOS with systemd for everything. Bare git repos over ssh for personal projects. Git server hooks for CI/CD. Email server for phone notifications (upgrade failures, service down, low disk space etc). NixOS nightly upgrades. I never understood the hate systemd gets, but I also never really took the time to learn it until now, and I really love the simplicity when paired with NixOS. I finally feel like I'm satisfied with the operation and management of my server (aside from a semi frequent kernel panic that I've been struggling to resolve).
mbac32768 2y ago
Reading the comments here makes me feel old. Doesn't anyone just use ssh and nginx anymore? Cram everything onto one box. Back the box up aggressively. Done. I really don't need microservices management for my home stuff.
- AndrewStephens 2y ago
  I do (nginx plus a couple of custom services) but my needs are very minimal. As soon as you need something a little complex or redundancy by spinning up multiple nodes then containers start to make a huge amount of sense.
  - athrowaway3z 2y ago
    I really have a love/hate relationship with containers. On the one hand they are entirely self contained, make redundancy simple, and - if used well - are more legible than some adhoc set up procedures. At the same time, i've seen some horrible decisions made because of them: Redis for things that do not need them. Projects with ~10.000 users (and little potential growth) tripping over themselves to adopt k8 when my desktop could run the workload of 100.000 users just fine. A disregard for backups / restore procedures because redundancy is good enough. "Look I can provision 64 extra servers for my batch job that pre-calculates a table every day". --- It seems every year fewer teams appreciate how fast modern hardware with a language like Rust or Go can be if you avoid all the overhead. My standard advice is to use a single container that holds everything. Only after its build and in use can you make the best choice at which angle to scale. > A complex system that works is invariably found to have evolved from a simple system that worked. - John Gall
    - AndrewStephens 2y ago
      Containers help in two ways. First in deployment, if you really have a complex system (and "modern" development practices seem to encourage complexity). But containers really shine during development if you have more than a few developers working on the same projects. The ability to have a standard dev container for coding and testing saves so much time. And once you have that, deploying with containers is almost free.
acanavan 2y ago
be a man, use k8s
siliconc0w 2y ago
Kamal is also a decent solution if you just have a static set of webapps that you want to easily deploy to static set of systems but still want 'production-like' features like no-downtime-deploys. And I'm pretty familiar with Kubernetes but, yeah, for small tasks it can feel like taking an Apache to the store to buy a carton of milk.
johnea 2y ago
All your container are belong to us!!!
hadlock 2y ago
CoreOS (now part of redhat) offered "fleet" which was multi-computer container orchestration, and used systemd style fleet files and raft protocol to do leader election etc (back when raft was still a novel concept). We ran it in production for about a year (2015) before it became abundantly clear (2016) the future was Kubernetes. I think the last security update Fleet got was Feb 2017 so it's pretty much dead for new adoption, if anyone is still using it. But TL;DR we've already done systemd style container orchestration, and it was ok.
Tractor8626 2y ago
Article about podman, what systemd has to do with anything? What's wrong with docker + watchtower? How is is possible to both use gitops and not "remember" which flags you used for containers? So many questions...
reddit_clone 1y ago
As a wannabe home lab hacker, but never had the energy or knowhow to get started, this thread is a goldmine of information!

436 points by birdculture 2y ago

https://github.com/coreos/fleet https://github.com/coreos/fleet I feel like fleet deserved more of a shot than it ultimately got.

deleted 2y ago
[deleted]
pa7ch 2y ago
agreed, coreos pivoted to k8s almost immediately after releaseing fleet and didn't really get the chance to use and develop it much.
- llarsson 2y ago
  That's also where etcd came from. It really felt, to me, like the precursor to Kubernetes.
thebeardisred 2y ago
Well, it's funny you mention that because I started working on a PoC of running vLLM atop Fleet this morning. :grin:
- klooney 2y ago
  I'm glad to hear it still works!

byrnedo 2y ago

I created skate (https://github.com/skateco/skate https://github.com/skateco/skate) to be basically this but multihost and support k8s manifests. Under the hood it’s podman and systemd

woile 2y ago
This looks awesome!
nemofoo 2y ago
Thank you for building this. I appreciate you.
psviderski 2y ago
This is a great approach which resonates with me a lot. It's really frustrating that there is no simple way to run a multi-host Docker/Podman (Docker Swarm is abandonware since 2019 unfortunately). However, in my opinion K8s has the worst API and UX possible. I find Docker Compose spec much more user friendly. So I'm experimenting with a multi-host docker-compose at the moment: https://github.com/psviderski/uncloud https://github.com/psviderski/uncloud
- byrnedo 2y ago
  Uncloud looks awesome and seems to have a great feature set!! Nice work!
  - psviderski 2y ago
    Thanks! I'm more than happy to catch up to discuss the challenges. Feel free to reach out to me on twitter @psviderski or email 'me at psviderski.name'
- byrnedo 2y ago
  Wouldn’t argue with you abut the k8s ux. Since it has all the ground concepts ( service, cronjob etc ) it required less effort than rolling yet another syntax.

zokier 2y ago

Why on earth would you run services on a server with --user and then fiddle with lingering logins instead of just using the system service manager?

phoronixrly 2y ago
I'll answer this for you. You want rootless podman because docker is the defacto standard way of packaging non-legacy software now including autoupdates. I know, sad... Podman still does not offer convenient and mature way for systemd to run it with an unprivileged user. It is the only gripe I've had with this approach...
- ggpsv 2y ago
  This is no longer true as of Podman 5 and Quadlet? You can define rootless containers to run under systemd services as unprivileged users. You can use machinectl to login as said user and interact with systemctl.
  - jcgl 2y ago
    Quadlet debuted with Podman 4.4 iirc.
    - ggpsv 2y ago
      Oh yes, correct!
  - phoronixrly 2y ago
    Can you please link the docs for this?
    - ggpsv 2y ago
      This is a good intro [0], courtesy of Redhat. This is a good example [1], cited elsewhere in this post. Documentation for quadlet systemd units [2]. [0]: https://www.redhat.com/en/blog/quadlet-podman https://www.redhat.com/en/blog/quadlet-podman [1]: https://mo8it.com/blog/quadlet/ https://mo8it.com/blog/quadlet/ [2]: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

maxclark 2y ago

I know some large AWS environments that run a variation of this Autoscaling fleet - image starts, downloads container from registry and starts on instance 1:1 relationship between instance and container - and they’re running 4XLs When you get past the initial horror it’s actually beautiful

immibis 1y ago
doesn't Amazon already provide a service which does this for you?

masneyb 2y ago

The next step to simplify this even further is to use Quadlet within systemd to manage the containers. More details are at https://www.redhat.com/en/blog/quadlet-podman https://www.redhat.com/en/blog/quadlet-podman

rsolva 2y ago
This us the way! Quadlets is such a nice way to run containers, really a set and forget experience. No need to install extra packages, at least on Fedora or Rocky Linux. I should do a write up of this some time...
- aorth 2y ago
  Yep! My experience on Ubuntu 24.04 LTS was that I needed to create a system user to reserve the subuids / subgids for Podman (defaults to looking for a `containers` user): useradd --comment "Helper user to reserve subuids and subgids for Podman" \ --no-create-home \ --shell /usr/sbin/nologin \ containers I also found this blog post about the different `UserNS` options https://www.redhat.com/en/blog/rootless-podman-user-namespace-modes https://www.redhat.com/en/blog/rootless-podman-user-namespac... very helpful. In the end it seems that using `UserNS=auto` for rootful containers (with appropriate system security settings like private devices, etc) is easier and more secure than trying to get rootless containers running in a systemd user slice (Dan Walsh said it on a GitHub issue but I can't find it now).
  - aorth 2y ago
    I found Dan's recommendation to use rootful with `userns=auto`: > User= causes lots of issues with running podman and rootless support is fairly easy. I also recomend that people look at using rootful with --userns=auto, which will run your containers each in a unique user namespace. ― https://github.com/containers/podman/issues/12778#issuecomment-1791562753 https://github.com/containers/podman/issues/12778#issuecomme...
overtone1000 2y ago
I came to the comments to make sure someone mentioned quadlets. Just last week, I migrated my home server from docker compose to rootless podman quadlets. The transition was challenging, but I am very happy with the result.

VWWHFSfQ 2y ago

We went back to just packaging debs and running them directly on ec2 instances with systemd. no more containers. Put the instances in an autoscaling group with an ALB. A simple ansible-pull installs the debs on-boot. really raw-dogging it here but I got tired of endless json-inside-yaml-inside-hcl. ansible yaml is about all I want to deal with at this point.

SvenL 2y ago
Yes, I also have gone this route for a very simple application. Systemd was actually delightful, using a system assigned user account to run the service with the least amount of privileges is pretty cool. Also cgroup support does really make it nice to run many different services on one vps.
secabeen 2y ago
I also really like in this approach that if there is a bug in a common library that I use, all I have to do is `apt full-upgrade` and restart my running processes, and I am protected. No rebuilding anything, or figuring out how to update some library buried deep a container that I may (or may not) have created.
r3trohack3r 2y ago
The number of human lifetimes wasted on the problem domain of "managing YAML at scale"...

drivenextfunc 2y ago

I share the author's sentiment completely. At my day job, I manage multiple Kubernetes clusters running dozens of microservices with relative ease. However, for my hobby projects—which generate no revenue and thus have minimal budgets—I find myself in a frustrating position: desperately wanting to use Kubernetes but unable to due to its resource requirements. Kubernetes is simply too resource-intensive to run on a $10/month VPS with just 1 shared vCPU and 2GB of RAM. This limitation creates numerous headaches. Instead of Deployments, I'm stuck with manual docker compose up/down commands over SSH. Rather than using Ingress, I have to rely on Traefik's container discovery functionality. Recently, I even wrote a small script to manage crontab idempotently because I can't use CronJobs. I'm constantly reinventing solutions to problems that Kubernetes already solves—just less efficiently. What I really wish for is a lightweight alternative offering a Kubernetes-compatible API that runs well on inexpensive VPS instances. The gap between enterprise-grade container orchestration and affordable hobby hosting remains frustratingly wide.

thenewwazoo 2y ago
> I'm constantly reinventing solutions to problems that Kubernetes already solves—just less efficiently. But you've already said yourself that the cost of using K8s is too high. In one sense, you're solving those solutions more efficiently, it just depends on the axis you use to measure things.
- randallsquared 2y ago
  The original statement is ambiguous. I read it as "problems that k8s already solves -- but k8s is less efficient, so can't be used".
- AdrianB1 2y ago
  That picture with the almost-empty truck seems to be the situation that he describes. He wants the 18 wheeler truck, but it is too expensive for just a suitcase.
sweettea 2y ago
Have you seen k0s or k3s? Lots of stories about folks using these to great success on a tiny scale, e.g. https://news.ycombinator.com/item?id=43593269 https://news.ycombinator.com/item?id=43593269

dullcrisp 2y ago

I’m here just doing docker compose pull && docker compose down && docker compose up -d and it’s basically fine.

Aluminum0643 2y ago
I believe you can skip the "down" :)
- dullcrisp 2y ago
  Good to know
  - jdsleppy 2y ago
    You also skip the docker compose pull if you configure it to always pull in the compose file or in the up command.
- chupasaurus 2y ago
  docker-compose had very unfun mechanism of detecting image updates so it depends (I haven't dug deep into V2).

arjie 2y ago

I also use systemd+podman. I manage the access into the machine via an nginx that reverse proxies the services. With quadlets things will probably be even better but right now I have a manual flow with `podman run` etc. because sometimes I just want to run on the host instead and this allows for me to incrementally move in.

sureglymop 2y ago
I do this with traefik as the reverse proxy. To host something new, all I need to do is add a label to the new container for traefik to recognize. It's neat with a wildcard cert that traefik automatically renews. I've also heard good things about caddy, a similar alternative.
- arjie 2y ago
  Yeah, I've heard that these new reverse proxies are great like that. I have to run certbot (which I do) and I should have created wildcard certs but I didn't. I use traefik on k3s and it's good there for other stuff.

lrvick 2y ago

Or drop all brittle c code entirely and replace systemd with kubernetes: https://www.talos.dev/ https://www.talos.dev/

weikju 2y ago
Just replace the battle tested brittle C code with a brittle bespoke yaml mess!
- sepositus 2y ago
  To suggest that Kubernetes doesn't fall under the scope of "battle tested" is a bit misleading. As much as systemd? Of course not. But it's not 2016 anymore and we have billions of collective operational hours with Kubernetes. Just the other day I found a cluster I had inadvertently left running on my macBook using Kind. It literally had three weeks of uptime (running a full stack of services) and everything was still working, even with the system getting suspended repeatedly.
  - deleted 2y ago
    [deleted]
  - djhn 2y ago
    I once discovered that I had left a toy project postgres instance running on my macbook for two and a half years. Everything was working perfectly, and this was more than a decade ago, on an intel mac.
    - sepositus 2y ago
      Ha, I figured someone was going to come with a retort like that. Fair, but it would be better to compare it to running a distributed postgres cluster on your local mac.
nullpoint420 2y ago
+1, easiest bare-metal installation of k8s ever.

iAm25626 2y ago

on the resource part - try running k8s on Talos linux. much less overhead.

arevno 2y ago

This is cool, but it doesn't address the redundancy/availability aspect of k8s, specifically, being able to re-schedule dead services when a node (inevitably) dies.

pphysch 2y ago
I like to look at Kubernetes as "distributed systemd". "What if I could define a systemd unit that managed a service across multiple nodes" leads naturally to something like k8s.
psviderski 2y ago
Generally speaking, redundancy/availability could also be achieved through replication rather than automatic rescheduling, where you deploy multiple replicas of the service across multiple machines. If one of them dies, the other one still continues service traffic. Like in good old days when we didn't have k8s and dynamic infra. This trades off some automation for simplicity. Although, this approach may requires manual intervention when a machine fails permanently.

mdeeks 2y ago

From what I read, I think you can replace this all with a docker compose command and something like Caddy to automatically get certs. It's basically just this command once you have compose.yaml: `docker compose up -d --pull always` And then the CI setup is this: scp compose.yaml user@remote-host:~/ ssh user@remote-host 'docker compose up -d --pull always' The benefit here is that it is simple and also works on your development machine. Of course if the side goal is to also do something fun and cool and learn, then Quadlet/k8s/systemd are great options too!

rollcat 2y ago
Do this (once): docker context create --docker 'host=ssh://user@remote-host' remote-host Then try this instead: docker -c remote-host compose -f compose.yaml up -d --pull always ^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ No need to copy files around. Also, another pro tip: set up your ~/.ssh/config so that you don't need the user@ part in any ssh invocations. It's quite practical when working in a team, you can just copy-paste commands between docs and each other. Host *.example.com User myusername
jdsleppy 2y ago
Do what the sibling comment says or set DOCKER_HOST environment variable. Watch out, your local environment will be used in compose file interpolation!

thebeardisred 2y ago

Funny, because when we built Fleet (https://github.com/coreos/fleet/ https://github.com/coreos/fleet/) Kubernetes didn't exist.

mattbillenstein 2y ago

I never moved to containers and seeing the churn the community has went through with all of this complicated container tooling, I'm happy orchestrating small-scale systems with supervisord and saltstack-like chatops deployments - it's just stupid simple by comparison and provides parity between dev and prod environments that's nice.

baggy_trough 2y ago
try the built-in systemd containers - via nspawn. Underrated tool!
- candiddevmike 2y ago
  Too many gaps around image management. It seems like an unfinished feature that wasn't completely thought out IMO. Podman is what systemd-nspawns OCI interface should've become.
  - MrDrMcCoy 2y ago
    The incomplete resource controls compared with other units is also annoying. Probably the biggest reason for me that I haven't used nspawn much.
    - baggy_trough 2y ago
      What's an example of that?
cydmax 2y ago
It looks like supervisord had it last release in December 2022. GitHub issue for a new release are not answered: https://github.com/Supervisor/supervisor/issues/1635#issue-2198571175 https://github.com/Supervisor/supervisor/issues/1635#issue-2... The original author seems to have moved on to NixOS.
mardifoufs 2y ago
What churn? For 95% of users, the way to use containers hasn't changed in the past decade. It's just a combination of docker CLI, maybe some docker compose for local testing and then pushing that image somewhere.

TZubiri 2y ago

Looking forward to the follow up "replacing systemd with init and bash scripts."

egorfine 2y ago
That will cut down memory usage by half, again. No, just kidding. As much as I'd like to, reverting back to a sane initsystem is not possible and not an option anymore.

IAmNotACellist 2y ago

That's the most disgusting sentence fragment I've ever heard. I wish it could be sent back in a localized wormhole and float across the table when Systemd was being voted on to become the next so-called init system. Edit: Nevermind, I misunderstood the article from just the headline. But I'm keeping the comment as I find the reference funny

dorongrinstein 2y ago

At https://controlplane.com https://controlplane.com we give you the power of Kubernetes without the toil of k8s. A line of code gets you a tls terminated endpoint that is geo routed to any cloud region and on-prem location. We created the Global Virtual Cloud that let's you run compute on any cloud, on premises hardware or vm's and any combination. I left vmware to start the company because the cognitive load on engineers was becoming ridiculous. Logs, metrics, tracing, service discovery, TLS, DNS, service mesh, network tunnels and much more - we made it easy. We do to the cloud what vmware did to hardware - you don't care what underlying cloud you're on. Yet you can use ANY backing service of AWS, GCP and Azure - as if they merged and your workloads are portable - they run unmodified anywhere and can consume any combination of services like RDS, Big Query, Cosmos db and any other. It is as if the cloud providers decided to merge and then lower your cost by 60-80%. Check it out. Doron

Iwan-Zotow 2y ago
Interesting, thanks
nwellinghoff 2y ago
Why / how is it 60% cheaper?

mattrighetti 2y ago

Do you actually need the container at that point? I host all of my hobby projects on a couple of raspi zeros using systemd alone, zero containers. Haven’t had a problem since when I started using it. Single binaries are super easy to setup and things rarely break, you have auto restart and launch at startup. All of the binaries get generated on GitHub using Actions and when I need to update stuff I login using ssh and execute a script that uses a GitHub token to download and replace the binary, if something is not okay I also have a rollback script that switches things back to its previous setup. It’s as simple as it gets and it’s been my go-to for 2 years now.

bbkane 2y ago
I do this too; but I'm looking to try the container approach when I can't use a single binary (i.e. someone else's Python project).
mrbluecoat 2y ago
How does your rollback work without containers?
- eitland 2y ago
  If it is a single binary, replace the current with the previous. If it is deployed as folders, install new versions as whatever.versionnumber and upgrade by changing the symlink that points to the current version to point to the new one.
stackskipton 2y ago
Sure, if you are using single binary output language like Golang, Rust, C or .Net/Java self contained, containers are overkill if you are not using container management system. However, Ruby, Python, JS/TS, Java/.Net are all easier inside a container then outside. Not to say it's not doable, just hair pulling.

kgeist 2y ago

When a service is updated like this: - is there downtime? (old service down, new service hasn't started yet) - does it do health checks before directing traffic? (the process is up, but its HTTP service hasn't initialized yet) - what if the new process fails to start, how do you rollback? Or it's solved with nginx which sits in front of the containers? Or systemd has a builtin solution? Articles like this often omit such details. Or no one cares about occasional downtimes?

LinuxAmbulance 2y ago
Hobbyists care not for such things. Sure does matter at the enterprise level though.
- kgeist 2y ago
  If your site has almost no users, the approach outlined in the article is viable. In all other cases, if my site were greeting visitors with random errors throughout the day, I'd consider that a pretty poor job.

teleforce 2y ago

The article is more than one year old, systemd now even has specialized officially supported OS distro for immutable workflow namely ParticleOS [1],[2]. [1] ParticleOS: https://github.com/systemd/particleos https://github.com/systemd/particleos [2] Systemd ParticleOS: https://news.ycombinator.com/item?id=43649088 https://news.ycombinator.com/item?id=43649088

egorfine 2y ago
Nice. The next logical step is to replace the Linux kernel with "systemd-kernel" and that will make it complete.
- noisy_boy 2y ago
  Not deep enough; we need systemd to replace the BIOS and preferably the CPU microcodes too.

lstodd 2y ago

was actually fun running crypto traders without all this container bullshit. one service is one service. and so much simpler. it helped of course that people writing them knew what they were doing.

kaylynb 2y ago

I've run my homelab with podman-systemd (quadlet) for awhile and every time I investigate a new k8s variant it just isn't worth the extra hassle. As part of my ancient Ansible playbook I just pre-pull images and drop unit files in the right place. I even run my entire Voron 3D printer stack with podman-systemd so I can update and rollback all the components at once, although I'm looking at switching to mkosi and systemd-sysupdate and just update/rollback the entire disk image at once. The main issues are: 1. A lot of people just distribute docker-compose files, so you have to convert it to systemd units. 2. A lot of docker images have a variety of complexities around user/privilege setup that you don't need with podman. Sometimes you need to do annoying userns idmapping, especially if a container refuses to run as root and/or switches to another user. Overall, though, it's way less complicated than any k8s (or k8s variant) setup. It's also nice to have everything integrated into systemd and journald instead of being split in two places.

Touche 2y ago
You can use podlet to convert compose files to quadlet files. https://github.com/containers/podlet https://github.com/containers/podlet
- kaylynb 2y ago
  It works pretty well. I've also found that some AI models are pretty decent at it too. Obviously need to fix up some of the output but the tooling for conversion is much better than when I started.
mati365 2y ago
Nice! I’ve been using a similar approach for years with my own setup: https://github.com/Mati365/hetzner-podman-bunjs-deploy https://github.com/Mati365/hetzner-podman-bunjs-deploy. It’s built around Podman and systemd, and honestly, nothing has broken in all that time. Super stable, super simple. Just drop your units and go. Rock solid.
- kaylynb 2y ago
  Neat. I like to see other takes on this. Any reason to use rootless vs `userns=auto`? I haven't really seen any discussion of it other than this issue: https://github.com/containers/podman/discussions/13728 https://github.com/containers/podman/discussions/13728

MartinGAugustin 2y ago

[dead]

TacticalCoder 2y ago

[dead]

stavros 2y ago

I am of the opinion that deploying stuff to a single server shouldn't be this complicated, and I wrote a tool to deploy the way I wanted: https://harbormaster.readthedocs.io/ https://harbormaster.readthedocs.io/ Harbormaster uses a YAML file to discover repositories, clones and updates them every so often, and runs the Docker Compose files they contain. It also keeps all state in a single directory, so you can easily back everything up. That's it. It's by far the easiest and best tool for container orchestration I've come across, if all you need is a single server. I love how the entire config is declared in a repo, I love how all the state is in one directory, and I love how everything is just Compose files, nothing more complicated. I know I'm tooting my own horn, I just love it so much.

malteg 2y ago

What about using nomad? https://developer.hashicorp.com/nomad https://developer.hashicorp.com/nomad

tgz 2y ago
... or (maybe) incus.
- iotapi322 2y ago
  it's what I'm deploying everything on right now... it's still early and the update restarts everything... eeek.
jcgl 2y ago
I ran it for a couple years. While it had some quirks at the time, it (and the rest of the Hashi stack) were lightweight, nicely integrated, and quite pleasant. However, it’s no longer open source. Like the rest of Hashicorp’s stuff.

zhex-hn 2y ago

[dead]

xwowsersx 2y ago

FYI the link to Flux in > Particularly with GitOps and [Flux](https://www.weave.works/oss/flux/?ref=blog.yaakov.online https://www.weave.works/oss/flux/?ref=blog.yaakov.online), making changes was a breeze. appears to be broken. EDIT: oh, I hadn't realized the article was a year old.

strangus 2y ago

I just deployed a couple containers this way, was pretty easy to port the docker-compose. However, I then tried to get them to run rootless, and well, that turned out to be headache after headache. Went back to rootful, other than I'm pretty happy with the deployment.

godelski 2y ago

Systemd gets a lot of hate but it really solves a lot of problems. People really shouldn't dismiss it. I think it really happened because when systemd started appearing on distros by default people were upset they had to change Here's some cool stuff: - containers - machinectl: used for controlling: - nspawn: a more powerful chroot. This is often a better solution than docker. Super lightweight. Shares kernel - vmspawn: when nspawn isn't enough and you need full virtualization - importctl: download, import, export your machines. Get the download features in {vm,n}spawn like we have with docker. There's a hub, but it's not very active - homed/homectl: extends user management to make it easier to do things like encryption home directories (different mounts), better control of permissions, and more - mounts: forget fstab. Make it easy to auto mount and dismount drives or partitions. Can be access based, time, triggered by another unit (eg a spawn), sockets, or whatever - boot: you can not only control boot but this is really what gives you access to starting and stopping services in the boot sequence. - timers: forget cron. Cron can't wake your machine. Cron can't tell a service didn't run because your machine was off. Cron won't give you fuzzy timing, do more complicated things like wait for X minutes after boot if it's the third Sunday of the month and only if Y.service is running. Idk why you'd do that, but you can! - service units: these are your jobs. You can really control them in their capabilities. Lock them down so they can only do what they are meant to do. - overrides: use `systemctl edit` to edit your configs. Creates an override config and you don't need to destroy the original. No longer that annoying task of finding the original config and for some reason you can't get it back even if reinstalling! Same with when the original config changes in an install, your override doesn't get touched!! It's got a lot of stuff and it's (almost) all there already on your system! It's a bit annoying to learn, but it really isn't too bad if you really don't want to do anything too complicated. But in that case, it's not like there's a tool that doesn't require docs but allows you to do super complicated things.

holuponemoment 2y ago
Nice list, I'd add run0 as the sudo replacement. My only bugbear with it is that there's no equivalent to the old timeout default you could set (note that doas explicitly said they won't implement this too). The workaround is to run it in `sudo -i` fashion and not put a command afterwards which is reasonable enough even though it worked hard against my muscle memory + copypaste commands when switching over. > Systemd gets a lot of hate I'd argue it doesn't and is simply another victim of loud internet minority syndrome. It's just a generic name at this point, basically all associated with init and service units and none of the other stuff. https://man.archlinux.org/man/run0.1.en https://man.archlinux.org/man/run0.1.en

LelouBil 2y ago

I don't know if someone knows a better stack for my fleet of self hosted applications, maybe moving to quadlet would simplify stuff ? Right now I have an Ansible playbook responsible for updating my services, in a git repo. The playbook stops changed services, backups their configs and volumes, applies the new docker-compose.yml and other files, and restarts them. If any of them fail to start, or aren't reachable after 3 minutes, it rolls back everything *including the volumes* (using buttervolume, docker volumes as btrfs subvolumes to make snapshots free). I am looking into Kubernetes, but I didn't find a single stack/solution that would do all that this system does. For example I found nothing that can auto rollback on failure *including persistent volumes*. I found Argo Rollback but it doesn't seem to have hooks that would allow me to add the functionality.

rollcat 2y ago
YMMV, no warranty, IMHO, etc. Disclaimer: I haven't used k8s in a long while, mostly because I don't really have a good use case for it. You'd need to slightly rethink rollbacks, express them in terms of always rolling forward. K8s supports snapshots directly (you'd need a CSI driver; https://github.com/topolvm/topolvm https://github.com/topolvm/topolvm, https://github.com/openebs/zfs-localpv https://github.com/openebs/zfs-localpv, or similar). Restores happen by creating a new PVC (dataSource from a VolumeSnapshot). So in case rollout to version N+1 fails, instead of a rollback to N you'd roll forward to N+2 (which itself would be a copy of N, but referencing the new PVC/PV). You'd still have to script that sequence of actions somehow - perhaps back to Ansible for that? Considering there might be some YAML parsing and templating involved. Of course this looks (and likely is) much more complicated, so if your use case doesn't justify k8s in the first place, I'd stick to what already works ;)
- LelouBil 2y ago
  Thanks for the recommendation, I'll look into it. I'm interested in moving to Kubernetes to make use of the templating languages available that are better than plain Ansible jinja2, and also offer features like schema checking. Because my services are pretty integrated together and to avoid having hardcoded values in multiple places my Ansible files are a pain to maintain

lovelysoni03 2y ago

[dead]

OptionOfT 2y ago

Wrt to the updates: Does it mean it maintains the original environment variables you passed in separate from the ones set by the container? E.g. when updating a container with watchtower: You deploy a container 'python-something'. The container has PYTHON=3.11. Now, the container has an update which sets PYTHON=3.13. Watchtower will take the current settings, and use them as the settings to be preserved. So the next version will deploy with PYTHON=3.11, even though you haven't set those settings.

sph 2y ago

I remember seeing a project in development that built k8s-like orchestration on top of systemd a couple years ago, letting you control applications across nodes and the nodes themselves with regular systemd config files and I have been unable to find it again. IIRC it was either a Redhat project or hosted under github.com/containers and looked semi-official. Anyone knows what I’m talking about? Is it still alive? EDIT: it’s not CoreOS/Fleet, it’s something much more recent, but was still in early alpha state when I found it.

deleted 2y ago
[deleted]
stryan 2y ago
Maybe you're thinking of BlueChi[0]? It used to be Red Hat project called 'hirte' and was renamed[1] and moved to the Eclipse foundation for whatever reason. It lets you control systemd units across nodes and works with the normal systemd tools (D-bus etc). [0] https://github.com/eclipse-bluechi/bluechi https://github.com/eclipse-bluechi/bluechi [1] https://www.redhat.com/en/blog/hirte-renamed-eclipse-bluechi https://www.redhat.com/en/blog/hirte-renamed-eclipse-bluechi
- sph 2y ago
  Oh that might be it! Of course I couldn't find it, what with the rename and now it's part of Eclipse (?) Seemed quite promising to me at the time, though it seem they've changed the scope of the project a little bit. Here's an old post about it: https://www.redhat.com/en/blog/introducing-hirte-deterministic-multi-node-service-controller https://www.redhat.com/en/blog/introducing-hirte-determinist...

abhisek 2y ago

I think you are only looking at Kubernetes for running and updating container images. If that’s the use-case then I guess it’s overkill. But Kubernetes does much more in terms of providing the resources required for these containers to share state, connect to each other, get access to config or secrets etc. That’s where comes the CPU and memory cost. The cost of managing your containers and providing them the resources they need. > basically acts as a giant while loop Yep. That’s the idea of convergence of states I guess. In a distributed system you can’t always have all the participating systems behave in the desired way. So the manager (or orchestrator) of the system continuously tries to achieve the desired state.

jpc0 2y ago
> But Kubernetes does much more in terms of providing the resources required for these containers to share state, connect to each other, get access to config or secrets etc. This was OPs argument, and mine as well. My side project which is counting requests per minute or hour really doesn’t need that, however I need to eat the overhead of K8s just to have the nice dx of being able to push a container to a registry and it gets deployed automatically with no downtime. I don’t want to pay to host even a K3s node when my workload doesn’t even tickle a 1vCPU 256mb ram instance, but I also don’t want to build some custom scaffold to so the work. So I end up with SSH and SCP… quadlets and podman-systemd solves those problems I have reasonably well and OPs post is very valuable because it builds awareness of a solution that solves my problems.

fpoling 2y ago

At some point I tried to run a few small websites dedicated for activism (couple of Wordpress instances, a forum and custom PHP code) using docker. It was time sink as updating and testing the images turned out to be highly non-trivial. Eventually I replaced everything with a script that generated systemd units and restarted the services on changes under Debian using the Wordpress that comes with it. Then I have a test VM on my laptop and just rsync changes to the deployment host and run the deployment script there. It reduced my chores very significantly. The whole system runs on 2GB VPS. It could be reduced to 1GB if Wordpress would officially support SQLite. But I prefer to pay few more euros per month and stick to Mariadb to minimize support requirements.

forty 2y ago

I do that too, I run everything in rootless podman managed by systemd units it's quite nice. With systemd network activation I could even save the cost of user space networking, though for my single user use case, it's not really needed and for now I could not bother. I also have Quadlet on my backlog, I'm waiting the release of next stable version of Debian (which I think should be released sometimes this year) as the current version of Debian has a podman slightly too old which doesn't include Quadlet

grimblee 2y ago

For anyone interested in quadlet, I've discovered a very usefull tool to convert compose files and manifests into quadlets: podlet. It dramatically speeds up the process of converting the usual provided files into quadlets. https://github.com/containers/podlet https://github.com/containers/podlet

dgan 2y ago

why does it require so much RAM, can someone more knowledgeable explain? what does it store un such quantities ?

Aldipower 2y ago

Whoever thought running your personal blog on Kubernetes is a good idea? Kubernetes is good for large scale applications. Nothing else. I do not get why even mid-sized companies take the burden of Kubernetes although they have low infrastructur needs.

JamesSwift 2y ago
Its a tool that lets you punch way above your weight resource wise. As long as you dont manage the control plane, if you already know k8s there is very little reason not to use it. Otherwise you end up replicating what it does piecemeal.
- Aldipower 2y ago
  There are a lot of reasons not to use a complex additional layer, like k8s, even if _you_ know it inside out. Long-term maintainability for example. This is especially important for low traffic sites, where it does not pay out to maintain them every 1-2 years. And it adds an additional point of failure. Trust me, I've maintained some applications running on k8s. They did always fail due to the k8s setup, not the application itself.
  - JamesSwift 2y ago
    Why do I need to trust you when I can trust myself and the clusters Ive maintained : D Until something better comes I will start with k8s 100% of the time for production systems. The minor one time pains getting it stood up are worth it compared to the migration later and everything is in place waiting to be leveraged.

dpbriggs 2y ago

I've had a really great experience with docker compose and systemd units. I use a generic systemd unit and it's just as easy as setting up users then `systemctl enable --now docker-compose@service-name`.

Tabular-Iceberg 2y ago

Couldn’t it be argued that this is a bug in Kubernetes? Most of what it does is run programs with various cgroups and namespaces, which is what systemd does, so should it really be any more resource intensive?

zoobab 2y ago

Soon back to chroot with a busybox cgi shell script as an API. I am working on proot-docker: https://github.com/mtseet/proot-docker https://github.com/mtseet/proot-docker OK it does not need process namespaces, but that's good enough for most containers.

goku12 2y ago

Well, if you're planning to run a single-node container server, then K8s is probably an overkill compared to Podman Quadlets. You just choose the lightest solution that meets your requirement. However, there was a noteworthy project named Aurae [1]. It was an ambitious project intended to replace systemd and kubelets on a server. Besides running containerized and baremetal loads, it was meant to take authenticated commands over an API and had everything that was expected on K8s worker nodes. It could work like K8s and like Docker with appropriate control planes. Unfortunately, the project came to an abrupt end when its main author Kris Nova passed away in an unfortunate accident. [1] https://aurae.io/ https://aurae.io/

NTARelix 2y ago

A couple years ago I upgraded my desktop hardware, which meant it was time to upgrade my homelab. I had gone through various operating systems and methods of managing my services: systemd on Ubuntu Server, Docker Compose on CentOS, and Podman on NixOS. I was learning about Kubernetes at work and it seemed like such a powerful tool, so I had this grand vision of building a little cluster in my laundry room with nodes net booting into Flatcar and running services via k3s. When I started building this, I was horrified by the complexity, so I went the complete opposite direction. I didn't need a cluster, net booting, blue-green deployments, or containers. I landed on NixOS with systemd for everything. Bare git repos over ssh for personal projects. Git server hooks for CI/CD. Email server for phone notifications (upgrade failures, service down, low disk space etc). NixOS nightly upgrades. I never understood the hate systemd gets, but I also never really took the time to learn it until now, and I really love the simplicity when paired with NixOS. I finally feel like I'm satisfied with the operation and management of my server (aside from a semi frequent kernel panic that I've been struggling to resolve).

mbac32768 2y ago

Reading the comments here makes me feel old. Doesn't anyone just use ssh and nginx anymore? Cram everything onto one box. Back the box up aggressively. Done. I really don't need microservices management for my home stuff.

AndrewStephens 2y ago
I do (nginx plus a couple of custom services) but my needs are very minimal. As soon as you need something a little complex or redundancy by spinning up multiple nodes then containers start to make a huge amount of sense.
- athrowaway3z 2y ago
  I really have a love/hate relationship with containers. On the one hand they are entirely self contained, make redundancy simple, and - if used well - are more legible than some adhoc set up procedures. At the same time, i've seen some horrible decisions made because of them: Redis for things that do not need them. Projects with ~10.000 users (and little potential growth) tripping over themselves to adopt k8 when my desktop could run the workload of 100.000 users just fine. A disregard for backups / restore procedures because redundancy is good enough. "Look I can provision 64 extra servers for my batch job that pre-calculates a table every day". --- It seems every year fewer teams appreciate how fast modern hardware with a language like Rust or Go can be if you avoid all the overhead. My standard advice is to use a single container that holds everything. Only after its build and in use can you make the best choice at which angle to scale. > A complex system that works is invariably found to have evolved from a simple system that worked. - John Gall
  - AndrewStephens 2y ago
    Containers help in two ways. First in deployment, if you really have a complex system (and "modern" development practices seem to encourage complexity). But containers really shine during development if you have more than a few developers working on the same projects. The ability to have a standard dev container for coding and testing saves so much time. And once you have that, deploying with containers is almost free.

acanavan 2y ago

be a man, use k8s

siliconc0w 2y ago

Kamal is also a decent solution if you just have a static set of webapps that you want to easily deploy to static set of systems but still want 'production-like' features like no-downtime-deploys. And I'm pretty familiar with Kubernetes but, yeah, for small tasks it can feel like taking an Apache to the store to buy a carton of milk.

johnea 2y ago

All your container are belong to us!!!

hadlock 2y ago

CoreOS (now part of redhat) offered "fleet" which was multi-computer container orchestration, and used systemd style fleet files and raft protocol to do leader election etc (back when raft was still a novel concept). We ran it in production for about a year (2015) before it became abundantly clear (2016) the future was Kubernetes. I think the last security update Fleet got was Feb 2017 so it's pretty much dead for new adoption, if anyone is still using it. But TL;DR we've already done systemd style container orchestration, and it was ok.

Tractor8626 2y ago

Article about podman, what systemd has to do with anything? What's wrong with docker + watchtower? How is is possible to both use gitops and not "remember" which flags you used for containers? So many questions...

reddit_clone 1y ago

As a wannabe home lab hacker, but never had the energy or knowhow to get started, this thread is a goldmine of information!