7 ms·

The main purpose of K8s is just to run microservices. It makes things convenient by automatically restarting services, running multiple copies across servers, r

The main purpose of K8s is just to run microservices. It makes things convenient by automatically restarting services, running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ...All things Systemd does. Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone. All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. Managed Distributions of K8s go away if every Linux distro can do the same things out of the box. "Managing fleets" just becomes running more Linux VMs or not. You wouldn't need "clusters", you'd just run a bunch of boxes and run replicas of whatever service you wanted, and all resources would be managed using normal Linux system permissions and Systemd configs. No volume mounts or persistent storage setup, it's just regular filesystem mounts and disks. Containers would be optional, eliminating a huge class of complexity (static apps make a lot of containerization moot). And most importantly to admins - no more bullshit 15 month upgrade death march. Imagine just sticking to a stable Debian for 3 years but still having all the power of a distributed orchestrator. Imagine not having 50 components with APIs listening on the internet to be hacked. Imagine just using IPTables rather than custom plugins that change based on the Cloud provider just to filter network traffic, and not letting everything on the network access everything else by default. K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs. I also predict RedHat will block patches to make this happen, since their money comes from K8s. But someone just needs to make a Systemd fork using Systemd's own API and we can finally fix all the crap Systemd devs like to pretend is somebody else's bug.

deleted 4y ago
[deleted]
loriverkutya 4y ago
I have the "If you have a SystemD everything is a nail" feeling from this comment.
TakeBlaster16 4y ago
> K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs. So "all" we need to do is rewrite systemd as a distributed system, and make it "cloud-focused" and "modern"? No thanks. Right tool for the right job.
- 0xbadcafebee 4y ago
  It's basically a distributed system already! All you need for feature parity with K8s is a parent node sending child nodes commands via its existing API. Everything else is already there.
  - zozbot234 4y ago
    So, like systemd socket activation (which is already there, too)?
    - 0xbadcafebee 4y ago
      More like the job controller: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/job_controller.go https://github.com/kubernetes/kubernetes/blob/master/pkg/con... It's a loop that schedules jobs. When you create a job it starts them, when one crashes it schedules another, when you delete one it stops them. Systemd already does this for a single node's service units, so all that's lacking is doing this across nodes.
  - TakeBlaster16 4y ago
    Yes... kind of. A distributed system all on one machine skips many of the hard parts. How does systemd handle a network partition? How does systemd mount storage or direct network connections to the right nodes? How do you migrate physical hosts without downtime?
    - 0xbadcafebee 4y ago
topspin 4y ago
> The main purpose of K8s is just to run microservices. Agreed, except I'd add 'stateless' to the microservices part. That's all k3s is good for and trying to make k8s do more is awful. > Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone. Far more is needed. CRIU needs to work; live migrating containers between hosts. Network overlay, with transparent encryption, name resolution and access control are necessary. Bulletproof network storage management is crucial. Containers need to easily float among hosts, and the networking and storage they depend on must follow seamlessly. Most of the pieces exist, but we're a long way off. I believe the future is an operating system with integrated clustering solely focused on hosting containers and the networking and storage they need.
- 0xbadcafebee 4y ago
  You don't need any of that stuff to run microservices. Stateless is up to the app, not the orchestrator (you can still run a stateful containerized app in K8s). You absolutely do not need containers, network overlay, "transparent encryption" (do you mean a service mesh?), name resolution (dnsmasq works fine), and access controls are inherent in systemd. Network storage is unrelated if you mean NFS; if you mean persistent storage volume claims shuffled around nodes, yeah, that's... not a stateless microservice. And.... no, we don't need CRIU, but it works just fine without K8s. All you need to run microservices is 1) a statically-compiled stateless app, 2) a systemd unit file to run the app and restart it when it stops, 3) load any credentials into systemd's credential manager, 4) iptables for network ACLs, 5) linux user/group permissions, 6) an Nginx load balancer to route traffic to different services, 7) journald to keep logs, 8) systemd to set per-service CPU/memory limits, 9) a dynamic DNS client to register a service name in dnsmasq when a service starts/stops. Systemd already handles intra-service network communication by handing an open filehandle to a service; alternately each service could grab its own random port and register it with Nginx along with its service name and the HTTP routes and hosts it wants to listen on. If you shackle yourself to the design of K8s, or some implementation of K8s (K8s doesn't have all the features you described out of the box) and say "well that other system isn't K8s!", then yeah, you will find systemd wanting.
  - 4y ago
zozbot234 4y ago
You can run containers under systemd already via systemd-nspawn (and even manage/orchestrate them via machinectl) but configuring them is just as complex as on any other containerization solution... Don't see much of a point. And containers provide a lot more than just replicating static linking, they're about namespacing of all resources and not just a different filesystem root.
zaat 4y ago
> All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. For example, two operators I'm currently implementing for a client, ArgoCD and RabbitMQ Topology operator. The first allow GitOps, meaning that we update config file and the software running in the clusters immediately gets updated or configuration is being set. The second let me declare exchanges and queues like any other Kubernetes object. Now I'm sure it is possible to do the same in SystemD, anything can be done with enough engineering. But is it really the right tool for the job? in any case, the function of Operators is not to simplify and complex abstractions, this is simply incorrect.
qbasic_forever 4y ago
Systemd is great... for one machine. Systemd has no concept of multi-machine orchestration. It has no distributed state or consensus which is the foundation of multiple machines orchestrating work. I suspect systemd by design will _never_ try to do multi-machine orchestration and prefer people use an existing solution... like kubernetes. I love systemd but it's not designed for managing a cluster of machines.
candiddevmike 4y ago
Ha, I had a similar thought and started creating exactly that: clusterd, a way to cluster and schedule things on systemd instances. I haven't gotten very far, but if you're interested in helping me please reach out (email in profile).
devmunchies 4y ago
How would you run multiple instances of an app on a single machine with systemd? I.e. if they are listening in the same port.
Fiahil 4y ago
> The main purpose of K8s is just to run microservices. No. The main purpose of K8s is to make portable « running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ». Systemd could do all that, but, then, it would not be Systemd anymore and would be like another K8s.
rad_gruchalski 4y ago
> All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. CNI isn’t some silly Kubernetes abstraction to scratch an itch. CNI is a specification and a set of libraries for container networking. Calico is built on top of CNI, Nomad requires CNI for multi-machine networking, Mesos uses CNI for network isolation, Firecracker uses CNI extensively. Docker does not use CNI directly but their libnetwork code depends on exactly the same principles as CNI. Life everywhere is simpler thanks to CNI. Do a thought experiment. If you think it’s that easy, why don’t you try marrying systemd with Docker overlay networks and see how far you get?

deleted 4y ago

[deleted]

loriverkutya 4y ago

I have the "If you have a SystemD everything is a nail" feeling from this comment.

TakeBlaster16 4y ago

> K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs. So "all" we need to do is rewrite systemd as a distributed system, and make it "cloud-focused" and "modern"? No thanks. Right tool for the right job.

0xbadcafebee 4y ago
It's basically a distributed system already! All you need for feature parity with K8s is a parent node sending child nodes commands via its existing API. Everything else is already there.
- zozbot234 4y ago
  So, like systemd socket activation (which is already there, too)?
  - 0xbadcafebee 4y ago
    More like the job controller: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/job_controller.go https://github.com/kubernetes/kubernetes/blob/master/pkg/con... It's a loop that schedules jobs. When you create a job it starts them, when one crashes it schedules another, when you delete one it stops them. Systemd already does this for a single node's service units, so all that's lacking is doing this across nodes.
- TakeBlaster16 4y ago
  Yes... kind of. A distributed system all on one machine skips many of the hard parts. How does systemd handle a network partition? How does systemd mount storage or direct network connections to the right nodes? How do you migrate physical hosts without downtime?
  - 0xbadcafebee 4y ago

topspin 4y ago

> The main purpose of K8s is just to run microservices. Agreed, except I'd add 'stateless' to the microservices part. That's all k3s is good for and trying to make k8s do more is awful. > Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone. Far more is needed. CRIU needs to work; live migrating containers between hosts. Network overlay, with transparent encryption, name resolution and access control are necessary. Bulletproof network storage management is crucial. Containers need to easily float among hosts, and the networking and storage they depend on must follow seamlessly. Most of the pieces exist, but we're a long way off. I believe the future is an operating system with integrated clustering solely focused on hosting containers and the networking and storage they need.

0xbadcafebee 4y ago
You don't need any of that stuff to run microservices. Stateless is up to the app, not the orchestrator (you can still run a stateful containerized app in K8s). You absolutely do not need containers, network overlay, "transparent encryption" (do you mean a service mesh?), name resolution (dnsmasq works fine), and access controls are inherent in systemd. Network storage is unrelated if you mean NFS; if you mean persistent storage volume claims shuffled around nodes, yeah, that's... not a stateless microservice. And.... no, we don't need CRIU, but it works just fine without K8s. All you need to run microservices is 1) a statically-compiled stateless app, 2) a systemd unit file to run the app and restart it when it stops, 3) load any credentials into systemd's credential manager, 4) iptables for network ACLs, 5) linux user/group permissions, 6) an Nginx load balancer to route traffic to different services, 7) journald to keep logs, 8) systemd to set per-service CPU/memory limits, 9) a dynamic DNS client to register a service name in dnsmasq when a service starts/stops. Systemd already handles intra-service network communication by handing an open filehandle to a service; alternately each service could grab its own random port and register it with Nginx along with its service name and the HTTP routes and hosts it wants to listen on. If you shackle yourself to the design of K8s, or some implementation of K8s (K8s doesn't have all the features you described out of the box) and say "well that other system isn't K8s!", then yeah, you will find systemd wanting.
- 4y ago

zozbot234 4y ago

You can run containers under systemd already via systemd-nspawn (and even manage/orchestrate them via machinectl) but configuring them is just as complex as on any other containerization solution... Don't see much of a point. And containers provide a lot more than just replicating static linking, they're about namespacing of all resources and not just a different filesystem root.

zaat 4y ago

> All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. For example, two operators I'm currently implementing for a client, ArgoCD and RabbitMQ Topology operator. The first allow GitOps, meaning that we update config file and the software running in the clusters immediately gets updated or configuration is being set. The second let me declare exchanges and queues like any other Kubernetes object. Now I'm sure it is possible to do the same in SystemD, anything can be done with enough engineering. But is it really the right tool for the job? in any case, the function of Operators is not to simplify and complex abstractions, this is simply incorrect.

qbasic_forever 4y ago

Systemd is great... for one machine. Systemd has no concept of multi-machine orchestration. It has no distributed state or consensus which is the foundation of multiple machines orchestrating work. I suspect systemd by design will _never_ try to do multi-machine orchestration and prefer people use an existing solution... like kubernetes. I love systemd but it's not designed for managing a cluster of machines.

candiddevmike 4y ago

Ha, I had a similar thought and started creating exactly that: clusterd, a way to cluster and schedule things on systemd instances. I haven't gotten very far, but if you're interested in helping me please reach out (email in profile).

devmunchies 4y ago

How would you run multiple instances of an app on a single machine with systemd? I.e. if they are listening in the same port.

Fiahil 4y ago

> The main purpose of K8s is just to run microservices. No. The main purpose of K8s is to make portable « running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ». Systemd could do all that, but, then, it would not be Systemd anymore and would be like another K8s.

rad_gruchalski 4y ago

> All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. CNI isn’t some silly Kubernetes abstraction to scratch an itch. CNI is a specification and a set of libraries for container networking. Calico is built on top of CNI, Nomad requires CNI for multi-machine networking, Mesos uses CNI for network isolation, Firecracker uses CNI extensively. Docker does not use CNI directly but their libnetwork code depends on exactly the same principles as CNI. Life everywhere is simpler thanks to CNI. Do a thought experiment. If you think it’s that easy, why don’t you try marrying systemd with Docker overlay networks and see how far you get?