3 ms·

> The main purpose of K8s is just to run microservices. Agreed, except I'd add 'stateless' to the microservices part. That's all k3s is good for and trying to

> The main purpose of K8s is just to run microservices. Agreed, except I'd add 'stateless' to the microservices part. That's all k3s is good for and trying to make k8s do more is awful. > Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone. Far more is needed. CRIU needs to work; live migrating containers between hosts. Network overlay, with transparent encryption, name resolution and access control are necessary. Bulletproof network storage management is crucial. Containers need to easily float among hosts, and the networking and storage they depend on must follow seamlessly. Most of the pieces exist, but we're a long way off. I believe the future is an operating system with integrated clustering solely focused on hosting containers and the networking and storage they need.

0xbadcafebee 4y ago
You don't need any of that stuff to run microservices. Stateless is up to the app, not the orchestrator (you can still run a stateful containerized app in K8s). You absolutely do not need containers, network overlay, "transparent encryption" (do you mean a service mesh?), name resolution (dnsmasq works fine), and access controls are inherent in systemd. Network storage is unrelated if you mean NFS; if you mean persistent storage volume claims shuffled around nodes, yeah, that's... not a stateless microservice. And.... no, we don't need CRIU, but it works just fine without K8s. All you need to run microservices is 1) a statically-compiled stateless app, 2) a systemd unit file to run the app and restart it when it stops, 3) load any credentials into systemd's credential manager, 4) iptables for network ACLs, 5) linux user/group permissions, 6) an Nginx load balancer to route traffic to different services, 7) journald to keep logs, 8) systemd to set per-service CPU/memory limits, 9) a dynamic DNS client to register a service name in dnsmasq when a service starts/stops. Systemd already handles intra-service network communication by handing an open filehandle to a service; alternately each service could grab its own random port and register it with Nginx along with its service name and the HTTP routes and hosts it wants to listen on. If you shackle yourself to the design of K8s, or some implementation of K8s (K8s doesn't have all the features you described out of the box) and say "well that other system isn't K8s!", then yeah, you will find systemd wanting.
- chucky_z 4y ago
  You could run Nomad and use parameterized jobs with raw_exec to get something like this. I know that systemd has some lifecycle stuff where you can create/run systemd units directly through the cli instead of using unit files which should allow you to hook the lifecycle into a nomad style service invocation but I can't recall exactly how to do this. This doesn't quite roll into the 'only systemd,' but it allows you to build exactly what you're looking for.
  - 0xbadcafebee 4y ago
    I would rather systemd added a single feature to its already vast arsenal, and then Nomad wouldn't be needed either.
    - chucky_z 4y ago
      Figure out how to revive https://github.com/coreos/fleet https://github.com/coreos/fleet as something native in systemd?
      - 0xbadcafebee 4y ago
        Yes, but ideally much simpler
- mongol 4y ago
  What scale do you have in mind for your solution? How many applications, how much resources would you comfortably run like this?
- rad_gruchalski 4y ago
  “You don’t need Kubernetes to run stateless apps, you can do it simpler by…” goes on and lists 20 things one needs to deploy, maintain and learn to manage. I don’t think your solution is simpler but you are right about one thing: one does not need Kubernetes, alternatives exist.
yjftsjthsd-h 4y ago
> CRIU needs to work; live migrating containers between hosts Wait, k8s can use CRIU to live migrate containers?
- metadat 4y ago
  Not commonly, I've never gotten CRIU working in real life. AFAIK it's not feasible unless you go all-in on a VMWare stack.

0xbadcafebee 4y ago

You don't need any of that stuff to run microservices. Stateless is up to the app, not the orchestrator (you can still run a stateful containerized app in K8s). You absolutely do not need containers, network overlay, "transparent encryption" (do you mean a service mesh?), name resolution (dnsmasq works fine), and access controls are inherent in systemd. Network storage is unrelated if you mean NFS; if you mean persistent storage volume claims shuffled around nodes, yeah, that's... not a stateless microservice. And.... no, we don't need CRIU, but it works just fine without K8s. All you need to run microservices is 1) a statically-compiled stateless app, 2) a systemd unit file to run the app and restart it when it stops, 3) load any credentials into systemd's credential manager, 4) iptables for network ACLs, 5) linux user/group permissions, 6) an Nginx load balancer to route traffic to different services, 7) journald to keep logs, 8) systemd to set per-service CPU/memory limits, 9) a dynamic DNS client to register a service name in dnsmasq when a service starts/stops. Systemd already handles intra-service network communication by handing an open filehandle to a service; alternately each service could grab its own random port and register it with Nginx along with its service name and the HTTP routes and hosts it wants to listen on. If you shackle yourself to the design of K8s, or some implementation of K8s (K8s doesn't have all the features you described out of the box) and say "well that other system isn't K8s!", then yeah, you will find systemd wanting.

chucky_z 4y ago
You could run Nomad and use parameterized jobs with raw_exec to get something like this. I know that systemd has some lifecycle stuff where you can create/run systemd units directly through the cli instead of using unit files which should allow you to hook the lifecycle into a nomad style service invocation but I can't recall exactly how to do this. This doesn't quite roll into the 'only systemd,' but it allows you to build exactly what you're looking for.
- 0xbadcafebee 4y ago
  I would rather systemd added a single feature to its already vast arsenal, and then Nomad wouldn't be needed either.
  - chucky_z 4y ago
    Figure out how to revive https://github.com/coreos/fleet https://github.com/coreos/fleet as something native in systemd?
    - 0xbadcafebee 4y ago
      Yes, but ideally much simpler
mongol 4y ago
What scale do you have in mind for your solution? How many applications, how much resources would you comfortably run like this?
rad_gruchalski 4y ago
“You don’t need Kubernetes to run stateless apps, you can do it simpler by…” goes on and lists 20 things one needs to deploy, maintain and learn to manage. I don’t think your solution is simpler but you are right about one thing: one does not need Kubernetes, alternatives exist.

yjftsjthsd-h 4y ago

> CRIU needs to work; live migrating containers between hosts Wait, k8s can use CRIU to live migrate containers?

metadat 4y ago
Not commonly, I've never gotten CRIU working in real life. AFAIK it's not feasible unless you go all-in on a VMWare stack.