4 ms·

This is an apt point. Kubernetes models Borg, and Borg has no concept of ingress. That's an entirely different problem space. Obviously that doesn't fly if t

This is an apt point. Kubernetes models Borg, and Borg has no concept of ingress. That's an entirely different problem space. Obviously that doesn't fly if there isn't an equivalent open solution, so we did what we could with the system to make it not terrible. We can do more. The point about Swarm is interesting, and has been much on my mind. Some of Kubernetes' perceived complexity is because we go to great lengths to avoid ever having two users collide, with escape hatches for the people who really need "unfriendly" features. This is because, again, Kubernetes models Borg. Borg clusters are giant, shared, multi-user, multi-app animals, where the users are in different business units and chances of collisions are high. Swarm, on the other hand, thinks of a cluster more as an application construct. Sharing is not a big problem, and coordination is easy and local. This allows them to make different tradeoffs. I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I still believe the large-shared-cluster model is right in the limit. There are so many efficiencies to be had. But there are legit reasons it is hard to achieve right now. I'm very interested in ways to make Kubernetes easier to use, ESPECIALLY in this regard. Real user feedback is critical.

sandGorgon 9y ago
that's an interesting perspective - however, Docker Swarm also does that. Docker Swarm secrets have been GA for longer thank k8s. The new UCP mechanism in the Docker Datacenter product is fairly interesting (has not made it to swarm yet). Its a paid product but makes RBAC pretty simple - https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Securing_Docker_EE_and_Security_Best_Practices https://success.docker.com/Architecture/Docker_Reference_Arc... . One very interesting tool that Docker makes available is https://store.docker.com/community/images/docker/docker-bench-security https://store.docker.com/community/images/docker/docker-benc... I think the issue with k8s is that it is competing with the "Ruby on Rails" of frameworks viz Docker Swarm. I think the pluggability of critical pieces like ingress and secrets was taken too far. > I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I dont think that is true, it does manage its overlay networks pretty well. Which is FWIW, another place where k8s took the non-opinionatedness too far. I think the number of bugs on "my stuff doesnt work with flannel but works with calico" should tell you that. To be honest, Docker Swarm has some of these issues as well - https://github.com/moby/moby/issues/25526 https://github.com/moby/moby/issues/25526. But the fixes are included in the "batteries". On kuberenetes, I have to run behind upstream projects with heterogenous configuration (nginx vs haproxy ingress. or flannel vs calico configuration) to try and fix it.
- thockingoog 9y ago
  > Docker Swarm secrets have been GA for longer thank k8s I don't think that's true. Kube secrets were introduced 2015-02-17 and was considered GA in Kubernetes v1.0 > I think the pluggability of critical pieces like ingress and secrets was taken too far. I think the pluggability is not the concern but the lack of an included solution. Part of the problem is that SOME platforms have an included solution - e.g. Google Cloud, and some need 3rd party code like nginx. > I dont think that is true, it does manage its overlay networks pretty well Overlays are a waste for most people. I get that making it simple is attractive, but it's (IMO) not something everyone wants or needs. Again, we could/should have had a built-in option. Last I looked (admittedly a while ago) Swarm had a pretty deeply rooted notion of exposing ports on all nodes in the swarm, which means that if you have multiple containers that need to expose the same port, it was a problem. Kube takes extra complexity here, to make it possible to share arbitrarily. Anyway, it's not my intent to bad-mouth Swarm or try to convince you that you're wrong. Different trade-offs were chosen for the two systems. Your feedback is noted and appreciated. :)

sandGorgon 9y ago

that's an interesting perspective - however, Docker Swarm also does that. Docker Swarm secrets have been GA for longer thank k8s. The new UCP mechanism in the Docker Datacenter product is fairly interesting (has not made it to swarm yet). Its a paid product but makes RBAC pretty simple - https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Securing_Docker_EE_and_Security_Best_Practices https://success.docker.com/Architecture/Docker_Reference_Arc... . One very interesting tool that Docker makes available is https://store.docker.com/community/images/docker/docker-bench-security https://store.docker.com/community/images/docker/docker-benc... I think the issue with k8s is that it is competing with the "Ruby on Rails" of frameworks viz Docker Swarm. I think the pluggability of critical pieces like ingress and secrets was taken too far. > I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I dont think that is true, it does manage its overlay networks pretty well. Which is FWIW, another place where k8s took the non-opinionatedness too far. I think the number of bugs on "my stuff doesnt work with flannel but works with calico" should tell you that. To be honest, Docker Swarm has some of these issues as well - https://github.com/moby/moby/issues/25526 https://github.com/moby/moby/issues/25526. But the fixes are included in the "batteries". On kuberenetes, I have to run behind upstream projects with heterogenous configuration (nginx vs haproxy ingress. or flannel vs calico configuration) to try and fix it.

thockingoog 9y ago
> Docker Swarm secrets have been GA for longer thank k8s I don't think that's true. Kube secrets were introduced 2015-02-17 and was considered GA in Kubernetes v1.0 > I think the pluggability of critical pieces like ingress and secrets was taken too far. I think the pluggability is not the concern but the lack of an included solution. Part of the problem is that SOME platforms have an included solution - e.g. Google Cloud, and some need 3rd party code like nginx. > I dont think that is true, it does manage its overlay networks pretty well Overlays are a waste for most people. I get that making it simple is attractive, but it's (IMO) not something everyone wants or needs. Again, we could/should have had a built-in option. Last I looked (admittedly a while ago) Swarm had a pretty deeply rooted notion of exposing ports on all nodes in the swarm, which means that if you have multiple containers that need to expose the same port, it was a problem. Kube takes extra complexity here, to make it possible to share arbitrarily. Anyway, it's not my intent to bad-mouth Swarm or try to convince you that you're wrong. Different trade-offs were chosen for the two systems. Your feedback is noted and appreciated. :)