4 ms·
On the one hand, great. The other hand, one of the main criticisms of Kubernetes is that it has no composition or orchestration capabilities. It's great about
by jauntywundrkind 3y ago
On the one hand, great.
The other hand, one of the main criticisms of Kubernetes is that it has no composition or orchestration capabilities. It's great about defining pieces of state, but managing blocks of state & multiple things at once is left almost entirely to external tools.
The ability to compose &sequence multiple containers feels like a very specific example of a much broader general capability. There's bedevilling infinite complexity to trying to figure out a fully expressive state of state management system - I get why refining a couple specialized existing capabilities is the way - but it does make me a little sad to see a lack of appetite for the broader crosscutting system problem at the root here.
- penciltwirler 3y agoThere's lots of tools built on top of K8s to accomplish this tho. For example, Argo, Tekton, Flyte etc.
- jauntywundrkind 3y agoAbsolutely, no shortage of things atop. Helm is probably the most well used composition tool. It seems unideal to me to forever bunt on this topic, leaving it out of core forever. Especially when we are slowly adding im very specialized composition orchestration tools in core.
- numbsafari 3y agoHelm really solves a different use case than this. This is about describing the desired coordination among running containers. Helm is about how you template or generate your declarative state. You could certainly add this description to your templates with Helm, but you couldn't actually implement this feature with Helm itself.
- jauntywundrkind 3y agoI bundled both composition & orchestration under the same header. It so happens that pods have multiple containers, which is another example of Kubernetes having a specialized specific composition or orchestration implementation. One that started as composition, and here iterates towards orchestration.
- birdyrooster 3y agoRequiring the user to write their own operators to manage state using the kubernetes api is very much a feature and not something which is missing.
- smarterclayton 3y agoAgreed that is a feature and not a bug. But! The one thing that custom orchestrators can’t do is easily get the benefit of kubelet isolation of containers and resource management. Part of slowly moving down this path is to allow those orchestrators to get isolation from the node without having to reimplement that isolation. But it will take some time.
- birdyrooster 3y agoOh I see orchestrating runtimes is quite different. Good points!
- NathanKP 3y agoYeah I work on the team that builds Amazon Elastic Container Service so I can't help but compare this implementation with how we solved this same problem in ECS. Inside of an ECS task you can add multiple containers and on each container you can specify two fields: `dependsOn` and `essential`. ECS automatically manages container startup order to respect the dependencies you have specified, and on shutdown it tears things down in reverse order. Instead of having multiple container types with different hardcoded behaviors there is one container type with flexible, configurable behavior. If you want to chain together 4 or 5 containers to start up one by one in a series you can do that. If you want to run two things in parallel and then once both of them have become healthy start a third you can do that. If you want a container to run to completion and then start a second container only if the first container had a zero exit code you can do that. The dependency tree can be as complex or as simple as you want it to be: "init containers" and "sidecar containers" are just nodes on the tree like any other container. In some places I love the Kubernetes design philosophy of more resource types, but in other aspects I prefer having fewer resource types that are just more configurable on a resource by resource basis.
- politelemon 3y agoAs a consumer, ECS + Fargate is my happy path. I appreciate the lack of complexity. Thanks.
- 0xbadcafebee 3y agoDitto. ECS/Fargate has always been the easiest, most flexible, most useful containerization solution. It's the one AWS service with the most value to containerized services, and the least appreciated.
- scarby2 3y agothere was a pretty big feature gulf between it and K8s when it first launched. I found myself wishing i had a number of kubernetes controllers initially (Jobs (with restart policies), Cronjobs, volume management etc). From what i've head they've made a great many quality of life improvements but as is often the case it can be hard to regain share when you've already lost people.
- orf 3y agoI like it this way to be honest. We needed to create a custom controller for Dask clusters consisting of a single scheduler, an auto-scaling set of nodes, an ingress and a myriad of secrets, configmaps and other resources. It wasn’t simple, but with meta controller[1] it was relatively easy to orchestrate the complex state transitions this single logical resource needed and to treat the whole thing as a single unit. I’m not saying Kubernetes can’t make simple patterns easier, but baking it into core leads to the classic “tragedy of the standard library” problem where it becomes hard to change that implementation. And the k8s ecosystem is definitely all about change. 1. https://metacontroller.github.io/metacontroller/intro.html https://metacontroller.github.io/metacontroller/intro.html
- hosh 3y agoCompositions of blocks of state may not end up with a more reliable software. Each of state management are controlled by independent processes that may interact with each other (example: horizontal pod autoscalers are not directly aware of cluster-autoscaler). The whole system is more like an ecology or a complex adaptive system than it is something you can reason directly with abstractions. In the Cynefin framework (https://en.wikipedia.org/wiki/Cynefin_framework https://en.wikipedia.org/wiki/Cynefin_framework), you can reason through "complicated" domains the way you are suggesting, but it will not work when working in the "complex" domain. And I think what Kubernetes help manage is in "complex" not "complicated" domain.
- theptip 3y agoThis is all true, and if you read the KEPs they were thinking about this. One camp was advocating for solving the problem of specifying the full dependency graph spec (of which sidecars are one case), another advocating for just solving the most needed case with a sidecar-specific solution to get a solution shipped. The latter was complicated by a desire to at least leave the door open for the former. Pragmatism won out, thankfully IMO. Edit to add: see this better description from one of the senior k8s maintainers: https://news.ycombinator.com/item?id=36666359 https://news.ycombinator.com/item?id=36666359
- 0xbadcafebee 3y agoOrchestration of k8s wouldn't be necessary if they had made K8s' operation immutable. As it stands now you just throw some random YAML at it and hope for the best. When that stops working, you can't just revert back to the old working version, you have to start throwing more crap at it and running various operations to "fix" the state. So you end up with all these tools that are effectively configuration management tools to continuously "fix" the cluster back to where you want it. I hope the irony is lost on no one that this is an orchestration tool for an immutable technology, and the orchestrator isn't immutable.
- ed_mercer 3y agoYou can use gitops (eg fluxcd) to revert to previous cluster states.
- 0xbadcafebee 3y agoIf you wanted to do the opposite of what I'm saying, sure