3 ms·
I would love to understand the design rational on why a custom controller is needed to run etcd as opposed to leveraging the existing k8s constructs such as rep
by darren0 10y ago
I would love to understand the design rational on why a custom controller is needed to run etcd as opposed to leveraging the existing k8s constructs such as replicasets or petsets. While this is a very useful piece of technology it gives me the wrong impression that if you want to run a persistent or "more complicated" work load then you must develop a significant amount of code for that to work on k8s. Which I don't believe is the case, which I why I'm asking why this route was chosen.
- theptip 10y agoThe FAQ at the end of the OP addresses this: "Q: How is this different than StatefulSets (previously PetSets)? A: StatefulSets are designed to enable support in Kubernetes for applications that require the cluster to give them "stateful resources" like static IPs and storage. Applications that need this more stateful deployment model still need Operator automation to alert and act on failure, backup, or reconfigure. So, an Operator for applications needing these deployment properties and could use StatefulSets instead of leveraging ReplicaSets or Deployments." There is inevitably some app-specific logic required to modify a complex stateful deployment; the Operator encapsulates this logic so that the external interface is a simple config file.
- philips 10y agoDarren, the FAQ is at the overview post here: https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html
- darren0 10y agoThanks, but this alludes to more operators coming. Why is such a thing really needed. The proliferation of this approach seems like a potential downfall of k8s. Similar to Mesos where a framework is quite powerful, the cost of developing one is too high. This blog post basically implies that k8s base constructs can't run postgres, redis, prometheus, etcd, cassandra, etc. But why? Are we saying that stateful services fundamentally require one off domain specific logic to run in k8s?
- philips 10y agoI don't think stateful applications require the use of something like an Operator. It really just comes down to where the state lives. For example if you are wanting to run your database on top of an EBS, SAN, or something like that it is no problems to just throw up a StatefulSet and go for it. However, if you start to think about things like orchestrating scaling of databases that have a administrative tool like Cassandra, Vitess, RethinkDB, read-replicated Postgres, etc you need to have some glue/gel that integrates those admin tools with the Kubernetes APIs. And that is what a minimal Operator should do. The other thing Operators can do is glue existing software to the Kubernetes APIs. Which is what the Prometheus Operator does. Prometheus has its own configuration system for finding monitoring targets; instead of forcing the user to drop down into that different format the Operator adapts Kubernetes concepts like label queries and generates the equivalent Prometheus config. Overall I don't think this is required for every application. Static databases persisting to shared storage, stateless applications, or cluster wide daemons all fit nicely in Kubernetes abstractions before Operators. But, there is a class of clustered applications that are served well by this pattern.
- tristanz 10y agoI think philips is exactly right. The whole design of Kubernetes is geared toward allowing users write their own controllers for advanced use cases. You could view something like Jenkins or Vitess as controllers because they spawn Kubernetes pods on demand. The beauty is Kubernetes gives you great primitives, so you often will be controlling these objects, not the underlying pods. Of course many simple applications don't need a controller, although I suspect more and more simple use cases will be managed by an external controller like Helm which orchestrates the lifecycle of applications. The questions is: Why isn't this just called a controller? What's this new term Operator?
- philips 10y agoController wasn't quite the term to capture the combination of an application specific controller and third party resource to manage a collection of user created application instances. So, we arrived at Operator. It felt like a good term that we could put after X that encapsulates the intent of the pattern. It helps you operate instances of an application.
- Terretta 10y ago"An Operator represents human operational knowledge in software, to reliably manage an application." Loving this.
- SEJeff 10y agoDo you ever see this pattern being used to host/run the etcd cluster for the k8s apiserver to speak with? That would be quite meta (and kind of amazing if it would actually work).