8 ms·
Writing a Kubernetes Operator
- _muff1nman_ 4y agoThis article is mistaken from the get-go as an operator is not the same as an apiservice. Rather an operator is a wider term for something that includes a controller. See https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat... Also it's important for people reading this article - an apiservice (which this article talks about) is very rarely something that should be done. An operator is more appropriate for nearly all cases except for when you truly need your state stored outside of the internal Kubernetes etcd datastore.
- reedjosh 4y agoCustom Resource + Controller = Operator. Good call! > Operators are clients of the Kubernetes API that act as controllers for a Custom Resource.
- jhoelzel 4y agoexactly! controlling refers to directing or regulating the behavior of something, while operating refers to the actual execution or manipulation.
- tenac23 4y agoAfter reading the comments we updated the article
- darren0 4y agoI’m not sure why this is a top post. The definitions of controller and operator are completely wrong. The example code is for creating a custom api server which is only done in the most advanced of advanced use cases. The implementation of the apiserver is too naive to demonstrate they have any understanding of the complexity that supporting watch will cause.
- timelapse 4y ago> The definitions of controller and operator are completely wrong. mind clarifying?
- mfer 4y agoThe article has a description of what an operator is wrong. The definition of an operator originally was... > An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate common tasks. This is the original definition of an operator [1]. People no use them for stateless things and domain specific work has taken off. You can look at the Kubernetes docs [2] to see refinements on it... > Kubernetes' operator pattern concept lets you extend the cluster's behaviour without modifying the code of Kubernetes itself by linking controllers to one or more custom resources. Operators are clients of the Kubernetes API that act as controllers for a Custom Resource. [1] https://web.archive.org/web/20190113035722/https://coreos.com/blog/introducing-operators.html https://web.archive.org/web/20190113035722/https://coreos.co... [2] https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...
- richardwhiuk 4y agoYou don't need to implement a custom API server to implement an operator - you can just watch a CR.
- jhoelzel 4y agofor an operator you do, what you mean is a controller =)
- richardwhiuk 4y agoNo, you don't. You can have a CR, and implement an Operator which just watches the CR and acts upon it.
- fedreg 4y agoHere's another example of a custom rust operator, https://github.com/mach-kernel/databricks-kube-operator https://github.com/mach-kernel/databricks-kube-operator Written by a co-worker to help manage our databricks projects across clusters. Works wonderfully!!
- alexott 4y agoBut why such complexity? Is it easier to maintain than terraform code?
- EdwardDiego 4y agoYes. Terraform doesn't actively manage resources, opererators do.
- jzelinskie 4y agoSince Go got generics, working with the Kubernetes API could become far more ergonomic. It's been pulling teeth until now. I'm eager to see how the upstream APIs change over time. In the mean time, one of the creators of the Operator Framework[0] built a bunch of useful patterns using generics that we used to build the SpiceDB Operator[1] called controller-idioms[2]. Does anyone know of other efforts to improve the status quo? [0]: https://operatorframework.io https://operatorframework.io [1]: https://github.com/authzed/spicedb-operator https://github.com/authzed/spicedb-operator [2]: https://github.com/authzed/controller-idioms https://github.com/authzed/controller-idioms
- aliasxneo 4y agoThis was part of why I chose Python with the last operator I wrote. Granted, it was a slow process that wasn't time-sensitive (and couldn't be paralleled), so performance wasn't a concern. Regardless, I avoided Go because of previous experiences with how hard a lack of generics made working with such things. I'm hoping things are improving. Maybe I'll consider it for the next one I write.
- debarshri 4y agoA better way to write an operator these days is to use kubebuilder [1]. My complaint is that I have seen orgs write operators for random stuff, often reinventing the wheel. Lot of operators in orgs are result of resume driven development. Having said that it often comes handy for complex orchestration. [1] https://github.com/kubernetes-sigs/kubebuilder https://github.com/kubernetes-sigs/kubebuilder
- casperc 4y agoWhat would be a good example where an operator would make sense?
- jhoelzel 4y ago- creating databases for your app on the fly. - scaling up and down applications because of time instead of demand. or based on non metric based actions - Extending kubernetes to understand your workload - Automating configuration and management of complex applications - Managing legacy applications that cannot be easily containerized or migrated to the cloud. if you love k8s youll love operators the list is endless!
- dilyevsky 4y agoWith respect, being “in love” with a technology is not a good way to go about it - it leads to tunnel vision
- jhoelzel 4y agoi know what you mean, but i am doing this for more than 20 years now. From bare-metal over openstack to serverless, i have pretty much provisioned all of them. Kubernetes is more like a way of doing things than a technology. Basically APIs all the way down. and thus the operators and controllers do deserve love. Im not saying that you need an operator to change the dipers of a baby, but as far as a stack goes, k8s is the best i have ever worked with.
- 4y ago
- jhoelzel 4y agoOh i love operators they usually tie the entire cluster together and lead to amazing things! Think of Kubernetes as an advanced API server that can be extended endlessly and operators are the way to do it. There really is no magic, is all there and with go the images are usually what? like 10 mb? It's essential to have a solid understanding of Kubernetes architecture, concepts such as custom resources and controllers, and the tools and APIs available for working with Operators. Dont use rust though, use and sdk like the operator sdk or kubebuilder. Its native to k8s and you will have a much easier time too.
- kimbernator 4y agoI didn't really enjoy my experience with the few operators I've worked with, mainly because they require the maintainer to build in some sort of access to basic kubernetes functionality. I see the benefit of operators, but I hated that in order to do something as simple as define memory/CPU limits to certain containers I would need to open a PR to the repo and wait weeks, sometimes months, for a new release. It's frustrating to be a kubernetes admin but not have access to basic configuration options because the maintainers of even some very high-profile operators (looking at you, AWX) neglected to build in access to basic functionality.
- evancordell 4y agoThis is a common frustration of mine as well! In the latest release of the spicedb-operator[0], I added a feature that allows users to specify arbitrary patches over operator-managed resources directly in the API (examples in the link). There are some other projects like Kyverno and Gatekeeper that try to do this generically with mutating webhooks, but embedding a `patches` API into the operator itself gives the operator a chance to ensure the changes are within some reasonable guardrails. [0]: https://github.com/authzed/spicedb-operator/releases/tag/v1.2.0 https://github.com/authzed/spicedb-operator/releases/tag/v1....
- Thaxll 4y agoUsing Rust for that is a bad idea, just use the official and native SDKs ( in Go ). Rust does not have any equivalent to https://sdk.operatorframework.io/ https://sdk.operatorframework.io/
- devkulkarni 4y agoWe have an FAQ about Operators here: https://github.com/cloud-ark/kubeplus/blob/master/Operator-FAQ.md https://github.com/cloud-ark/kubeplus/blob/master/Operator-F... It should be helpful if you are new to the Operator concept. Operators are generally useful for handling domain-specific actions - for example, performing database backups, installing plugins on Moodle/Wordpress, etc. If you are looking for application deployment then a Helm chart should be sufficient.
- crabbone 4y agoI've written (well, participated in development of) two Kubernetes operators, and support about a dozen of them (in our own deployment of Kubernetes): Jupyter, PostgreSQL, a bunch of Prometheus operators and a handful of proprietary ones. In my years of working with Kubernetes I cannot shake the feeling that it's, basically, an MLM. It carefully obscures it's functionality by hiding behind opaque definitions. It doesn't really work, when push comes to shove. And, most importantly, it survives in a parasitic kind of way: by piggybacking on those who develop all kinds of extensions, be it operators, custom networking or storage plugins, authentication and so on. My problem is I cannot find who stands at the top of the pyramid. There's Cloudnative Foundation, but all it does is selling certifications nobody really needs... so, that cannot possibly be it. No big name doesn't really benefit from this in an obvious way... So... anyways, when I hear people argue about how to implement this or another extension of Kubernetes, it rings the same as when people argue about styles of agile, or code readability etc. nonsense. There isn't a good way. There is not acceptance criteria. The whole system is flawed to no end.
- rdtsc 4y agoYou have a problem: orchestrating some thing in kube, so you write some custom operator logic running alongside your main product; but now you have two problems to worry about. I've seen just as much if not more issues with debugging the operator logic itself as with the main pods/deployments it was trying to manage. So just from a practical point of view, I think it should be a last resort after everything else fails (helm charts, etc).
- sigwinch28 4y agoI find myself conflicted between two approaches at work: 1. Write a provider/extension/whatever for a tool like Terraform or Pulumi. I live in a world where the infrastructure doesn't move underneath my feet. I am the source of truth. I feel like I only need to reconcile changes when I make changes to my IaC repositories. 2. I could write something that exists in a control plane, like Kubernetes operators or Crossplane. I live in a world where I look at the world, find the delta between current state and desired state, then try to reconcile. This is an endless loop. I feel like these are different approaches with the same goal. Why should I decide either way beyond tossing a coin? Some use cases: - an internal enterprise DNS system which is not standards-compliant with the world at large - an internal certificate authority and certificate issuing system.
- aliasxneo 4y agoTerraform doesn't run a control loop. If you've never been in a situation where you've needed a control loop to reconcile the deployment state, then you've probably never needed an operator. In most cases, the native controllers (and resources) from Kubernetes are enough, but sometimes, an application requires additional hand-holding. I find this to be especially true for legacy applications.
- oso2k 4y agoBesides the frameworks TFA lists, you can also build Operators in Ansible or Helm. https://sdk.operatorframework.io/ https://sdk.operatorframework.io/