8 ms·

Writing a Kubernetes CRD Controller in Rust

166 points by lukastyrychtr 6y ago

praveenperera 6y ago
The library used in the this post has gotten much better since this post was written (in 2019) Recently the ability to do `exec` and `attach` was added. It should reach feature parity with the Go version soon once `port forward` is added. https://github.com/clux/kube-rs https://github.com/clux/kube-rs
- ircoder 6y ago
  Hmm! Interesting. I'm wondering if adding a test framework like kubebuilder is on the agenda. I'm not a great fan of it (kubebuilder) but the simple-to-use (in a relative term, i.e. as simple as it can get with Go) is a huge benefit. So you won't have to deploy after every build to see how your changes look like.
  - praveenperera 6y ago
    I'm not sure about that, but the maintainers are very active and receptive so maybe make an issue? Better yet a PR.
  - Nullabillity 6y ago
    We don't currently support anything like Kubebuilder's envtest helper, but it is on our radar (and I finally got around to making an issue for it[0]). The rest of Kubebuilder's testing page seems to be general for Go's test framework (in which case Rust should have native equivalents). Disclaimer: Co-maintainer of kube-rs [0]: https://github.com/clux/kube-rs/issues/382 https://github.com/clux/kube-rs/issues/382
- Nullabillity 6y ago
  Apart from that, the runtime module has also been overhauled completely since this blog post. In particular, the new `Controller`[0] adds a bunch of stuff over `Informer` to make it usable in production (merging duplicate queued events, parallel execution, delayed retries, fixed a bunch of internal race conditions, etc). We really need to make a 2020 edition of this blog post, but we do at least have an updated example controller.[1] (Not to dunk on TechnoSophos, none of that existed back when the blog post was written.) Disclaimer: Co-maintainer of kube-rs [0]: https://docs.rs/kube-runtime/0.47.0/kube_runtime/controller/struct.Controller.html https://docs.rs/kube-runtime/0.47.0/kube_runtime/controller/... [1]: https://github.com/clux/kube-rs/blob/8611e5cdd23d4ff487fdcea090e22343d57e4e56/examples/configmapgen_controller.rs https://github.com/clux/kube-rs/blob/8611e5cdd23d4ff487fdcea...
sanxiyn 6y ago
This is (2019).
parhamn 6y ago
How has Kubernetes being a giant reactive resource API panned out? My first reaction was to like it, but near the end of when I was very invested in the space it started feeling like there were too many moving parts and a hectic mess. Stacks became more difficult to reproduce (controllers, their versions and platform limitations) and people moved too many things into custom resources. Did the custom resource controllers pan out well?
- eecc 6y ago
  Don’t know but it’s a shame that the k8s API doesn’t have link fields... it kind of breaks the whole concept of automatic interfacing
  - BenTheElder 6y ago
    Can you elaborate on what you mean by link field? I've searched a bit but I'm not certain what you're referring to in this context.
    - Pirate-of-SV 6y ago
      HATEOAS I suppose. https://restfulapi.net/hateoas/ https://restfulapi.net/hateoas/
- jsabo 6y ago
  I'm not sure what you mean by too many things ended up as custom resources -- it's the recommended extension point and you can do a lot with them that you can't do with core types. Custom resources had/have some growing pains but I think worked out pretty well. It can be very very hard to test and distribute them though. As someone who maintained a controller with paid support, your test matrix gets pretty large pretty fast accounting for different k8s versions, different hosted versions (GKE, AKS, etc), different distros (openshift, rancher, etc). And that's before you even get into specific configurations like pod security policies, can the control plane communicate with the data plane, is there a service mesh. Resource versioning is hard to get right. Once a resource type is v1, it becomes difficult to extend it. You can't add a beta field to it easily. Revising schema can be hard since anything more than a no op conversion between versions requires a webhook, which requires a certificate chain, and while cert-manager is popular it is not ubiquitous and regularly has breaking changes. Webhook setup issues made up a large portion of our support requests. As far as the general "reconciliation loop" architecture goes, you end up with something similar in most orchestration systems I've worked on, or you wish that you did. So overall I think that worked out well. Getting it right can be hard, but I think that's the nature of the beast.
Thaxll 6y ago
"The Go version was over 1700 lines long and was loaded with boilerplate and auto-generated code. The Rust version was only 127 lines long" Right ...
- jacques_chester 6y ago
  I've written CRD controllers. This is an accurate description of what your code looks like. Because of Golang's anemic type system, massive code generation is the only viable way to deal with the dynamic blackboard that the API Server presents to clients.
  - Thaxll 6y ago
    I also write CRD, but I don't beleive for one second that you have more than 10x less code, especially for generated code that you don't touch at all. It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that.
    - jacques_chester 6y ago
      It's a different language with a different type system. I've also interacted with the API Server using a Ruby library that relied on metaprogramming. I could do things in a dozen lines that would require thousands of lines of checked-in Go. Replying to your edit: > It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that. Well, a few things. First, using client-go doesn't save you from version shifts. It is explicitly unsupported, so if they break your downstream system, stiff shit. Second, client-go is derived from code generators. The same generators also produce other clients. Unfortunately they are terrible, by virtue of OpenAPI generator being a hellish maze and because they leak Go-isms like a spiteful sieve. I've used that code generator to develop a proxy. It sucked. Third, this library is derived independently. It uses the same OpenAPI definitions as client-go, client-java and so forth. It's as easy to keep up to date as any of the others. Fourth, the Kubernetes API isn't fully tested. Not even close. Many libraries can rightfully claim to be just as conformant as client-go, despite being broken on ~66% of endpoints: https://apisnoop.cncf.io/ https://apisnoop.cncf.io/
hardwaresofton 6y ago
Go bills itself as simple, but Rust, despite it's complexity (which is hidden from you with libraries like serde and others) is simpler to use at the higher level. Rust has a near-best-of-class type system, better abstractions, and runs on everything from embedded to the browser. Go has captured the hearts of sysadmin types and those looking for something just a step up from python (as evidenced by much of the devops code written in it -- which I'm very grateful for, I don't think I'd use kubernetes if it was in Java still), so I can see it growing in that field. I don't see how Rust doesn't become the most important emerging systems language (C & C++ obviously aren't going anywhere just by virtue of being incumbents) of the next decade.
- throwaway894345 6y ago
  Because building things quickly is important and Rust makes you think a lot about memory management. This is a fine thing for performance critical systems, but for systems where Python’s performance could be described as adequate, the extra performance Rust offers is immaterial. Of course, Rust gives you a bit more type safety than Go as well, but still not enough to justify trading off so much development velocity (or rather, type safety and correctness simply aren’t nearly as important as velocity from an economic perspective). Rust is getting better all the time—non-lexical lifetimes and rust-analyzer have provided a surprising improvement in development velocity, but Go is still in another class. Personally I think Go is the best language for developing quickly—even better than Python for non-toy projects (I say that as one with 15 years of Python experience and almost a decade of ago experience).
  - praveenperera 6y ago
    I really don't think Rust MAKES you think a lot about memory management. You CAN think a lot about memory management if you care about performance. If not, you can throw `clone()` around everywhere and still have a very fast program.
    - throwaway89456 6y ago
      It is not because you can that you should.
deleted 6y ago
[deleted]

166 points by lukastyrychtr 6y ago

The library used in the this post has gotten much better since this post was written (in 2019) Recently the ability to do `exec` and `attach` was added. It should reach feature parity with the Go version soon once `port forward` is added. https://github.com/clux/kube-rs https://github.com/clux/kube-rs

ircoder 6y ago
Hmm! Interesting. I'm wondering if adding a test framework like kubebuilder is on the agenda. I'm not a great fan of it (kubebuilder) but the simple-to-use (in a relative term, i.e. as simple as it can get with Go) is a huge benefit. So you won't have to deploy after every build to see how your changes look like.
- praveenperera 6y ago
  I'm not sure about that, but the maintainers are very active and receptive so maybe make an issue? Better yet a PR.
- Nullabillity 6y ago
  We don't currently support anything like Kubebuilder's envtest helper, but it is on our radar (and I finally got around to making an issue for it[0]). The rest of Kubebuilder's testing page seems to be general for Go's test framework (in which case Rust should have native equivalents). Disclaimer: Co-maintainer of kube-rs [0]: https://github.com/clux/kube-rs/issues/382 https://github.com/clux/kube-rs/issues/382
Nullabillity 6y ago
Apart from that, the runtime module has also been overhauled completely since this blog post. In particular, the new `Controller`[0] adds a bunch of stuff over `Informer` to make it usable in production (merging duplicate queued events, parallel execution, delayed retries, fixed a bunch of internal race conditions, etc). We really need to make a 2020 edition of this blog post, but we do at least have an updated example controller.[1] (Not to dunk on TechnoSophos, none of that existed back when the blog post was written.) Disclaimer: Co-maintainer of kube-rs [0]: https://docs.rs/kube-runtime/0.47.0/kube_runtime/controller/struct.Controller.html https://docs.rs/kube-runtime/0.47.0/kube_runtime/controller/... [1]: https://github.com/clux/kube-rs/blob/8611e5cdd23d4ff487fdcea090e22343d57e4e56/examples/configmapgen_controller.rs https://github.com/clux/kube-rs/blob/8611e5cdd23d4ff487fdcea...

sanxiyn 6y ago

This is (2019).

parhamn 6y ago

How has Kubernetes being a giant reactive resource API panned out? My first reaction was to like it, but near the end of when I was very invested in the space it started feeling like there were too many moving parts and a hectic mess. Stacks became more difficult to reproduce (controllers, their versions and platform limitations) and people moved too many things into custom resources. Did the custom resource controllers pan out well?

eecc 6y ago
Don’t know but it’s a shame that the k8s API doesn’t have link fields... it kind of breaks the whole concept of automatic interfacing
- BenTheElder 6y ago
  Can you elaborate on what you mean by link field? I've searched a bit but I'm not certain what you're referring to in this context.
  - Pirate-of-SV 6y ago
    HATEOAS I suppose. https://restfulapi.net/hateoas/ https://restfulapi.net/hateoas/
jsabo 6y ago
I'm not sure what you mean by too many things ended up as custom resources -- it's the recommended extension point and you can do a lot with them that you can't do with core types. Custom resources had/have some growing pains but I think worked out pretty well. It can be very very hard to test and distribute them though. As someone who maintained a controller with paid support, your test matrix gets pretty large pretty fast accounting for different k8s versions, different hosted versions (GKE, AKS, etc), different distros (openshift, rancher, etc). And that's before you even get into specific configurations like pod security policies, can the control plane communicate with the data plane, is there a service mesh. Resource versioning is hard to get right. Once a resource type is v1, it becomes difficult to extend it. You can't add a beta field to it easily. Revising schema can be hard since anything more than a no op conversion between versions requires a webhook, which requires a certificate chain, and while cert-manager is popular it is not ubiquitous and regularly has breaking changes. Webhook setup issues made up a large portion of our support requests. As far as the general "reconciliation loop" architecture goes, you end up with something similar in most orchestration systems I've worked on, or you wish that you did. So overall I think that worked out well. Getting it right can be hard, but I think that's the nature of the beast.

Thaxll 6y ago

"The Go version was over 1700 lines long and was loaded with boilerplate and auto-generated code. The Rust version was only 127 lines long" Right ...

jacques_chester 6y ago
I've written CRD controllers. This is an accurate description of what your code looks like. Because of Golang's anemic type system, massive code generation is the only viable way to deal with the dynamic blackboard that the API Server presents to clients.
- Thaxll 6y ago
  I also write CRD, but I don't beleive for one second that you have more than 10x less code, especially for generated code that you don't touch at all. It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that.
  - jacques_chester 6y ago
    It's a different language with a different type system. I've also interacted with the API Server using a Ruby library that relied on metaprogramming. I could do things in a dozen lines that would require thousands of lines of checked-in Go. Replying to your edit: > It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that. Well, a few things. First, using client-go doesn't save you from version shifts. It is explicitly unsupported, so if they break your downstream system, stiff shit. Second, client-go is derived from code generators. The same generators also produce other clients. Unfortunately they are terrible, by virtue of OpenAPI generator being a hellish maze and because they leak Go-isms like a spiteful sieve. I've used that code generator to develop a proxy. It sucked. Third, this library is derived independently. It uses the same OpenAPI definitions as client-go, client-java and so forth. It's as easy to keep up to date as any of the others. Fourth, the Kubernetes API isn't fully tested. Not even close. Many libraries can rightfully claim to be just as conformant as client-go, despite being broken on ~66% of endpoints: https://apisnoop.cncf.io/ https://apisnoop.cncf.io/

hardwaresofton 6y ago

Go bills itself as simple, but Rust, despite it's complexity (which is hidden from you with libraries like serde and others) is simpler to use at the higher level. Rust has a near-best-of-class type system, better abstractions, and runs on everything from embedded to the browser. Go has captured the hearts of sysadmin types and those looking for something just a step up from python (as evidenced by much of the devops code written in it -- which I'm very grateful for, I don't think I'd use kubernetes if it was in Java still), so I can see it growing in that field. I don't see how Rust doesn't become the most important emerging systems language (C & C++ obviously aren't going anywhere just by virtue of being incumbents) of the next decade.

throwaway894345 6y ago
Because building things quickly is important and Rust makes you think a lot about memory management. This is a fine thing for performance critical systems, but for systems where Python’s performance could be described as adequate, the extra performance Rust offers is immaterial. Of course, Rust gives you a bit more type safety than Go as well, but still not enough to justify trading off so much development velocity (or rather, type safety and correctness simply aren’t nearly as important as velocity from an economic perspective). Rust is getting better all the time—non-lexical lifetimes and rust-analyzer have provided a surprising improvement in development velocity, but Go is still in another class. Personally I think Go is the best language for developing quickly—even better than Python for non-toy projects (I say that as one with 15 years of Python experience and almost a decade of ago experience).
- praveenperera 6y ago
  I really don't think Rust MAKES you think a lot about memory management. You CAN think a lot about memory management if you care about performance. If not, you can throw `clone()` around everywhere and still have a very fast program.
  - throwaway89456 6y ago
    It is not because you can that you should.

deleted 6y ago

[deleted]