4 ms·

How has Kubernetes being a giant reactive resource API panned out? My first reaction was to like it, but near the end of when I was very invested in the space i

How has Kubernetes being a giant reactive resource API panned out? My first reaction was to like it, but near the end of when I was very invested in the space it started feeling like there were too many moving parts and a hectic mess. Stacks became more difficult to reproduce (controllers, their versions and platform limitations) and people moved too many things into custom resources. Did the custom resource controllers pan out well?

eecc 6y ago
Don’t know but it’s a shame that the k8s API doesn’t have link fields... it kind of breaks the whole concept of automatic interfacing
- BenTheElder 6y ago
  Can you elaborate on what you mean by link field? I've searched a bit but I'm not certain what you're referring to in this context.
  - Pirate-of-SV 6y ago
    HATEOAS I suppose. https://restfulapi.net/hateoas/ https://restfulapi.net/hateoas/
jsabo 6y ago
I'm not sure what you mean by too many things ended up as custom resources -- it's the recommended extension point and you can do a lot with them that you can't do with core types. Custom resources had/have some growing pains but I think worked out pretty well. It can be very very hard to test and distribute them though. As someone who maintained a controller with paid support, your test matrix gets pretty large pretty fast accounting for different k8s versions, different hosted versions (GKE, AKS, etc), different distros (openshift, rancher, etc). And that's before you even get into specific configurations like pod security policies, can the control plane communicate with the data plane, is there a service mesh. Resource versioning is hard to get right. Once a resource type is v1, it becomes difficult to extend it. You can't add a beta field to it easily. Revising schema can be hard since anything more than a no op conversion between versions requires a webhook, which requires a certificate chain, and while cert-manager is popular it is not ubiquitous and regularly has breaking changes. Webhook setup issues made up a large portion of our support requests. As far as the general "reconciliation loop" architecture goes, you end up with something similar in most orchestration systems I've worked on, or you wish that you did. So overall I think that worked out well. Getting it right can be hard, but I think that's the nature of the beast.
- dhruvt10 6y ago
  Really interested to know about the challenges you faced in revisioning but couldn't find the exact thing, do you happen to know a source for understanding it more?
  - kitd 6y ago
    Not OP but the feature you're looking for is OLM https://olm.operatorframework.io/docs/ https://olm.operatorframework.io/docs/
Thaxll 6y ago
Well look how popular Kubernetes is, how many thousands of things built on top of it because its powerful and extensible API model.
jacques_chester 6y ago
> How has Kubernetes being a giant reactive resource API panned out? It was never reactive, at least in the sense of the Reactive Manifesto. There is no system of backpressure. > Did the custom resource controllers pan out well? I think it will prove to be useful for a handful of uses, but that in general, it will wind up like Wordpress and Jenkins plugins. Powerful, popular, but a guaranteed mess. Kubernetes was not originally designed to accommodate CRDs. There's no concept of tenancy, only things you can cobble together into tenancy-ish-alike-kinda shapes. There's RBAC, but it was designed before CRDs and means that poorly designed custom controllers can be attack vectors. I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws. Unfortunately, I don't have the wits to join into the multi-billion dollar market of "papering over things Kubernetes doesn't do very well but which can't now be changed". [0] I mean, I wrote a book about Knative, which is a purely CRD-based architecture.
- bogomipz 6y ago
  >"I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws." Could you elaborate on the "design flaws" part? Do you feel that the CRD model itself has design flaws or are you referring to specific project's CRDs?
  - psoundy 6y ago
    tl;dr If you’re operating with CRDs at trivial scale, you probably having nothing to worry about. But operating with CRDs at scale is a different story and suggests careful testing with the specific applications involved. ——- The usage patterns of native k8s types and the implications those patterns have on the scalability and reliability of etcd and the apiserver are relatively well-understood. CRDs can be a wild-card, though, and afaik testing efforts thus far have not investigated worst-case usage of CRD-based applications. As commonly deployed, CRDs are served from the same apiservers and etcd cluster that serves the native types for a k8s cluster. That can result in contention between the CRDs supporting 3rd party additions to a cluster and the native types critical to the health of a cluster. This kind of contention has the potential to bring a cluster to its knees. Efforts like priority and fairness seek to ensure that the apiserver can prioritize at the level of the API call. But that won’t prevent watch caches from OOM’ing the apiserver if excessive numbers of CRDs are present. The judicious use of quotas could head off the creation of an excessive number of objects, but it’s not just count that matters - the size of each resource is also a factor. In theory, CRDs could be isolated from native types by serving them from an aggregated apiserver backed by a separate etcd cluster. afaik this not a supported configuration today, and even if it were the additional resources required to support it (especially the separate etcd cluster) may be prohibitive for many use cases.
freedomben 6y ago
Custom resources are really "operators" and there are some really great operators, and there are some not so great ones. Notably OpenShift 4 is implemented as operators on top of K8s, so there is a lot you can do with them.
amerine 6y ago
The “kuberbetes as a reconciler loop” for CRDs pattern is its must useful quality to us at Heroku.

eecc 6y ago

Don’t know but it’s a shame that the k8s API doesn’t have link fields... it kind of breaks the whole concept of automatic interfacing

BenTheElder 6y ago
Can you elaborate on what you mean by link field? I've searched a bit but I'm not certain what you're referring to in this context.
- Pirate-of-SV 6y ago
  HATEOAS I suppose. https://restfulapi.net/hateoas/ https://restfulapi.net/hateoas/

jsabo 6y ago

I'm not sure what you mean by too many things ended up as custom resources -- it's the recommended extension point and you can do a lot with them that you can't do with core types. Custom resources had/have some growing pains but I think worked out pretty well. It can be very very hard to test and distribute them though. As someone who maintained a controller with paid support, your test matrix gets pretty large pretty fast accounting for different k8s versions, different hosted versions (GKE, AKS, etc), different distros (openshift, rancher, etc). And that's before you even get into specific configurations like pod security policies, can the control plane communicate with the data plane, is there a service mesh. Resource versioning is hard to get right. Once a resource type is v1, it becomes difficult to extend it. You can't add a beta field to it easily. Revising schema can be hard since anything more than a no op conversion between versions requires a webhook, which requires a certificate chain, and while cert-manager is popular it is not ubiquitous and regularly has breaking changes. Webhook setup issues made up a large portion of our support requests. As far as the general "reconciliation loop" architecture goes, you end up with something similar in most orchestration systems I've worked on, or you wish that you did. So overall I think that worked out well. Getting it right can be hard, but I think that's the nature of the beast.

dhruvt10 6y ago
Really interested to know about the challenges you faced in revisioning but couldn't find the exact thing, do you happen to know a source for understanding it more?
- kitd 6y ago
  Not OP but the feature you're looking for is OLM https://olm.operatorframework.io/docs/ https://olm.operatorframework.io/docs/

Thaxll 6y ago

Well look how popular Kubernetes is, how many thousands of things built on top of it because its powerful and extensible API model.

jacques_chester 6y ago

> How has Kubernetes being a giant reactive resource API panned out? It was never reactive, at least in the sense of the Reactive Manifesto. There is no system of backpressure. > Did the custom resource controllers pan out well? I think it will prove to be useful for a handful of uses, but that in general, it will wind up like Wordpress and Jenkins plugins. Powerful, popular, but a guaranteed mess. Kubernetes was not originally designed to accommodate CRDs. There's no concept of tenancy, only things you can cobble together into tenancy-ish-alike-kinda shapes. There's RBAC, but it was designed before CRDs and means that poorly designed custom controllers can be attack vectors. I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws. Unfortunately, I don't have the wits to join into the multi-billion dollar market of "papering over things Kubernetes doesn't do very well but which can't now be changed". [0] I mean, I wrote a book about Knative, which is a purely CRD-based architecture.

bogomipz 6y ago
>"I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws." Could you elaborate on the "design flaws" part? Do you feel that the CRD model itself has design flaws or are you referring to specific project's CRDs?
- psoundy 6y ago
  tl;dr If you’re operating with CRDs at trivial scale, you probably having nothing to worry about. But operating with CRDs at scale is a different story and suggests careful testing with the specific applications involved. ——- The usage patterns of native k8s types and the implications those patterns have on the scalability and reliability of etcd and the apiserver are relatively well-understood. CRDs can be a wild-card, though, and afaik testing efforts thus far have not investigated worst-case usage of CRD-based applications. As commonly deployed, CRDs are served from the same apiservers and etcd cluster that serves the native types for a k8s cluster. That can result in contention between the CRDs supporting 3rd party additions to a cluster and the native types critical to the health of a cluster. This kind of contention has the potential to bring a cluster to its knees. Efforts like priority and fairness seek to ensure that the apiserver can prioritize at the level of the API call. But that won’t prevent watch caches from OOM’ing the apiserver if excessive numbers of CRDs are present. The judicious use of quotas could head off the creation of an excessive number of objects, but it’s not just count that matters - the size of each resource is also a factor. In theory, CRDs could be isolated from native types by serving them from an aggregated apiserver backed by a separate etcd cluster. afaik this not a supported configuration today, and even if it were the additional resources required to support it (especially the separate etcd cluster) may be prohibitive for many use cases.

freedomben 6y ago

Custom resources are really "operators" and there are some really great operators, and there are some not so great ones. Notably OpenShift 4 is implemented as operators on top of K8s, so there is a lot you can do with them.

amerine 6y ago

The “kuberbetes as a reconciler loop” for CRDs pattern is its must useful quality to us at Heroku.