4 ms·

I see "operators"--as in people responsible for keeping k8s infra healthy--are still not using "operators"--as in the k8s code--as much as they should. It's a p

I see "operators"--as in people responsible for keeping k8s infra healthy--are still not using "operators"--as in the k8s code--as much as they should. It's a powerful abstraction to extend a control loop into worker nodes. One customer we work with standardized around "operator or nothing"--i.e. never deploy anything in a cluster that is not an operator. Their infrastructure remains minimalistic, yet super functional and debuggable.

aliasxneo 4y ago
It's interesting to see the different views around it. I tend to agree that we tend to exclusively use operators to make Kubernetes speak the same language as our business. It's sort of like object-oriented deployments...can make some pretty power abstractions.
- akhayam 3y ago
  object-oriented deployment... huh... That's an interesting mental model that I didn't have. Thanks for sharing! For me, it's a standard which works well for short-cycle control loop. Standards are never perfect, but the best ones are minimalistic and built out of real world experience. Operators check these boxes for me. Not sure if you can share, but if you can, would love to know the type of business logic you moved into operators.
twalla 4y ago
I see issues with going full on kubernetes-as-controlplane due to how operators do reconciliation. The represented resources are more or less eventually consistent which present challenges if you have workflows that need explicit orchestration and strong requirements for certain dependencies to be met before progressing. If the logic for a particular operator (quality varies here) doesn't handle missing pre-reqs or downstream resources being in a weird state gracefully you end up having to chase down tons of events and logs that are distributed across many components with no real way to quickly and effectively pinpoint where things went sideways (this comes with experience for most human cluster operators but most application developers are not going to know you need to go check the logs of these 5 controllers and check the status of their attendant custom resources to see what needs unwedging) I understand Flux has some capabilities to pause or perform incremental deployment until certain dependencies are met, which is a step in the right direction.
- akhayam 3y ago
  I hear you, and am not advocating a full, one-size-fits-all, controller solution for control plane and workers. My take was mainly on using operators for short-cycle control loops inside workers which are inherently simpler in their business logic. I do agree with you on deployment systems taking up more responsibility in understanding app properties and abstracting infrastructure away from app developers. However, it's notoriously hard to get deployment systems to do this correctly. At Amazon, this was a somewhat solved problem but took their best brains, many years of pain, and multiple iterations to get right. If you haven't already, I would recommend reading this thread from Joe Magerramov where he talks about the challenges of deployment systems and how many stars need to line up to get deployments done right: https://twitter.com/_joemag_/status/1587283479448150016 https://twitter.com/_joemag_/status/1587283479448150016. Both Flux and Argo are doing splendid work to solve this problem. I just wish we can democratize the known wisdom, failures and patterns/anti-patterns from cloud operators, so others don't have to repeat the same mistakes. I'll see if I can put together some blog posts on this topic.

aliasxneo 4y ago

It's interesting to see the different views around it. I tend to agree that we tend to exclusively use operators to make Kubernetes speak the same language as our business. It's sort of like object-oriented deployments...can make some pretty power abstractions.

akhayam 3y ago
object-oriented deployment... huh... That's an interesting mental model that I didn't have. Thanks for sharing! For me, it's a standard which works well for short-cycle control loop. Standards are never perfect, but the best ones are minimalistic and built out of real world experience. Operators check these boxes for me. Not sure if you can share, but if you can, would love to know the type of business logic you moved into operators.

twalla 4y ago

I see issues with going full on kubernetes-as-controlplane due to how operators do reconciliation. The represented resources are more or less eventually consistent which present challenges if you have workflows that need explicit orchestration and strong requirements for certain dependencies to be met before progressing. If the logic for a particular operator (quality varies here) doesn't handle missing pre-reqs or downstream resources being in a weird state gracefully you end up having to chase down tons of events and logs that are distributed across many components with no real way to quickly and effectively pinpoint where things went sideways (this comes with experience for most human cluster operators but most application developers are not going to know you need to go check the logs of these 5 controllers and check the status of their attendant custom resources to see what needs unwedging) I understand Flux has some capabilities to pause or perform incremental deployment until certain dependencies are met, which is a step in the right direction.

akhayam 3y ago
I hear you, and am not advocating a full, one-size-fits-all, controller solution for control plane and workers. My take was mainly on using operators for short-cycle control loops inside workers which are inherently simpler in their business logic. I do agree with you on deployment systems taking up more responsibility in understanding app properties and abstracting infrastructure away from app developers. However, it's notoriously hard to get deployment systems to do this correctly. At Amazon, this was a somewhat solved problem but took their best brains, many years of pain, and multiple iterations to get right. If you haven't already, I would recommend reading this thread from Joe Magerramov where he talks about the challenges of deployment systems and how many stars need to line up to get deployments done right: https://twitter.com/_joemag_/status/1587283479448150016 https://twitter.com/_joemag_/status/1587283479448150016. Both Flux and Argo are doing splendid work to solve this problem. I just wish we can democratize the known wisdom, failures and patterns/anti-patterns from cloud operators, so others don't have to repeat the same mistakes. I'll see if I can put together some blog posts on this topic.