9 ms·

This is more a criticism of operators than StatefulSets. The promise of operators is easier automated management of your resources. If the operator can't do bas

This is more a criticism of operators than StatefulSets. The promise of operators is easier automated management of your resources. If the operator can't do basic things like resizing disks, it's flawed and needs to be fixed. However, I've always been suspicious of operators, so maybe my bias is leaking here. Kubernetes is already a level of abstraction and adding another seems risky and unnecessary.

mjg235 4y ago
The flaw is definitely most acute when it is combined with operators, but it would make storage resizes vastly more accessible for less capable devs if it were able to be done declaratively via the spec instead of via a painful workaround.
dmead 4y ago
I agree with this. I manage a monorepo of 10s of terraform/terragrunt modules most of which are deployed to a k8s cluster. the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.
- encryptluks2 4y ago
  A Git repo can still be a good audit log even if you use templates, especially if you're adding a step to transform files with each push.
- raffraffraff 4y ago
  Infra as code can't actively manage things like database clusters, handling promotion of a secondary, automatic deployment of new replicas, scaling. Using it to deploy RDS Aurora doesn't count. My impression is that operators are designed to handle those "active" decisions in real-time, and that's outside the realm of terraform, GitOps etc.
  - dmead 4y ago
    Indeed, and if the k8s api was so precisely laid out that you could look at a log of side effect changes that came as the result of scalers, operator, templates etc that would be one thing, but we aren't there yet. Right now what i have is "oh the cluster is in a different state (there is drift) than the code, lets be a detective to see why this changed".
- nunez 4y ago
  To be fair there are tools that use operators (like Flux) to apply desired state onto Kubernetes clusters using Git, so the GitOps is still there. You can even combine flux with ClusterAPI to create clusters with CRDs through Git, which can be quite handy
nithril 4y ago
As long as the operator is really well designed but it is rarely the case. I switched from helm to bare k8s manifest for the same reason, helm/operator can be helpful, but not when the maintainer introduces breaking change (that does not relate to the service by itself) that needs a delete/deploy of the k8s resources
- encryptluks2 4y ago
  To be fair Helm is just a template layer for manifests, so it would take quite a bit of effort to break things.. especially if you're using a projects official Helm charts.
  - sieabahlpark 4y ago
- bavell 4y ago
  Yep, this is also the reason I don't use helm in any of my clusters. Too much magic and extra dependencies for a marginal (IMO) benefit.
debarshri 4y ago
Best operators are the operators that you don't have to deal with. The moment you have to investigate stuff, life is hell.
shp0ngle 4y ago
And now imagine it’s all in a Helm chart. And using kustomize magic So you start debugging the helm chart, and kustomize, and the operator. Not fun.
- mkl95 4y ago
  Your service is now crippled by a bunch of spaghetti APIs on top of the stuff that was crippling it before. You are cloud-scale!
  - code_biologist 4y ago
    Now that you've hit cloud scale, engineering leadership can make the case for budget for an ops hire to keep the thing working. Bigger headcount is always a plus.
    - mkl95 4y ago
      Also make sure that Dev and Ops are separate departments that are not allowed to work together. Devops? We don't do that here. BTW you need to fix that 15-second query ASAP.
    - ok_dad 4y ago
      > Bigger headcount is always a plus. You can't make a baby in 1 month with 9.5 women.
      - code_biologist 4y ago
        Listening to the empire builders among engineering management, they'll happily tell you that a few more hires will get the current baby back on schedule plus future babies can be done in 8, maybe even 7, months.
uberduper 4y ago
An operator could trivially create the pvc ahead of any sts scaling op. The sts will simply look for an existing pvc by a predictable name and use it rather than create a new one, regardless of whether the pre-created pvc matches the spec of what the sts defines.
nunez 4y ago
I like the idea of operator pattern (Operators aren't a built-in resource yet; they REALLY should be) but absolutely HATE how opaque they are. You really don't have a window into seeing what they are doing; often times (in my experience), you only know things are off much later on (because most operators rely on Kubernetes events to notify human operators that things have gone wrong).

mjg235 4y ago

The flaw is definitely most acute when it is combined with operators, but it would make storage resizes vastly more accessible for less capable devs if it were able to be done declaratively via the spec instead of via a painful workaround.

dmead 4y ago

I agree with this. I manage a monorepo of 10s of terraform/terragrunt modules most of which are deployed to a k8s cluster. the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.

encryptluks2 4y ago
A Git repo can still be a good audit log even if you use templates, especially if you're adding a step to transform files with each push.
raffraffraff 4y ago
Infra as code can't actively manage things like database clusters, handling promotion of a secondary, automatic deployment of new replicas, scaling. Using it to deploy RDS Aurora doesn't count. My impression is that operators are designed to handle those "active" decisions in real-time, and that's outside the realm of terraform, GitOps etc.
- dmead 4y ago
  Indeed, and if the k8s api was so precisely laid out that you could look at a log of side effect changes that came as the result of scalers, operator, templates etc that would be one thing, but we aren't there yet. Right now what i have is "oh the cluster is in a different state (there is drift) than the code, lets be a detective to see why this changed".
nunez 4y ago
To be fair there are tools that use operators (like Flux) to apply desired state onto Kubernetes clusters using Git, so the GitOps is still there. You can even combine flux with ClusterAPI to create clusters with CRDs through Git, which can be quite handy

nithril 4y ago

As long as the operator is really well designed but it is rarely the case. I switched from helm to bare k8s manifest for the same reason, helm/operator can be helpful, but not when the maintainer introduces breaking change (that does not relate to the service by itself) that needs a delete/deploy of the k8s resources

encryptluks2 4y ago
To be fair Helm is just a template layer for manifests, so it would take quite a bit of effort to break things.. especially if you're using a projects official Helm charts.
- sieabahlpark 4y ago
bavell 4y ago
Yep, this is also the reason I don't use helm in any of my clusters. Too much magic and extra dependencies for a marginal (IMO) benefit.

debarshri 4y ago

Best operators are the operators that you don't have to deal with. The moment you have to investigate stuff, life is hell.

shp0ngle 4y ago

And now imagine it’s all in a Helm chart. And using kustomize magic So you start debugging the helm chart, and kustomize, and the operator. Not fun.

mkl95 4y ago
Your service is now crippled by a bunch of spaghetti APIs on top of the stuff that was crippling it before. You are cloud-scale!
- code_biologist 4y ago
  Now that you've hit cloud scale, engineering leadership can make the case for budget for an ops hire to keep the thing working. Bigger headcount is always a plus.
  - mkl95 4y ago
    Also make sure that Dev and Ops are separate departments that are not allowed to work together. Devops? We don't do that here. BTW you need to fix that 15-second query ASAP.
  - ok_dad 4y ago
    > Bigger headcount is always a plus. You can't make a baby in 1 month with 9.5 women.
    - code_biologist 4y ago
      Listening to the empire builders among engineering management, they'll happily tell you that a few more hires will get the current baby back on schedule plus future babies can be done in 8, maybe even 7, months.

uberduper 4y ago

An operator could trivially create the pvc ahead of any sts scaling op. The sts will simply look for an existing pvc by a predictable name and use it rather than create a new one, regardless of whether the pre-created pvc matches the spec of what the sts defines.

nunez 4y ago

I like the idea of operator pattern (Operators aren't a built-in resource yet; they REALLY should be) but absolutely HATE how opaque they are. You really don't have a window into seeing what they are doing; often times (in my experience), you only know things are off much later on (because most operators rely on Kubernetes events to notify human operators that things have gone wrong).