4 ms·

Yeah, I wanted to emphasize the difference between the kubernetes resource and the actual things that the controllers create based on the existence and attribut

Yeah, I wanted to emphasize the difference between the kubernetes resource and the actual things that the controllers create based on the existence and attributes of those resources. I suppose "materialization" is a good word for it; you create a resource, and if it's something that actually needs something to execute, it is eventually materialized as a container scheduled to run on a kubelet, or an EBS volume, or some other actual thing that the appropriate controller creates. I like the core idea of Kubernetes (eventually consistent declarative infrastructure) though I'm not fully sold on the implementation and the tooling around it.

nyellin 5y ago
Yeah fair point. Regarding the implementation and tooling, what's lacking in your opinion?
- chousuke 5y ago
  Well, some thoughts in no particular order: 0. YAML everywhere. Augh. 1. Vanilla kubernetes by itself doesn't really do much; everyone basically assumes you'll use a cluster in a cloud somewhere that comes with all kinds of add-ons and functionality on top of the default. If you want to run an on-premises cluster, once you start adding external components to a Kubernetes cluster to make it actually useful, it starts getting complicated. 2. I don't think configuration management of the resources you do put in a cluster is a fully solved problem, especially if you depend on CRDs. 3. kubectl being context-sensitive is rather annoying. You basically can't share a kubeconfig between two terminal sessions without severe danger of footgunning. 4. If you just use Helm packages with defaults, you'll end up running software with very collision-prone UIDs and security contexts, so two separate components that you deploy can end up running as the same user on a host. I don't think that's good security design, but the system doesn't encourage you to do the correct thing. 5. All the "easy" tooling is basically just "run this magic incantation and it will do things for you". The culture around how things are done seem to encourage ignorance about what you're actually deploying. Many operations also modify cluster state and thus should be managed and not done willy-nilly, but that's also ignored by default. See also #2 Kubernetes has its upsides (something like ArgoCD is very nice to use when set up properly) but I guess my main beef with it is that it can make things look easy that actually aren't, and mislead its users.
  - nyellin 5y ago
    Yeah, fair points. Regarding (3) there are tools for that like kubie. I've still shot myself in the foot before. https://github.com/sbstp/kubie https://github.com/sbstp/kubie

nyellin 5y ago

Yeah fair point. Regarding the implementation and tooling, what's lacking in your opinion?

chousuke 5y ago
Well, some thoughts in no particular order: 0. YAML everywhere. Augh. 1. Vanilla kubernetes by itself doesn't really do much; everyone basically assumes you'll use a cluster in a cloud somewhere that comes with all kinds of add-ons and functionality on top of the default. If you want to run an on-premises cluster, once you start adding external components to a Kubernetes cluster to make it actually useful, it starts getting complicated. 2. I don't think configuration management of the resources you do put in a cluster is a fully solved problem, especially if you depend on CRDs. 3. kubectl being context-sensitive is rather annoying. You basically can't share a kubeconfig between two terminal sessions without severe danger of footgunning. 4. If you just use Helm packages with defaults, you'll end up running software with very collision-prone UIDs and security contexts, so two separate components that you deploy can end up running as the same user on a host. I don't think that's good security design, but the system doesn't encourage you to do the correct thing. 5. All the "easy" tooling is basically just "run this magic incantation and it will do things for you". The culture around how things are done seem to encourage ignorance about what you're actually deploying. Many operations also modify cluster state and thus should be managed and not done willy-nilly, but that's also ignored by default. See also #2 Kubernetes has its upsides (something like ArgoCD is very nice to use when set up properly) but I guess my main beef with it is that it can make things look easy that actually aren't, and mislead its users.
- nyellin 5y ago
  Yeah, fair points. Regarding (3) there are tools for that like kubie. I've still shot myself in the foot before. https://github.com/sbstp/kubie https://github.com/sbstp/kubie