3 ms·
I've found a few not-so-obvious pain points on my very limited k8s experience. 1) k8s makes a lot of sense for stateless applications (such as your website) bu
by dguaraglia 9y ago
I've found a few not-so-obvious pain points on my very limited k8s experience.
1) k8s makes a lot of sense for stateless applications (such as your website) but not so much for stateful applications that require a client to connect to the same container every time (there are ways to do it, but they are a pain in the ass.)
2) Tooling is getting better with time, but it's still pretty green. Packages for your usual orchestration tools like Puppet and Ansible are volunteer work so they get easily out of sync, or require more work than you'd expect to get going. Using their suggested YAML format leads to another problem: there's no easy way to keep secrets outside of the configuration files, unless you build your own process around it
3) Some pieces, like a replicated DB, might be easier to be kept outside of the k8s cluster. You can technically run them there, but they weren't designed for running in that kind of environment and sometimes it shows
4) The CI/CD story using Jenkins pipelines is far from solved. There are several packages that provide some solutions, but the documentation tends to be horrible and that leads to days of debugging through trial and error
5) Leaky abstractions everywhere. As an example, the Jenkins plugins to build using your k8s cluster suggests using the "credentials" system, but you need to add them manually after you boot the Jenkins service. Then your slaves stop receiving the credentials and you have to reboot Jenkins (I had to reboot it on average every 3 or 4 builds.)
Don't let that discourage you from using k8s as a "run" cluster, if your app is a shared-nothing, stateless app. It's so much easier to setup (specially on Azure and GCP) and it obviates the need for setting up Puppet + Sensu + load balancing just to make sure your service keeps running when a node dies.
- alexnewman 9y ago1) That's true, but why not support shared state on shared storage. If that's not an option. 2) What exactly do you not like tooling wise. 3) Totally agree. I am actually working on replacing etcd with postgresql though.
- gtirloni 9y agoCould you share more about #3 ? How do you view that in regard to this discussion? https://github.com/kubernetes/kubernetes/issues/1957 https://github.com/kubernetes/kubernetes/issues/1957
- dguaraglia 9y agoRegarding #2, I think it's hard to integrate k8s with the rest of your infrastructure. If you are using, say, a combination of Terraforms + Ansible or Puppet to keep everything in shape, right now you have two options: 1) Write a bunch of bash scripts around kubectl and a bunch of YAML files. While painful, this the way I ended up going for (plus blackbox to GPG encrypt/decrypt secrets on the repo) 2) Try to use your usual tools (Ansible/Puppet) as a replacement for kubectl. This is the dream, but the plugins for Ansible and Puppet only support subsets of the latest k8s features, or require annoying stuff like setting your API endpoint on every task. In other words, getting reproducible deploys between different k8s clusters (say, one for staging and one for production) is not really a thing yet. I guess there's an argument for saying that one should use k8s-native solutions for that (such as namespaces) but what about having clusters in different data centers?
- alexnewman 9y ago2) This is a very bad idea
- vkat 9y agoFor #1 have you tried using local storage available in 1.7, see example https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume https://github.com/kubernetes-incubator/external-storage/tre...
- kuschku 9y ago> Using their suggested YAML format leads to another problem: there's no easy way to keep secrets outside of the configuration files, unless you build your own process around it Isn’t that what the Secrets resources are for? Just use those, and mount them into your containers in the filesystem or environment variables.
- eropple 9y agoKubernetes "secrets" aren't actually...well...secret. They're stored unencrypted in etcd with no ACLs. You must go outside the Kubernetes system to do this. If I control a physical cluster of machines end-to-end, I might consider Kubernetes (because I don't have any better options and doing the work to actually secure it is probably less work than my alternatives), but it's also one of the reasons--though far from the only reason--that I couldn't begin to consider k8s if I'm running in AWS or another cloud environment.
- erickt 9y ago> Kubernetes "secrets" aren't actually...well...secret. They're stored unencrypted in etcd with no ACLs. You must go outside the Kubernetes system to do this. They are in progress towards encrypting at rest the etcd database. You can experimentally turn it on by following this doc: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...
- SEJeff 9y agoKubernetes 1.7 added the concept of a Provider to encrypt secrets to/from etcd3: https://docs.google.com/document/d/1lFhPLlvkCo3XFC2xFDPSn0jAGpqKcCCZaNsBAv8zFdE/edit# https://docs.google.com/document/d/1lFhPLlvkCo3XFC2xFDPSn0jA... https://github.com/kubernetes/kubernetes/pull/41939 https://github.com/kubernetes/kubernetes/pull/41939 https://github.com/kubernetes/kubernetes/pull/49350 https://github.com/kubernetes/kubernetes/pull/49350 etc. There is an implementation to use Hashicorp's Vault to encrypt secrets (using this functionality) which looks quite nice actually: https://docs.google.com/document/d/15-baW4i7qws1yxxIYjHXqKpk259ebauQbECQCpPD308/edit# https://docs.google.com/document/d/15-baW4i7qws1yxxIYjHXqKpk...
- markbnj 9y agoRegarding CI/CD we use gitlab pipelines to build and push containers and run 'helm upgrade --install' to update charts in the cluster to use the newly pushed images. Works very well and not at all difficult to set up.