17 ms·

I'm really curious how people administer their K8S clusters for installation, upgrades, etc. I'm very familiar with docker, which we've been using for over 2 y

by RRRA 9y ago

I'm really curious how people administer their K8S clusters for installation, upgrades, etc. I'm very familiar with docker, which we've been using for over 2 years. But now, we're trying to get k8s running, with either kargo, kubeadm, deb packages, etc.: They all failed with different bugs on different set of clouds / settings. (Trying to stick to running it on Ubuntu xenial). Not sure if it's because 1.6.* just came out of the oven when I started...? Thanks to Minikube, I understand how powerful k8s can be, and actually find kubectl quite simple to use, but I'm confused by how fragile and complex installation and setup seems to be. I'm unsure how someone is supposed to maintain this system considering how (overly?) modular it is and the bugs I've encountered. Knowing that docker has a LOT of bugs, and k8s builds on top of it, I'm a bit scared. And there is no clean documentation on how to install it, with sections for all your choices, in a generic/agnostic way (deb+rpm distros, cloud integration or simple abstract VMs, ...) What is you workflow? :)

ben_jones 9y ago
I agree with k8s not being environment agnostic. It's most of the way there but your decision to run on GCP, AWS, or some other platform, can play a large role in decision making and k8s adoption. I feel like right now in particular development is weighting to make k8s more friendly on GCP probably because the GCP team is working really hard and they're using that momentum (and a lot of open source / conference presence) to keep that advantage going forward. I think minikube is a crucial early step in allowing at least agnostic development where a "professional" can then ease it into a cloud provider by turning knobs here and there. It'll probably stay that way for awhile but we'll see.
MattRogish 9y ago
We (ReactiveOps.com) use kops: https://github.com/kubernetes/kops https://github.com/kubernetes/kops It works well with Ubuntu and we are working on CoreOS (although no promises it gets merged back into core). Kops can do upgrades although historically it was safer/easier to stand up another cluster alongside and migrate. Happy to help if you have any questions - Matt [at] reactiveops.com.
pat2man 9y ago
We are using https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible which gets you a k8s cluster + user management and a great web interface. I actually learned a lot about cluster management from that repo since it goes through all the standard best practices.
pas 9y ago
Use CoreOS, use kubeadm. Knowledge of go-lang is necessary (I hope this will change, as go's readability is ... very lacking). You have to understand that there are still a lot of unfinished features (for example, almost no real documentation), still a lot of operational aspects left uncovered (persistent volume backed by local disk - for example running software that needs low-latency I/O, eg. DB servers). The general installation flow is to beat it into submission. Drag the thing kicking and screaming into a cluster until it forms a quorum. (etcd, apiserver + controller-manager + cloud-stuff, scheduler, kubelets), and don't forget about the overlay network. And that's it, if it works and ugly, it works. As you say there are too many bugs in docker/rkt (OCR, libcontainer, the container filesystem problems - overlayfs, aufs, btrfs, devicemapper, AppArmor, SELinux labels and so on issues, and other Linux kernel related issues), and in Kubernetes itself too, and then there's the whole networking layer/aspect, still very much in flux. But it's usable, because it's "antifragile", so if it can reach a working state, you can be pretty confident that it'll be able to reach it again if you add more nodes, nodes crash, load fluctuates, updates happen, deployments happen, etc.

by RRRA 9y ago

ben_jones 9y ago

I agree with k8s not being environment agnostic. It's most of the way there but your decision to run on GCP, AWS, or some other platform, can play a large role in decision making and k8s adoption. I feel like right now in particular development is weighting to make k8s more friendly on GCP probably because the GCP team is working really hard and they're using that momentum (and a lot of open source / conference presence) to keep that advantage going forward. I think minikube is a crucial early step in allowing at least agnostic development where a "professional" can then ease it into a cloud provider by turning knobs here and there. It'll probably stay that way for awhile but we'll see.

MattRogish 9y ago

We (ReactiveOps.com) use kops: https://github.com/kubernetes/kops https://github.com/kubernetes/kops It works well with Ubuntu and we are working on CoreOS (although no promises it gets merged back into core). Kops can do upgrades although historically it was safer/easier to stand up another cluster alongside and migrate. Happy to help if you have any questions - Matt [at] reactiveops.com.

pat2man 9y ago

We are using https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible which gets you a k8s cluster + user management and a great web interface. I actually learned a lot about cluster management from that repo since it goes through all the standard best practices.

pas 9y ago

Use CoreOS, use kubeadm. Knowledge of go-lang is necessary (I hope this will change, as go's readability is ... very lacking). You have to understand that there are still a lot of unfinished features (for example, almost no real documentation), still a lot of operational aspects left uncovered (persistent volume backed by local disk - for example running software that needs low-latency I/O, eg. DB servers). The general installation flow is to beat it into submission. Drag the thing kicking and screaming into a cluster until it forms a quorum. (etcd, apiserver + controller-manager + cloud-stuff, scheduler, kubelets), and don't forget about the overlay network. And that's it, if it works and ugly, it works. As you say there are too many bugs in docker/rkt (OCR, libcontainer, the container filesystem problems - overlayfs, aufs, btrfs, devicemapper, AppArmor, SELinux labels and so on issues, and other Linux kernel related issues), and in Kubernetes itself too, and then there's the whole networking layer/aspect, still very much in flux. But it's usable, because it's "antifragile", so if it can reach a working state, you can be pretty confident that it'll be able to reach it again if you add more nodes, nodes crash, load fluctuates, updates happen, deployments happen, etc.