5 ms·
Getting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do. The tricky part is long term maintenance and storage. I'm not reall
by mrweasel 6y ago
Getting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do. The tricky part is long term maintenance and storage.
I'm not really sure what people expect gain from these kinds of article, they're great as notes, but it's not something I'd use as a starting point for installing a production Kubernetes cluster.
The initial setup of a Kubernetes cluster is something most HN readers could do in half a day or so. Learning to manage a cluster, that's tricky. Even if you resort to tools like Rancher or similar, you're still in deep waters.
Also why would people assume that there's any difference in installing Kubernetes on an operating system running on physical hardware vs. on virtual machines?
- Spivak 6y agoWhen people talk about “bare metal” k8s they mean on-prem without the support of integrated cloud infrastructure. All the external stuff like load balancers, ingress controllers, routing, bgp, public ip pools, shared storage, vm creation for scaling, are all things the cluster can’t do for itself and have to be implemented and integrated.
- Already__Taken 6y agoShared storage being a particular head-f to get solved. Everything else is a reasonably approachable problem you can tackle as it comes.
- tene 6y agohttps://rook.io/ https://rook.io/ is a really great way to get shared storage up quickly and simply.
- Damogran6 6y agoI'm still looking for a good resource as someone who's taken their eye off the ball for a decade or so. Things have (and continue to) move at an astounding pace...and I'm needing something that goes from 'Pets and autoconf' to where we are now.
- neurostimulant 6y agoIt's still using the same old stack, just containerized. For example, you're no longer manually editing nginx conf, but instead writing the config as yaml annotation to tell the nginx ingress what to do. Instead of editing wp-config.php when deploying wordpress, you specify the db config as environmental variables in yaml file, etc.
- mrweasel 6y ago>For example, you're no longer manually editing nginx conf, You now have to do the configuration in YAML, which is MUCH MUCH worse.
- zimbatm 6y agoOr more accurately, you now have to understand both the nginx.conf file format, and the right way to nudge the YAML so it produces what you want. Plus some intermediate wrapper script that sets things up on container start that does more funky things based on environment variables.
- what2build 6y agoWhich is really weird we do this since 80% of tasks fall within sane defaults territory. My last employer had something like 10 hits every 5 minutes. Somehow we needed the entire Kube ecosystem for that. I’ve been storing good JSON templates in Dynamodb and iterating on code to query and deploy. Building up a collection that lets me deploy to AWS or GCP with one tool, editing fields I want to edit per setup. Working with just my code and validating outputs is way less. On the one hand everyone is all “write less code, it’s too complicated.” while ignoring ops tooling always ends up a fiasco of DSLs and templating hell.
- nouney 6y agoI don't get it, how do you configure nginx with a YAML file?
- nicolast 6y ago> Getting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do. The tricky part is long term maintenance and storage. This times 100. Deploying basic clusters is easy. Keeping a test/dev-cluster running for a while? Sure. Keeping production clusters running (TLS cert TTLs expiring, anyone?), upgrading to new K8s versions, proper monitoring (the whole stack, not just your app or the control-plane), provisioning (local) storage,... is where difficulties lie.
- ninjha 6y agoI’m working on this right now. My theory is that having every cluster object defined in git (but with clever use of third party helm charts to reduce maintenance burden) is the way to go. Our cluster configuration is public[1] and I’m almost done with a blog post going over all the different choices you can make wrt the surrounding monitoring/etc infrastructure on a Kubernetes cluster. [1] https://github.com/ocf/kubernetes https://github.com/ocf/kubernetes
- deleted 6y ago[deleted]
- nicolast 6y agoMy comment above did not come out of the blue, but based on real-world experience ;-) You may be interested in our MetalK8s project [1] which seems related to yours. [1] https://github.com/scality/metalk8s https://github.com/scality/metalk8s
- aprdm 6y agoThis looks really cool. Would you include dev apps in the mono repo too? Say you have 40 services by 20 different teams.
- tutfbhuf 6y agoIf I already have an production cluster up and running for a while, how can I get it git tracked now? Another question: how about things like autoscaler which automatically edits numbers in deployments, how to git track that?
- INTPenis 6y agoI was one of those backwards people who opposed containers, until late 2019. My start was the official kubernetes docs, step by step. Try everything, write it down, make ansible playbooks. Even used some of their interactive training modules, but I quickly had my own cluster up in vagrant so didn't really need the online shell. Now we have two clusters at work, I have ansible playbooks I'm really happy with that help me manage both our on-prem clusters and my managed LKE with the same playbooks. I'm completely sold on this container thing. :)
- aprdm 6y agoAre they OSS by any chance? And what kind of tasks do they do?
- INTPenis 6y agoYes but I won't link them. They are focused only on CentOS 7, you might be able to find them yourself. As far as I can tell they include everything except persistent storage and HA control plane.
- rmdashrfstar 6y agoJust use Kubespray, not much sense in wasting time writing your own unless your intention is specifically learning how to manage cluster setup with stateless Ansible
- INTPenis 6y agoI actually looked at kubespray first but first of all it did not work out of box as promised. And to troubleshoot it you basically had to know about kubernetes already. So it made a lot of sense to start at the official docs. And while I'm reading the docs, why not build the cluster at the same time. And while I'm building the cluster, why not write each step down in Ansible so I won't have to repeat myself. So end result is my own ansible setup that I'm happy with and I know inside and out. It always bothered me to run other people's ansible playbooks. I'm too much of a control freak.
- 6y ago
- throwaway201103 6y ago> Getting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do. Care to share an "easy" recipe, because I haven't found one that actually works. It always falls apart for me with the networking. Let's say I have a cluster of 8 physical nodes and a management node available in my data center and I want to set up k8s for use by my internal users. I'm a solo admin with responsibilty for ~250 physical servers so any ongoing management necessary will be very much a task among many. Is this blog post a good guide? Is there a better one?
- nicolast 6y agoAn 'easy' way to deploy a cluster could be using kubeadm. Then you'll need a CNI like Calico to get Pod networking up-and-running. However, you'll want to install a bunch of other software on said cluster to monitor it, manage logs,... Given you're running on physical infrastructure, MetalK8s [1] could be of interest (full disclosure: I'm one of the leads of said project, which is fully open-source and used as part of our commercial enterprise storage products) [1] https://github.com/scality/metalk8s https://github.com/scality/metalk8s