3 ms·
Exactly. A year or so ago I thought, hey, maybe I should redo my personal infrastructure using Kubernetes. Long story short, it was way too much of a pain in th
by wpietri 7y ago
Exactly. A year or so ago I thought, hey, maybe I should redo my personal infrastructure using Kubernetes. Long story short, it was way too much of a pain in the ass.
As background, I've done time as a professional sysadmin. My current infrastructure is all Chef-based, with maybe a dozen custom cookbooks. But Chef felt kinda heavy and clunky, and the many VMs I had definitely seemed heavy compared with containerization. I thought switching to Kubernetes would be pretty straightforward.
Surprise! It was not. I moved the least complex thing I run, my home lighting daemon to it; it's stateless and nothing connects to it, but it was still a struggle to get it up and running. Then I tried adding more stateful services and got bogged down in bugs, mysteries, and Kubernetes complexity. I set it aside, thinking I'd come back to it later when I had more time. That time never quite arrived, and a month or so ago my home lights stopped working. Why? I couldn't tell. A bunch of internal Kubernetes certificates expired, so none of the commands worked. Eventually, I just copy-pasted stuff out of Stack Overflow and randomly rebooted things, and eventually it started working again.
I'll happily look at it again when I have to do serious volume and can afford somebody to focus full-time on Kubernetes. But for anything small or casual, I'll be looking elsewhere.
- busterarm 7y agoAt work we're building an entire service platform on top of managed kubernetes services, agnostic to cloud provider. We had already had bad experiences running K8s ourselves. Going into it we knew how much of a PITA it would be but we vastly underestimated how much, IMO. Would not do again -- I would quit first.
- pnako 7y agoFire and Motion https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ Written 18 years ago, so obviously not about Kubernagus, but it does explain the same phenomenon. Replace Microsoft with cloud providers and that's more or less the same argument.
- cnf 7y agoI run my home automation and infrastructure on kubernetes, and for me that is one of the smoothest ways of doing it. I find it quite easy to deal with, and much prefer it to the “classic” way of doing it.
- dnautics 7y agowhat dark magic are you using? Not joking. I've tried learning kubernetes several times and gave up. Maybe I'm not the smartest. Can you point to guides that helped you get up and running smoothly? This is probably something I should put some more effort into in the coming months.
- yebyen 7y agoI think this is really hard, it's a bit like how we talk about learning Rails in the Ruby community. "Don't do it" Not because it's bad or especially hard, but because there's so much to unpack, and it's so tempting to unpack it all at once, and there's so much foundational stuff (Ruby language) which you really ought to learn before you try to analyze in detail exactly how the system is built up. I learned Kubernetes around v1.5 just before RBAC was enabled by default, and I resisted upgrading past 1.6 for a good long while (until about v1.12) because it was a feature I didn't need, and all the features after it appeared to be something else which I didn't need. I used Deis Workflow as my on-ramp to Kubernetes, and now I am a maintainer of the follow-on fork, which is a platform that made great sense to me, as I was a Deis v1 PaaS user before it was rewritten on top of Kubernetes. Since Deis left Workflow behind after they were acquired by Microsoft, I've been on Team Hephy, which is a group of volunteers that maintains the fork of Deis Workflow. This was my on-ramp, and it looks very much like it did in 2017, but now we are adding support for Kubernetes v1.16+ which has stabilized many of the main APIs. If you have a way to start a Kubernetes 1.15 or less cluster, I can recommend this as something to try[1]. The biggest hurdle of "how do I get my app online" is basically taken care of you. Then once you have an app running in a cluster, you can start to learn about the cluster, and practice understanding the different failure modes as well as how to proceed with development in your new life as a cluster admin. If you'd rather not take on the heavyweight burden of maintaining a Workflow cluster and all of its components right out of the gate (and who could blame you) I would recommend you try Draft[2], the lightweight successor created by Deis/Azure to try to fill the void left behind. Both solutions are based on a concept of buildpacks, though Hephy uses a combination of Dockerfile or Heroku Buildpacks and by comparison, Draft has its own notion of a "Draftpack" which is basically a minimalistic Dockerfile tailored for whatever language or framework you are developing with. I'm interested to hear if there are other responses, these are not really guides so much as "on-ramps" or training wheels, but I consider myself at least marginally competent, and this is how I got started myself. [1]: https://web.teamhephy.com https://web.teamhephy.com [2]: https://github.com/azure/draft https://github.com/azure/draft
- mixmastamyk 7y agoSo it sounds like it doesn't "just run" after all.
- yebyen 7y agoSo, assuming that Kubernetes was installed using kubeadm, this was the document you needed to find: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/... Moreover, if you are keeping pace with kubeadm upgrades at all (minor releases are quarterly, and patches are more frequent) then since the most recent minor release, Kubernetes 1.17, certificate renewal as an automated part of the upgrade process is enabled by default. You would have to do at least one cluster upgrade per year to avoid expired certs. tl;dr: this cert expiration thing isn't a problem anymore, but you do have to maintain your clusters. (Unless you are using a managed k8s service, that is...) https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#automatic-certificate-renewal https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/... The fact remains also that this is the very first entry under "Administration with Kubeadm", so if you did use kubeadm and didn't find it, I'm going to have to guess that either docs have improved since your experience, or you really weren't looking to administrate anything at all.
- wpietri 7y agoI appreciate the links, but for my home stuff I'll be ripping Kubernetes out. The notion that one has to keep pace with Kubernetes upgrades is exactly the kind of thing that works fine if you have a full-time professional on the job, and very poorly if it's a sideline for people trying to get actual productive work done. Which is fine; not everything has to scale down. But it very strongly suggests that there's a minimum scale at which Kubernetes makes sense.
- yebyen 7y agoOr, that there is a minimum scale/experience gradient behind which you are better served by a decent managed Kubernetes, when you're not prepared to manage it yourself. Most cloud providers have done a fairly good job to make it affordable. I think it's fair to say that the landscape of Kubernetes proper itself (the open source package) has already reached a more evolved state than the landscape of managed Kubernetes service providers, and that's potentially problematic, especially for newcomers. It's hard enough to pick between the myriad choices available; harder still when you must justify your choice to a hostile collaborator who doesn't agree with part or all. IMO, the people who complain the loudest about the learning curve of Kubernetes are those who have spent a decade or more learning how to administer one or more distributions of Linux servers, who have made the transition from SysV init to SystemD, and in many cases who are now neck deep in highly specialized AWS services, which in many cases they have used successfully to extricate from the nightmare-scape where one team called "System Admins" is responsible for broadly everything that runs or can run on any Linux server (or otherwise), from databases, to vendor applications, to monitoring systems, new service dev, platforming apps that were developed in-house, you name it... I basically don't agree that there is a minimum scale for Kubernetes, and I'll assert confidently that declarative system state management is a good technology, that is here to stay. But I respect your choice and I understand that not everyone shares my unique experiences, that led me to be more comfortable using Kubernetes for everything from personal hobby projects, to my own underground skunkworks at work. In fact it's a broadly interesting area of study for me, "how do devs/admins/(people at large) get into k8s" since it is such a steep learning curve, and this has all happened so fast, there is so much to unpack before one can start to feel comfortable that there isn't really that much more complexity buried behind that you haven't deeply explored already and understood.
- imron 7y ago> Long story short, it was way too much of a pain in the ass. Kubernetes has a model for how your infrastructure and services should behave. If you stray outside that model, then you'll be fighting k8s the entire way and it will be painful. If however you design your services and infrastructure to be within that model, then k8s simplifies many things (related to deployment). The biggest issue I have with k8s as a developer is that while it simplifies the devops side of things, it complicates the development/testing cycle by adding an extra layer of complication when things go wrong.