11 ms·
The only part missing in most Kubernetes write ups is that you probably don't need it. I'm hopeful most engineers recognise this, although my day to day experie
by tnolet 8y ago
The only part missing in most Kubernetes write ups is that you probably don't need it. I'm hopeful most engineers recognise this, although my day to day experience paints a darker picture. I know of at least one hot startup deploying Kubernetes just to attract engineering talent.
- stemcc 8y agoIn your experience with these deployments, which K8s features aren't needed? Is it the orchestration as such that's not necessary, or K8s approach wrt to the workloads deployed?
- tnolet 8y agoSorry for not being clear. There is nothing wrong with K8s features. It solves a specific problem really well. However, most people & companies I interact with don't have that problem. Or that problem is just a very marginal piece of their full set of problems. Running K8s, maintaining it and making your apps work with it is pretty big undertaking. Only go there if the upside is so great it blows any alternative away. I guess the popular term is "cargo culting". Or maybe Heroku is not hip anymore although it probably is the best advice for many companies.
- meddlepal 8y agoI strongly disagree. I've built a couple in house PaaS type systems in the past with various degrees of success on more classical tech like straight up EC2 using a variety of Terraform/Ansible/CloudFormation/Scripting but the dev experience has always been painful compared to Kubernetes. Developers really love working with Kubernetes. It doesn't take a lot to get started and if you're not doing anything too crazy it is a huge productivity boon IMO. Where things get hairy is persistence, but that's the case regardless of Kubernetes or not.
- auslander 8y ago>Terraform/Ansible/CloudFormation/Script This number of tools speaks for itself. Yeah, you would be better off with out of the box stuff due to lack of skills.
- meddlepal 8y agoAh yes, you're right. I am sure you have a much better understanding of the requirements each of these build outs sitting behind your keyboard over there.
- rvanmil 8y agoHeroku is still amazing indeed. Our team recently moved all Node.js based systems from AWS to Heroku and it has given us a tremendous productivity gain and reduction in complexity. Before we made that decision we spent about 8 months experimenting with K8s, only to discover it actually made things way more complex for us and reducing productivity without any clear benefits.
- outside1234 8y agoYou probably don't need the cloud either, but everyone recognizes that it is a productivity gain. To me, this is the analogous case for K8s, you can probably do the same with VMs, but everything is a lot more productive with containers and K8s.
- cc81 8y agoI don't have much practical experience with Kubernetes but I think one argument is that there are other alternatives to it that might give equal or better productivity gains for a lot of people. For example Azure App Service or equivalent from the other vendors. You don't have to manage a VM.
- slow_donkey 8y agoApp service is alright but I'd recommend Heroku over app service any day. However, there is some visible work being done to improve app service like supporting docker compose files to run multiple containers but the basics are still very painful. Most notably, swapping servers (slots) always caused downtime, accessing logs was flaky and slow, azure interface is just plain slow, continuous deploy was flaky (or it just didn't kick in until 20 min later?) and I had to manually restart servers many times
- bartread 8y ago> You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. Care to quantify that? Because we don't use the cloud (in fact, in my current role, we don't even make use of virtualisation for production infrastructure) and we have no issues with productivity. Or rather, let's be a little more honest, we have no issues with productivity that would be solved by a move to AWS/Azure/$_some_other_cloud_provider.
- outside1234 8y agoYou forgot about all the folks managing / maintaining / cabling / air conditioning all your servers. That is the productivity gain you are missing.
- brodsky 8y agowhat would you recommend to use for container orchestration and service discovery instead of Kubernetes?
- oldmanhorton 8y agoI believe the parent comment was trying to say that most people don't need service orchestration or discovery
- tnolet 8y agoYes. That is exactly what I meant. Mea culpa for not being more clear.
- brodsky 8y agono worries; that makes sense and I agree with you. (it's strange that my question resulted in downvotes).
- pacala 8y agoMy team runs a grand total of 1 server, 1 daemon and up to 5 concurrent batch jobs. Sometimes we bring up 1 extra server for experimentation. It is handy to not break the bank and use GKE autoscaling for the batch jobs. Surely we could setup some batch job manager over some raw VMs, or maybe spend some time to select and learn to use yet-another AWS service. It is handy to go to the GKE "Workloads" page and see how many servers are currently active, using how many pods. Surely we could pull that off with raw VMs, but it would take me a few minutes extra / a brittle script to pull it off. Do we _need_ Kubernetes? No. It makes us more productive? Yes. Am I confident that in the next assignment I'll be able to reuse my skills, regardless of what VM provider they use? 90% yes, pending EKS becoming usable. Echoes of "you don't need C, just use Assembly, it's simpler and faster" from early '90s gaming industry.
- firebacon 8y agoI think the recommendation is not to use another orchestration service, but to keep the infrastructure simple and not use any orchestration or service discovery at all. Adding service discovery and container orchestration will probably not make your product better. Instead it will add more moving parts that can fail to your system and make operations more complex. So IMO a "containerized microservice architecture" is not a "feature" that you should add to your stack just because. It is a feature you should add to your stack once the benefits outweigh the costs, which IMO only happens at huge scale. Most people know that "Google {does, invented} containers". What not so many developers seem to realize is that a Google Borg "container" is a fundamentally different thing from a Docker container. Borg containers at Google are not really a dependency management mechanism; they solve the problem of scheduling heterogenous workloads developed by tens of thousands of engineers on the same shared compute infrastructure. This however, is a problem that most companies simply do not have, as they are just not running at the required scale. At reasonable scale, buying a bunch of servers will always be cheaper than employing a Kubernetes team. And if you do decide to add a clever cluster scheduler to your system it will probably not improve reliability, but will actually do the opposite. Even something like borg is not a panacea; you will occasionally have weird interactions from two incompatible workloads that happen to get scheduled on the same hardware, i.e. operational problems you would not have had without the "clever" scheduler. So again, unless you need it, you shouldn't use it. I think the problem that Docker does solve for small startups is that it gives you a a repeatable and portable environment. It makes it easy to create an image of your software that you are sure will run on the target server without having talk to the sysadmins or ops departments. But you don't need kubernetes to get that. And while I can appreciate this benefit of using docker, I still think shipping a whole linux environment for every application is not the right long-term way to do it. It does not really "solve" the problem of reliable deployments on linux; it is just a (pretty nice) workaround.
- rb808 8y ago> I know of at least one hot startup deploying Kubernetes just to attract engineering talent. Interesting proposition - if it helps attract better talent does that mean they needed k8s?
- swyx 8y agoi am a mainly js engineer and so far have lived entirely within the firebase world as far as my backend goes. in your (and everyone else's) opinion, then: when -do- you need it? set some quantifiable bars for me please?
- tnolet 8y agoI could throw the "you'll know it when you see it" but I'll try not to! 1. You have a large amount of underused hardware/VM 's you want to cluster. 2. You have an eloborate microservices system with a dynamic service discovery mechanism. 3. You have the above over many regions / teams. 4. All your services are mostly stateless. 5. Your team of rockstar ops engineers are building their own crappy version of Kubernetes just to deal with the engineering challenges your business faces.
- cogman10 8y agoFunnily, my current employer is just that. We predate K8s, so it can be forgiven somewhat. Transitioning, though, is just awful. We have every reason to transition, yet getting the system setup, running, and working with the current infrastructure is a daunting task. You don't need K8s, until you do, at which point switching can be a nightmare scenario.
- swyx 8y agothanks, i'm pretty far from that world then, haha. appreciate the list of concerns, i hope to someday worry about these things.
- firebacon 8y agoWhen you notice that you spend such significant money (i.e. multiple salaries) on hardware capacity planning that employing a team to operate kubernetes might be cheaper instead. Probably not going to happen unless you run on hundreds to thousands of machines.
- barrkel 8y agoOr vmware licensing is killing you and you can't use PaaS for contractual reasons.
- slow_donkey 8y agoWhat if you're on a hosted service like gke/aks? The work needed to maintain the system has been minimal for us with a lot of positives
- bauerd 8y agoYeah +1. I was super skeptical 1,5 months ago when I started my new job on a 22 node GKE cluster. "Surely K8s is overkill", "could've just used Nomad + Consul + Vault + X", "why not just use Fargate", "K8s seems like a distributed systemd", etc pp. Long story short, K8s on GKE just works, saves us tons of time, makes service-oriented architectures/microservices easy as pie and has led to a way better service level than the previous setup (Chef deployed on baremetal). Yes, it probably is overkill for your 10req/s app, granted. But I highly advise everyone to at least give it a try out of curiosity, because a lot of hard problems at medium scale and above just go poof with K8s.
- firebacon 8y ago> because a lot of hard problems at medium scale and above just go poof with K8s. No, they don't? I don't know why anybody would just assume that something as complex as kubernetes would just run flawlessly once you actually try to run it on thousands of servers. Must be something to do with google PR because people definitely don't seem to assume the same for e.g. hadoop or openstack. Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running?
- bauerd 8y agoI was commenting on a specific usage context: >when I started my new job on a 22 node GKE cluster >problems at medium scale vs >Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running K8s obviously is not a silver bullet and of course there's ops work to be done. I can't comment on whether operating K8s clusters in other contexts makes economic sense, but I know it does for our current team.
- tlarkworthy 8y agoI don't think that's true. If you are using bash or chef to interact with the cloud then k8s can replace those with a much cleaner model. My biggest fear is helm becomes chef cookbooks
- ridruejo 8y agoCan you elaborate on the concerns with helm?
- temikus 8y agoIt’s probably referring to people using Chef cookbooks blindly without any knowledge of how to configure or maintain the service itself. I have interacted with people who run production workloads via Helm but struggle to do very basic things with kubectl (like recreating a service). This is an argument that can be used for any high level abstraction, however. I guess Chef is used since they have very good testing tools so Cookbooks are relatively more stable than other CM definitions. Until things go wrong with the service that is.
- tlarkworthy 8y agoThat community sourced cookbooks are security nightmares or under maintained. I cannot see how helm charts can stay quality without a curated app store.
- geggam 8y agoAny unmaintained infrastructure as code piece will quit working. Success I have had with chef has been generating my own cookbooks that dont try to do everything for every OS
- tlarkworthy 8y agoYes exactly. Everything tends towards rolling your own.
- ridruejo 8y ago
- bruinjoe 8y agoI see a lot of value in Kubernetes. For one thing I don't want to wake up at 1 AM if one of my node goes down and brings down with it important containers. If I scaled the environment with excess capacity then Kubernetes should start up new pods on the remaining nodes.
- tnolet 8y agoThat is an awesome feature of K8s and similar orchestators. It is however just an HA (high availabilty) problem already solved many times over in many ways. Almost every PaaS system has it. All cloud providers have it. Getting K8s (or any of its competitors) for HA is total overkill.
- kuschku 8y agoThe point is that I can rent the cheapest $10/month servers I can find, a bunch of them, throw kubernetes on that, and it’ll still be up. Same reliability as AWS, similar performance, much cheaper.
- yongjik 8y agoUnless you're renting from multiple $10/month services, your service can be only as reliable as your service provider, which is probably less than AWS.
- bruinjoe 8y agoYou hit it on the head. Weave Net is participating in the Istio project. I hope they can bring WeaveNet to Istio. That might be able to provide a multi cloud vendor service mesh.
- kuschku 8y agoBut that’s easily possible – you can use all the French hosters at once. Latency between them is practically nil because their data centers are only a few hundred meters away from each other. Sure, in case all of Paris goes down, you’re down, too. But in day-to-day usage you can have good reliability on a shoestring budget.
- indescions_2018 8y ago"Kubernetes" is an Ancient Greek word meaning "more containers than customers" https://twitter.com/srbaker/status/1002286820078571532 https://twitter.com/srbaker/status/1002286820078571532
- smnrchrds 8y agoI assume it is written in the style of "Ubuntu is an ancient african word, meaning 'I can't configure Debian."
- StavrosK 8y agoIt is, "kubernetes" is actually the root of the word "governor", with similar meaning in ancient and modern Greek. The naming of the software makes for very unfortunate conversations if you're talking to other Greeks about it.
- kbenson 8y agoThis totally could have come from DevOps Borat. [1] 1: https://mobile.twitter.com/devops_borat https://mobile.twitter.com/devops_borat
- deleted 8y ago[deleted]
- ShabbosGoy 8y agoIn that specific case, Kubernetes is approaching YAGNI territory. But as others have stated, it’s easy to get started with something like Amazon EKS and not have to worry about a thing. Edit: Put another way, give me APIs instead of config file hell.
- AndyNemmity 8y agoDepends on your definition of "need", it enables me to manage a ton of stuff across multiple dcs and regions easily. It would feel like the dark ages to be without it as this point.
- collyw 8y agoSadly I doubt it. Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively. https://www.quora.com/If-you-were-to-build-a-website-right-now-from-scratch-what-language-and-tools-would-you-use https://www.quora.com/If-you-were-to-build-a-website-right-n...
- tnolet 8y agoWow. The top answer on that one is terrible. I would advise the exact opposite. Istio!? Nice idea, but have you tried using it? That is some serious alpha stage software almost noone needs till very late in their tech maturity.