12 ms·
Is K8s Too Complicated? 
- tarr11 9y agoIf you follow that link on Twitter, there are a lot of reasonable answers to his (rhetorical?) question - "In a single tweet — can you name a technical benefit you and your team have gained by switching to Kubernetes?" "Sensible configuration, fast deployments, awesome community, and flexible control plane...among others" "single root of truth for configuration" "predictable deploys" "Standardized orchestration, which makes talent easier to find." "the capability to deploy on more than one platform, more than one cloud." "Not called in at midnight when an entire node segment went down due to hardware failure." "The ability to deploy hundreds of services within minutes."
- __float 9y agoThe linked tweet wasn't this blog post's author, and the tweet author uses she/her pronouns. Gentle reminder that people of all genders are present in tech, and we should be mindful of potential exclusivity when assuming everyone is a man :)
- tarr11 9y agoFair enough (though i was referring to the blog author, not the tweet author, but could see how that could be confusing given the nested references) The tweet author seemed sincere in their desire to answer that question, whereas the blog author seemed to be quoting it ironically (eg, he felt there are no ways to express k8s values in a tweet)
- Proven 9y agolol
- matthewmacleod 9y agoI would argue that those were benefits of switching to any containerised infrastructure with an orchestrator, though. You could probably obtain most of those benefits with Nomad or Mesos.
- rco8786 9y agoMaybe. But the question wasn’t about the advantages of k8s over other containerization systems.
- matthewmacleod 9y agoI know, you’re right of course. It’s kinda like all those “we rewrote our Ruby app in Go and it’s 1000x faster so Go is awesome” articles. Containerisation is good for those things, but it’s still possible that Kubernetes is an excessively complex solution to the problem!
- rco8786 9y agoFair enough. To your original comment, I was part of a large transition from bare metal servers and custom rolled CI, deployment, monitoring, etc over to using Mesos and the end result was infinitely more pleasurable and productive to work with as an engineer.
- empath75 9y agoI like nomad but kubernetes just has a lot more tooling available for it. With nomad you’re going to have to come up with a lot of diy solutions.
- nimbius 9y agodisclosure: im an engine mechanic with a lot of interest in Linux as a hobby. Kubernetes seems like a great alternative to stuff like openstack that seems like it requires an entire datacenter to get going properly, but I feel like the hype (k8s? really?) is outliving the reality. youre also bucking up against a problem where on smaller scales, it just seems easier to use something else. Maybe not "agile" and all that nonsense but certainly easier.
- fear91 9y agoOpenStack is used mostly for the layer beneath Kubernetes. We use it to provision the VM's/networking for the Kubernetes cluster itself. Kubernetes might not make sense for your small scale projects but it's great when used for enterprise scale microservices. It makes deployments easier, faster and more secure (with things like network policies, namespacing and RBAC).
- AndyNemmity 9y agoWe run Kubernetes under Openstack, which can deploy hosted Kubernetes. Lots of variations.
- ghba66 9y ago?
- codefined 9y agoNot sure if this was an accidental key press or something, but generally here we try to be constructive in our comments. "?" is a fairly ambiguous post to make, I'm not sure many people understand what it is you're questioning.
- vs2 9y ago!
- lardo 9y agohttps://i.imgur.com/wJoo082.png https://i.imgur.com/wJoo082.png I imagine they're wondering about that symbol.
- CraftThatBlock 9y agoThe last character in the post title is a weird Unicode character and shows as "OBJ" on my Android device. The parent coment posted "<OBJ>?" Seems like your OS can't render the character
- hacknat 9y agoRight tool for the right job. Is K8s too complicated? For some use cases it is. They probably should do a better job of discouraging certain use cases, but calling their elevator pitch “bullshit” is hyperbolic. There are exceptions to every rule, but a good rule of thumb is cluster size. If you’re managing less than 25 servers than K8s is probably over kill. As you start to creep north of 40 servers K8s really starts to shine. The other place K8s really shines is dynamic load. I manage anywhere from 600-1000 16Gb VMs, I can’t imagine doing it without K8s. If cluster size isn’t a good rule of thumb then application architectur probably is. If no one person in you company has a complete mental model of the application architecture or it is impossible because it is so complex, again container orchestration might be a good way to go. Final point: If you’re struggling with K8s swallow your pride and buy a managed solution, then learn as you go.
- pstadler 9y agoEven though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful. Shameless plug, I‘ve been working on a project explaining how to run small scale clusters in the cloud for more than a year: https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide
- ad_hominem 9y agoYou may want to update your tutorial to use cert-manager because kube-lego is in maintenance-only mode
- jaequery 9y agoIs there a link to the tweet where he asked "what did you benefit from using K8?"
- jaequery 9y agook it's https://twitter.com/krisnova/status/994242908315312129 https://twitter.com/krisnova/status/994242908315312129
- jaequery 9y agowhat are some of the downsides of K8? any horror stories?
- dilatedmind 9y agoThey recently added namespaces to specs, and I think they should add context as well. Not really a horror story, but we ran into some mysterious behavior when one of our services appeared to alternate between two difference instances of an app. It was amusing to find someone had deployed prod specs to both prod and dev clusters, The only difference being their dns. The result was a fight between the two clusters' dns controllers to sync the A record to their respective endpoints.
- fear91 9y agoYou need to hire people who are competent with a new, quickly evolving technology. That means it's hard to hire someone good and it's expensive. The upside is that you can replace multiple "normal" ops people with just a few K8 folks but that also requires a bit of culture change of how you do development.
- AndyNemmity 9y agoI started a year ago on a team, and there are so many different technologies, things to know, gotchas everywhere, that even competent people can struggle outside of taking a single domain and owning it. It is not remotely easy to actually use and understand all parts of a massive system, at least ours.
- zie 9y agolike @fear91 almost said, but didn't quite get there.. it's operationally VERY complex. It has it's reasons for being so complex, but jeez! doesn't help that tooling and docs are all over the map. Right now I'd say it's useful only if you can afford to hire a handful of very talented k8s knowledgeable people to babysit it, because it def. needs babysitting. Maybe in a few years it will be nicely stable. Otherwise you make someone like Google, Microsoft, etc run it for you. Or skip the complexities and use something operationally simple and sane like Nomad or FreeBSD Jails.
- cygned 9y agoI started with k8s beginning this year and from my point of view, the documentation is not good - and a major pain point when trying to get started. Each part on its own is good and well written, but it lacks the overall picture and does not connect pieces well enough. For example, the schema definitions for the all the configuration files are not linked from the official docs (at least I wasn’t able to find them). The description of how to get started with an On Premise setup is scattered over multiple pages from multiple tools. So from my point of view, things could be improved for beginners.
- tomsthumb 9y agoThe kubernetes up and running book gives a better big picture view than the documentation online does. It does a thorough job of developing motivation and context for using a broad swath of the system in a cohesive way.
- cygned 9y agoI actually own that book and I definitely recommend it!
- geerlingguy 9y agoThis! So many parts of the docs are excellent in isolation, but if you jump from one page on Ingress Controllers to a page on Pod replicas, to a page on something else, the pages are obviously written by different authors who use different services (e.g. Google Cloud vs, bare metal vs. Azure) and some of the docs assume you’re also using said service, but don’t explicitly call it out. And then you wonder why some command or example isn’t working the same in minikube or some test bed environment. Too much of my ‘getting up and running’ time was spent figuring out things on my own based on Stack Exchange, blog posts, and experimentation.
- Yetanfou 9y agoMight be, but one thing it certainly is not: worthy of yet another acronym. Call the thing by its name, kubernetes. T2t w3d be so m2h e4r to r2d.
- lalp1 9y agoThe hidden cost of K8S is you have to shift a lot of things: configuration management, secrets, CI/CD, monitoringn users' habits... But it's worth it :)
- jchw 9y ago>In a single tweet — can you name a technical benefit you and your team have gained by switching to Kubernetes? - Continuous delivery made easier - Code-as-infra you can deploy anywhere (Pi, GCP, AWS...) - Ability to pack VMs tighter, viewing your cluster as a pool of resources. Kubernetes may seem too complicated if you miss the point. It's throwing the baby out with the bathwater, but it's doing so with purpose. Kubernetes didn't become popular by accident. The benefit is hard to explain for the same reason that it's hard to learn: it's a complicated piece of tool that solves a problem most people don't understand. That problem is scheduling. Most people view systems administration as the practice of managing machines that run programs. Google flipped the script: they began managing programs that run on clusters, probably about as early as any other company, if not earlier. The key insight here is that with Kubernetes, you are free from the days of SSHing into machines, apt-get installing some random things, git cloning stuff, and setting up some git hook for deployments. No matter how much more advanced your process is, whether you have God's gift of a Chef script, or you have the greatest Terraform setup in the world, you're still managing boxes first, and applications second, in the traditional model. You have to repeat the same song and dance every time. To be fair, Kubernetes is not the only platform to provide this sort of freedom. Obviously, it's based on Google's famous Borg, and Docker Swarm also exists in this realm, as well as Apache Mesos. I think Kubernetes is winning because it picked the right abstractions and the right features to be part of itself. Docker Swarm did not care enough about the networking issues that came with clustering until recently. Specifically, one of the first problems becomes "What if I need multiple applications that need to be exposed on port 80?" Kubernetes IMMEDIATELY decided that networking was important, providing pods and services with their own IP addresses first thing, including LoadBalancer support early on. In my opinion, pods and services are the sole reason why Kubernetes crushed everything else, and now that other solutions are catching up and implementing better abstractions, and other solutions for networking are appearing, the problem now is the massive headstart Kubernetes had. Kubernetes let you forget about managing ports the same way you forget about managing machines. Docker Swarm wasn't offering that. Yeah, it took a few paragraphs to explain why it makes sense, but once you "get" it, it's hard to unget. That does not mean that Kubernetes is not too complicated. It's probably way too complicated for most of us, and a lighter solution with similar properties would be fine. But that doesn't mean the complexity is all a waste; it's just not useful to all of us.
- anoncoward1234 9y agoOk, I'll bite. I posted this in another thread, but check this out: https://stackoverflow.com/questions/50195896/how-do-i-get-one-pod-to-network-to-another-pod-in-kubernetes-simple https://stackoverflow.com/questions/50195896/how-do-i-get-on.... That's the amount of crap I waded through trying to rubber-ducky myself into figuring out how to get two pods to talk to each other. In the end, I copied a solution my friend had gotten, and it's still not great. I'd love to be able to use Ingress or Calico or Fabric or something to get path routing to work in Kubernetes, but unfortunately all the examples I've seen online suffer from too much specificity. Which is the Kubernetes problem - it can do everything so trying to get it to do the one thing you want is hard. Here is the kubernetes cluster I ended up building. If anyone has any ideas on how to add path routing let me know - https://github.com/patientplatypus/KubernetesMultiPodCommunication https://github.com/patientplatypus/KubernetesMultiPodCommuni...
- jchw 9y agoI think part of the problem is that I can't immediately understand what is actually being done. You say you want a per se React frontend to talk to a Node.js backend. But that's not really a pod-to-pod communication issue; both frontend and backend will be communicating with the user's browser, outside the cluster. Secondly, you deployed an Nginx ingress controller. You don't need to deploy more than one of these in your whole cluster, so you can go ahead and separate this from your program's deployment manifests. Typically, cluster add-ons are installed by running kubectl -f with a URL to a GitHub raw URL, or, if you want to be much cleaner, using Helm (basically, a package manager. It installs with one command and then you can use it to install things into your Kubernetes cluster easily, such as an Nginx ingress controller.) If you're wondering why the process is such a mess, it's probably just because Ingress is still new. In the future, support for more environments will probably come by default, without needing to install third party controllers. Already, in Google Cloud, GKE clusters come ready with an Ingress Controller that creates a Google Cloud load balancer. As a side note, I found that the nginx ingress controller was not working by default in my cluster. I noticed some errors in the logs and had to change a parameter with Helm. Don't recall what it was, unfortunately.
- 9y ago
- lifeisstillgood 9y agoSide note - i stumbled across an attempt to rebuild K8 from scratch in python (it looked like a learning attempt) - but i forgot to bookmark it - anyone seen something similar
- justinsaccount 9y agoAre you sure it was k8s, and not maybe docker? https://github.com/tonybaloney/mocker https://github.com/tonybaloney/mocker
- perlgeek 9y agoSuppose I'm an application developer who is only interested in infrastructure because there needs to be some to run my stuff. In this scenario, would I actually learn Kubernetes, or would it make more sense to go to straight to a PaaS solution? Like OpenShift (which uses k8s in the backend, I believe), or Cloud Foundry, Stackato etc. I always get the impression that k8s has a lot of good ideas, but doesn't provide everything out-of-the-box for actually deploying complex application. How true is that?
- Jyaif 9y ago> Suppose I'm an application developer who is only interested in infrastructure because there needs to be some to run my stuff. Yes, you are the perfect candidate for a PaaS solution. In general, unless you are spending thousands of dollars in infra per month, managing your own services is a waste of money.
- oso2k 9y agoFull Disclosure: I work for Red Hat Consulting in the Container and PaaS Practice, an OpenShift/Kubernetes expert group. OpenShift's value (and its open source upstream Origin) is not just that it is a PaaS. It's a PaaS that you can install and run on many different infrastructure bases: baremetal, VM, public & private cloud, and others very soon. I personally run a couple flavors on my laptop for demo and training purposes. So maybe upstream Kubernetes is not so simple. But that is why we (Red Hat) call OpenShift "Kubernetes for the Enterprise". It means we make some sensible defaults, decisions, architecture choices and make installation supportable to get you started. However, the fact that OpenShift & Kubernetes allow you to define you application architecture in reusable and portable object definitions is it's biggest benefit, IMO. And you may not need to spend thousand of dollars in infra per month to enjoy that benefit.
- pm90 9y agoYou should start with a managed solution and see if it makes sense to write/move your code in k8s. If it does, awesome, keep using the managed solution. Roll your own only when you see your traffic/load increasing your costs to prohibitive levels or if you want to do something very unique.
- 013a 9y ago"In a single tweet — can you name a technical benefit you and your team have gained by switching to Kubernetes?" "The missing step in a comprehensive OSS devops strategy, from code (git) build (Docker) test (Docker) to deploy (Kube), which enables push-button CICD like no one has gotten right since Heroku."
- barnabee 9y agoIf they'd got it as right as Heroku then deployment (or CD configuration) would be a one liner once a server is set up and scaling etc. would be configurable via a very simple web console or CLI. Unless I missed something, last time we tried k8s (admittedly ~6 months ago) it was woefully far from that goal.
- bavell 9y agoAlthough it didn't come out of the box, my deploys are now one-liners. The command builds, tests and packages my app into a docker image, then bumps the version number on the k8s deployment which triggers a rolling update of production pods. I'm very happy with my current deployment flow, mostly because it works reliability and I can forget about what it's doing under the hood.
- memeis 9y agoCouldn't agree more with - "Like a lot of other tech that has ostensibly come out of google, it will likely have at least one major source of complexity that 95% of people do not need and will not want. " We really need to understand our applications, their architecture, and then the right way to build and run them. I've talked with many people focused on k8s and losing sight of what they are trying to build and why.
- stapled_socks 9y agoHaving worked with Docker Swarm previously - Kubernetes seems like they added 5 layers of abstractions and at the same time made it feel more low-level. I think Kubernetes has all the building blocks to be great but desperately needs a simplified flow/model and UI for developers who just want to run their apps.
- haolez 9y agoI prefer to work with Swarm as well, but I bump into stability issues now and then. It’s probably not on par with Kubernetes
- Steeeve 9y agoYes. There wouldn't be so many different vendor solutions if it wasn't. I wouldn't say that the complexity isn't warranted, and it obviously can be wrangled in and once you know it it's not all that bad. Conversely, it wouldn't be as popular as it is if the value it provided wasn't clear.
- matthewmacleod 9y agoI suppose we have to assume that we should use the right tool for the right job, and all that. And I'm sure that the Kubernetes folk know what they're doing. But I definitely think that it's too complicated without a cutting-edge Kubernetes expert in place to manage it. And even then, it's just a building block for a larger system. I've tried maybe half a dozen times to get started for relatively small workloads – let's say between 5 to 50 servers. There seems to be a lack of any good documentation on how to get started – sure, I can spin up a couple of pods on a cluster or whatever, but then actually taking that to an easy-to-deploy, load-balanced, publicly-accessible infrastructure seems to be much harder. It's maybe a lack of some kind of official, blessed version of a "this is how you should build a modern infrastructure from container to deployment" guide. There are so many different moving parts that it's hard to figure out what the optimal strategy is, and I've not found much to make it easier. Maybe there's actually a niche for a nice and simple "infrastructure in a box" kind of product. I'd love something like Heroku that I could run on my own bare metal, or on AWS – the various solutions I've tried were all lacking. In the end I've fallen back to using the Hashicorp stack of Nomad and Consul. This seems to work in a much simpler way that I can actually wrap my brain around – I get a nice cluster running balanced across machines, deploying and scaling my applications as required, and it was super easy to set up with good documentation. I've been nervous about this approach because of how Kubernetes-mad the industry seems to have gone – so maybe it'll be worth another look when there are some more comprehensive solutions in place that make it easier to get started!
- pm90 9y agoI’ve worked as a developer on 2 major managed k8s providers. Using k8s is easy but operating it is not. The biggest reason is that every cloud provider is using different underlying technologies to deploy k8s clusters. See for example all the different kinds of ingress controllers.
- geerlingguy 9y agoAnd it doesn’t help that docs sometimes mention how you do things assuming you’re using Google Cloud, other times with examples for Google, AWS, Azure, or bare metal, other times minikube or some other smaller scale wrapper, and other times it’s not clear what kind of platform the commands should ‘just work’ on as-written. I had a ‘fun’ time figuring out the basics of things like ingress controllers in a bare metal setup because of this mishmash of docs.
- thdxr 9y agoYes it's too complicated. But I think it's a low level tool that most people will use via an abstraction built on top. We use Rancher which makes using kubernetes a breeze.
- tejohnso 9y ago"I suspect that a significant source of programmers flooding into management is the decreasing wavelength of full scale re-education on how to accomplish the exact same thing slightly differently with diminshing returns on tangible benefits." Wow. I'm curious to see how others are interpreting this sentence. I've got: I suspect many developers opt to move on to management largely because of frustration with having to learn new abstractions with increasing frequency and decreasing benefit.
- NTDF9 9y agoI agree. Most of software engineering has turned into resume driven development, and engineers can only keep up at it for some time. Kubernetes did not have to be so complicated. Docker swarm just proved it. If it takes 10 engineers to really program and manage your kubernetes instances, what problem did it solve again?
- bradhe 9y agoI’ve been a Mesos/Marathon user for the past few years. I love it for its simplicity. Install the agent, point it at Zookeeper, and you’re done. I’ve got a catalog of Puppet manifests for managing the cluster that I hope to publish one day soon.
- shinzui 9y agoIt depends on your context. Having moved from ECS on AWS to K8s on GCP I found kubernetes simpler despite its vast number of concepts. It has the correct abstractions for a microservice architecture, and it made us write less boilerplate code to manage and deploy services.
- jkmcf 9y agoI’ve always sorta felt they solved the problem backwards, focusing on the infrastructure and not on running applications. Now we have all these competitors/confusion around orchestrating running an app. They could have just copied Heroku’s CLI as a base starting point and ease adoption.
- deleted 9y ago[deleted]
- noonespecial 9y agoThe best tools work at multiple levels and operator skills. Imagine a tool that can do everything from pound in a nail to launch a spacecraft. In an ideal situation, a carpenter can grab it and pound some nails and a rocket scientist can launch a Mars mission with it. In the worst case, the tool forces you to learn how to launch spacecraft before you can pound nails. In my limited experience (having twice now made attempts to get up to speed and actually accomplish something with K8s) it felt an awful lot like learning rocketry to pound nails. I may have had an entirely different experience if I had come looking to do a moon shot.
- sytse 9y agoThe OP linked to a set tweets from Joe Beda (author of Kubernetes) that read "When you create a complex deployment system with Jenkins, Bash, Puppet/Chef/Salt/Ansible, AWS, Terraform, etc. you end up with a unique brand of complexity that you are comfortable with. It grew organically so it doesn't feel complex." Kubernetes is a convention over configuration framework for operations. And by doing that it allows you to easily build on its abstractions. At GitLab the Kubernetes abstractions allowed is to make Auto DevOps that automatically builds, runs tests, checks performance, diagnoses security, and provisions a review app. We could have made that with gitLab CI and bash but it would be much harder for people to understand, maintain, and extend. Having an abstraction for a deployment is something that doesn't even come by default with Jenkins as far as I know. Now that we have that as an industry we can finally build the next step of tools with the usability of Heroku but the control, price, flexibility, and extensibility of self-hosted software.
- Animats 9y agoIf you can't even spell out the name, it's too complicated.
- andyidsinga 9y agothis: $WORK Maybe the most concise statement I've seen on the meaning of employment in our times. ..really reminds me of PG's refragmentation essay http://paulgraham.com/re.html http://paulgraham.com/re.html
- simonebrunozzi 9y agoAm I the only one who thinks it should be K8S, with a capital "S"?
- dankohn1 9y agoIt's a numeronym. From the horse's mouth: https://twitter.com/dankohn1/status/982013434555523072 https://twitter.com/dankohn1/status/982013434555523072
- brown9-2 9y agoInteresting that the article never mentions the word “container” (outside of a quoted tweet). If you don’t want a cluster management system that can schedule your many containers over a pool of many nodes, then yes Kubernetes is not aiming to solve your problems.
- dsnuh 9y agoI feel like managed Kubernetes solutions are a completely different beast from on-prem physical deployments, and that when people talk about how Kubernetes isn't complicated, they are probably running a webapp on GKE or Minikube on a DO droplet, or something similar. Running Kubernetes on your own private, on-prem infrastructure - integrating services that live outside the cluster, exposing your cluster services, rolling your own storage providers, adding a non-supported LoadBalancer provider, managing the networking policies, etc., etc. - can quickly become an incredibly messy and complex endeavor.
- KaiserPro 9y agoK8s is a beast, and it has a fairly specific workflow, which for most people is not a good fit. For example you need to understand that certain workloads dont fit well on the same box (DBs and anything IO/memory sensitive for example) Then there is the "default" network setup where each node is _statically_ assigned a /24. (because macvtap + dhcp is "unreliable" and inelastic apparently.) Now, I've heard a lot of talk about you either use k8s or ssh to manage a fleet of machines. That pretty much a wrong comparison. K8s provides two things a mechanism for shared state (ie. I am service x and I can be found on ips y & z) and a scheduler that places containers on hosts (and manages health checks.) If your setup has a simple config scheme, (using a simple shared state mechanism, like a DB, or a filesystem, or DNS) or you have no issues with creating highly automated deployments using tools like CF, chef, anisble cloudformation $other, then k8s has vanishing returns (it sure as hell doesn't scale to the 50,000 node count, because its so chatty. ) Basically its a poor man's mainframe, where all the guarantees of a process running correctly regardless of what is happening to the hardware has been taken away.
- rossdavidh 9y agoWow. I guess I am not the first person to look at K8s and think, "they re-invented the mainframe".
- peterwwillis 9y agoK8s does a relatively good job at tackling a very large problem space, but it never tried to tackle the other problem space: learning and using complicated things quickly and easily. The fact that there is no tool to walk a human through building commonly reproduced configuration files is proof that humans were an afterthought.
- stackzero 9y agoThere are certainly many moving parts- but there's nothing quite like running kubectl create -f <somefile> and sitting back to watch your entire application stack spin up
- sandGorgon 9y agoK8s comes with the inherent dependence that you are using a cloud infra with a lot of the setup done right. Most importantly - ingress. Getting your ingress on k8s is still a big issue. I'm not sure about all of these statements that networking in k8s is superior to swarm. For the longest time, it was a huge mess to configure weave vs flannel vs calico. Arguably it was because of the third party implementation themselves, but then I would argue that comparing the superiority of k8s to docker swarm is an apples to oranges comparison...Since swarm's networking was always built in and opinionated. For a long time , it worked much better than k8s.
- lowbloodsugar 9y agoToo complicated for whom? We just gave up on Mesos. It has been impossible to hire a team to support Mesos, which in many ways, is simpler than k8s while delivering greater scale[1]. But it's been me and one other guy building the whole thing when we planned on having ten. We're planning on replacing it with something else, and New Guy wants that something else to be k8s. Or wanted. Then we spent an hour or so asking "So, how will you handle failure mode (that happened) X?" or "What about the teams that need Y?" and no ECS is looking real good to this guy. At the literal end of the day, when a bug causes masters to lose quorum at 2am, do you want to be fixing that, or do you want an AWS or Azure team to be fixing that. When you have a hundred apps happily running but the coordination framework is down, so that if you lose and agent, or get lots of traffic, the system that would add or replace capacity is down, how fun is that? Well, we didn't have an outage, because all the apps were up and running, but thank goodness we didn't lose an agent of get a spike in traffic. You live in constant terror, because it never gets so bad (site down), but you have lots of near death experiences where you race against time to fix whatever fun way the system failed. God forbid you upgraded to the release (of Marathon) that has a failure mode of "shut everything down" (we didn't, because I'm "overly cautious", but others did). Now for some companies, where they have the need for a huge cluster, and the cash and reputation to be able to hire for it, then Mesos or k8s seem like a great idea. But for everyone else, use ECS and have a team at AWS keep that thing running. [1] I suspect it's probably easier to hire people for k8s, and at the same time more difficult to hire people who can actually do the work. k8s looks simpler than mesos, but is far more complicated. Mesos has a smaller following, but those I've met know what they're doing and respect the problem. And Mesos didn't get everything right, but it's "righter" than k8s IMHO. There's a need for a Mesos 2. =)
- ksajadi 9y agoFor a system that needs to take care of a great deal of different use cases on a wide variety of infrastructure, k8s is surprisingly simple in context. However I also think for a great deal of “every day” use cases solutions like GKE, EKS, Rancher, Cloud 66,... are good enough.
- megaman22 9y agoYes, it's madness, unless your system is actually that complicated. And I'm not sure designing a system to the strengths of k8s is wise either.
- doxcf434 9y agoA better question would be how much simpler does k8s make thing compared to all the rube goldberg scripts admins have written over the years? A lot.