23 ms·

Reasons Kubernetes is so complex

428 points by bumbledraven 5y ago

ggm 5y ago
Not mentioned in this writeup, is the added complexity of not using IPv6 inside. IPv6 has an on-demand, probabalistic-unique unroutable address assignment model called ULA and it was made for something like this. Instead of NAT and the like, this framework could have been using semantic address models in the bits of the IPv6 address space, and delivered V6 to the edge. But they went v4 only, and acquired all the growing pains of making that work.
- szszrk 5y ago
  If Kubernetes were IPv6 it would just add complexity for newcomers. Its adoption is actually impressive, but still not really something a junior dev or admin knows. And they do know basics of IPv4. The whole article is kind of nice, but I don't think Kubernetes itself is that hard. It's large, but after initial shock it's also very logical and well documented. I think people are afraid of k8s because it forces them to actually implement concepts that they ignored and neglected. Suddenly it occurs that they did not understand how storage works (and k8s has nothing new to offer). Or what their network admins were doing when they set up their "VIP" addresses. Watched a large webinar on "kubernetes" just yesterday. It was aimed at developers, had a huge positive response. Presenter basically did not say more then 2-3 sentences on Kubernetes itself. It was all regular concepts that were with us all the time. State/stateless, synchronous/async, tracing, monitoring. It even ignored storage completely yet still people were shocked "how much usefull kubernetes they learned". Kubernetes exposes how much we knew about system architecture, as our poor designs are exposed quickly.
  - atc 5y ago
  - talos2110 5y ago
    Is there a link maybe?
    - szszrk 5y ago
      Please see my other comment. Unfortunately it's Polish only.
  - victorhooi 5y ago
    Second that - any chance there's a link to the webinar?
    - szszrk 5y ago
      Well, it was in Polish :) https://m.youtube.com/watch?v=7NXJHM5oetg https://m.youtube.com/watch?v=7NXJHM5oetg It's "How to implement microservice applications on Kubernetes". It's a cool run through many topics, but it barely touches Kubernetes or even clear microservices as I understand it. I find those topics quite universal, thus I chose that example.
  - sascha_sl 5y ago
    Yes and no. There's a tendency in common distributions to pick complex base components to support seldomly used features at the expense of turning K8s from understandable to magic. I can tell you the main modes of operation and how everything fits together with flannel. Ask me to compare Canal to Calico or explain Cilium and I have no idea anymore. These systems also don't explain themselves well either. I have to dig into the code and docs pretty deeply to see it's BGP and eBPF, two technologies I wouldn't expect your average network-literate admin to know. And don't even get me started on Istio.
    - sparsely 5y ago
      Istio has lots of support for retrying and recovering from network errors, which is fortunate because adding it to our infra introduced a lot of transient network errors
    - hedora 5y ago
      I think this complexity is part of the business model. You either pay the cloud vendor to solve this issue, or you pay enterprise vendors for storage/network licenses, etc. J2EE was similar, from what I saw. In theory, you could run the open source stuff. In practice, IBM and Oracle made $$$ selling and supporting a proprietary version that supposedly worked out of the box.
    - xemdetia 5y ago
      I think that's the other thing that's understated too because a lot of these underlying choose-your-own-adventure pieces are also in wildly different levels of maturity when it comes to observing and debug. You can spend a while pursuing one solution only to find one of the pieces simply doesn't have the functionality or it is not achievable or the big thing for me is not able to be secured in any meaningful way or it's a TODO with no sign of completion. That's still the main thing for me that gives me pause, it ends up being very unreliable when you are trying to secure it or simply put walls up around things. Finding out that your Network plugin doesn't support the thing you need is often not obvious until you have put in a massive amount of resources and work and sometimes it's not reconcilable. The idea of k8s is good, and the model is good, but the observability of features and working through the operational costs are still really raw right now and still has a lot of room for growth.
  - egeozcan 5y ago
    > still not really something a junior dev or admin knows I don't think junior devs should deal with k8s but of course, it depends on how your teams are structured. Some people think juniors should get involved in everything to make them able to choose better what they want to specialize on and I agree partially. It's just that, before you touch k8s, you better are already involved in everything, and have good communication with bigger stakeholders.
    - szszrk 5y ago
      It's very specific to the company/environment you are in. Plain Kubernetes can be clean and pleasure to work with until you hit level that "sascha_sl" is describing. On conferences I've met a few guys who considered themselves "junior/mid admins" and did all infra (k8s based) AND desktop support for devs in a 40-50 people software houses. Sounds crazy, but some people just don't complicate their lives, like larger companies love to. Some make it a silo and a complex, huge piece of internal IaaS, some just do "fly my dear bird" like on the popular comic [1] You can manage your cluster externally and have an k8s API ready for you with (some of...) popular clouds with barely any knowledge, including ingress on a public facing ip and a DNS pointing to it. A junior can surely do it. "Does it have backups? YES. So it's production ready". [1] https://www.bluelock.com/wp-content/uploads/2017/06/enterprise-vs-startup-journey-to-cloud.png https://www.bluelock.com/wp-content/uploads/2017/06/enterpri...
  - prmoustache 5y ago
    I don't see vanilla kurnetes as complex. It is the whole ecosystem around it that is complex, especially as there are no 2 setups done the same way. Also each third party operator has its own set of pros/cons/limitations/learning curve. This is the reason when you follow a tutorial about how to publish application X using helm on your local kubernetes minikube/k3s whatever setup on your own laptop usually works fine but just doesn't necessarily translate quickly to deploy a production ready working app of your own on your company's managed k8s setup.
  - raesene9 5y ago
    Kubernetes Dual-stack is enabled by default... https://kubernetes.io/docs/concepts/services-networking/dual-stack/ https://kubernetes.io/docs/concepts/services-networking/dual...
- jmillikin 5y ago
  Kubernetes isn't IPv4-only. It makes very few assumptions about the network, and can easily be configured to use IPv6 (either directly or in an overlay). I wrote a small tutorial on exactly this: https://john-millikin.com/stateless-kubernetes-overlay-networks-with-ipv6 https://john-millikin.com/stateless-kubernetes-overlay-netwo...
- mschuster91 5y ago
  > But they went v4 only, and acquired all the growing pains of making that work. Because the major clouds didn't speak IPv6 well enough, back in the time - Kubernetes dates back to 2014 after all. And from what I've read on various threads here on HN and on Reddit, all three major clouds still have weird bugs and IPv4-only services...
  - ggm 5y ago
    Yes. Another thing that they made (in my opinion) poor choices on.
- raesene9 5y ago
  Kubernetes isn't IPv4 only. https://kubernetes.io/docs/concepts/services-networking/dual-stack/ https://kubernetes.io/docs/concepts/services-networking/dual... Dual stack is stable and has been around a while. Also you don't have to use NAT even with IPv4, it's just that most CNIs implement Kubernetes networking with non-addressable pod addresses. It's perfectly possible to make service and pod addresses routable from outside the cluster.
  - motoboi 5y ago
    > Also you don't have to use NAT even with IPv4, it's just that most CNIs implement Kubernetes networking with non-addressable pod addresses. It's perfectly possible to make service and pod addresses routable from outside the cluster. This is the default on Google Cloud now. Also an option on Azure. At first I found it to be a strange decision, but then I realized the beauty of it, having pod as first class citizens of the cloud.
  - bakies 5y ago
    Yes! At home I have the pod IP space routed and can connect to pods with out any port-forwarding.
- lyind 5y ago
  We use Kubernetes IPv6-only for some time now. Using Cilium CNI (software defined network) plugin. Works great after find workarounds for some minor issues. Much less hassle than with tight/inflexbile IPv4 address space.
- withinboredom 5y ago
  I also love how the ipv6 docs say to give each pod a /64. Like wtaf is a pod going to do with that many ip addresses?
  - BlueTemplar 5y ago
    Might be for ease of use (fitting MAC addresses with spare bits) and/or security, like residential subnets are supposed to be one (/64) suffix each (with from /56 to /48 per customer).
pastrami_panda 5y ago
Didn't K8 add an autopilot or similar feature to make it easier to grok?
- tecleandor 5y ago
  Google's Kubernetes as a service (GKE) has a "simpler" autopilot mode: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview https://cloud.google.com/kubernetes-engine/docs/concepts/aut...
tuananh 5y ago
off topic but buttondown.email looks very good for doing newsletter
- ayewo 5y ago
  Agreed. In fact, I signed up last week mainly because it is one of the few tools that makes it easy to add footnotes to your writing (although as a reader, the anchor links for footnotes don't work reliably in my email client). Until I discovered that Substack also supports footnotes :).
nathell 5y ago
(Disclaimer: I have never used k8s, so cannot assess the factual accuracy of this piece.) If Kubernetes hasn’t yet incorporated this article into their documentation somewhere prominent, they should. This is an excellent, and crucial, piece of documentation: providing a mental model and explaining key design decisions. All complex systems should have a document like this one. As a newcomer, this is what I want to read first. Bravo, author. I hope you get rewarded.
- thundergolfer 5y ago
  The article is only a few hours old, so K8s devs would have to be very quick on the uptake to have it incorporated. But agree that it provides a great mental mode for K8s
- pid-1 5y ago
  I would add one more thing: K8s is complex because oci containers are complex. Running a process in isolation in Linux means dealing with many different, resource specific APIs. Unfortunately, any Linux container orchestrator will inherit the resulting complexity.
  - motoboi 5y ago
    Actually running the container is a small part of kubernetes.
    - waynesonfire 5y ago
      Agree, this is abundently clear when attempting to get similar value on a freebsd system using jails.
  - randomswede 5y ago
    Kubernetes is complex because coordinating potentially hundreds och machines with potentially tens of thousands of containers so that it all flows smoothly, expediently, and allows change is a complex problem.
- 5y ago
cturner 5y ago
""" One could imagine a very imperative “cluster operating system,” like the above, which exposed primitives like “allocate 5 CPUs worth of compute” or “create a new virtual network,” which in turn backed onto configuration changes either in the system’s internal abstractions or into calls into the EC2 API (or other underlying cloud provider). """ There is a name for this: Single System Image. There is no need to write the operating system layer - Linux works well. I built a SSI. Each grid can host several apps, 'gridapps'. As priorities change during the day, an operator can rebalance resources using a bash-like shell. Each gridapp has (1) core nodes, which need to keep running, and (2) scale nodes which can be started and stopped. Nodes can be configured so they are taskset-bound to an individual processor, or sharing a processor between several nodes. There are equivalent primitives for memory and persistent storage. Commercial, 1000 cores. The author contrasts this with K8s by saying that K8s is declarative. This is not the important distinction. My system had declarative configuration, and did a fitting at startup. The key difference is motive. The design above is the natural choice when you build a grid computer from first-principles. K8s is designed to solve a different problem: scaling up the traditional model of programming. Here, a developer writes an algorithm on their workstation. The deployment circumstances and interactions with other systems are an afterthought. Often this developer does not deploy the code themselves, this is done by another team who don't understand the internals of the code.
- pjmlp 5y ago
  We don't need Linux, a type 1 hypervisor for Cloud infrastructure is good enough.
  - cturner 5y ago
    Question about drivers. If you had DMA-oriented/kernel-bypass network cards or wanted to resource-manage GPUs, would a type 1 hypervisor be able to support this?
    - pjmlp 5y ago
      I don't see why not, type 1 hypervisors are the modern version of what started on System/360, naturally depending on specific one the APIs for such stuff will be different.
lbriner 5y ago
I think it is related to but not the same as the OP but to me, something that you don't think of much is that this was invented by Google for Google-scale issues. What does this mean? Lots of things that would be relatively easy for Google make it harder for much smaller companies. Logging? Google can build a log aggregator. Hardware? Google have 100s of 1000s of servers. Documentation? Originally, certain knowledge could be kept in small teams who worked on it, the documentation then had to be generated after-the-fact, which is perhaps why the docs are quite factual and not so good at introducing some of the concepts. You want GUI tools for setting up e.g. rbac and certs? Great but Google don't because they have people who can write the relevant automation scripts in their sleep so don't need what us mere mortals might prefer when new to it. I do like it though and use it for some of our systems.
- motoboi 5y ago
  Although people love to think of Kubernetes as “the thing google runs on that was open sourced”, it is actually a simplified version, written from scratch, based on the same principles of the tool google actually runs on. On used to run on, because it seems that there is no public information about the new platform after Borg. As a comparison, Mesos was built with the same goal (open source version inspired on Borg) and made some design choices Kubernetes haven’t. UPDATE: - I don’t really know why Google spent the effort on building Kubernetes, but I believe it was for Google Cloud. And it needed to be open sourced to form a community.
  - ayewo 5y ago
    Your last sentence with respect to Apache Mesos led me to this previous HN discussion: Apache Mesos to be moved to Attic [1]. 1: https://news.ycombinator.com/item?id=26713082 https://news.ycombinator.com/item?id=26713082
  - menage 5y ago
    > On used to run on, because it seems that there is no public information about the new platform after Borg. I left Google quite a few years ago, but my impression is that Borg is still going strong for many of Google's services. There was an attempt to replace it with a new system named Omega, but that ended up being more of an experiment - some of the findings of which were rolled back into future Borg development.
Splizard 5y ago
Let me SSH into a cluster and navigate the configuration database like a virtual filesystem. Make it feel more like a traditional operating system. That will make Kubernetes more approachable.
- withinboredom 5y ago
  I was thinking the exact same thing. Something like /proc or /dev.
- pjmlp 5y ago
  Without looking into it, I bet there are companies on CNCF selling exactly that.
- wbl 5y ago
  Plan 9 was made for this.
- tedd4u 5y ago
  Try k9s https://github.com/derailed/k9s https://github.com/derailed/k9s
- motoboi 5y ago
  What you just said is fun, because that is a design choice in Kubernetes. You DO have a virtual hierarchical configuration. It’s just that it’s not a txt file, it’s a json object. The whole Kubernetes things is actually that: a single json defining a supercomputer. Changes to the json become changes in the supercomputer. The whole complexity aims just that: make that task easy. You have a lot of tools to interact with that json, but you actually only need a http client, because it is exposed as an API.
- throwaway984393 5y ago
  What we need is a real distributed OS.
smitty1e 5y ago
I explain qualitativley to people that the 'cloud' provider is doing a lot on the control plane below the hypervisor. K8s is a 'puff' within the 'cloud', and a bunch of additional control plane activity going on below the 'cloud cover' of the hypervisor line is now yours. Happy days have arrived with the added IP managent, scaling, DNS, &c requirements.
Traubenfuchs 5y ago
There is no reason for kubernetes not being just as simple as heroku or its many alternatives, none. The only reason I‘d believe is that it‘s a conspiracy for long lasting job security for anyone in devops. They managed to shit out an application server even more complex than enterprise websphere, which is both fascinating and abhorrent. Same for terraform and all the other overengineered cloud crap companies pay their devops wizards for.
- Sebb767 5y ago
  A lot of easy things suddenly become very complex if you need to deal with high availability and/or a lot of distributed computers.
  - onlyrealcuzzo 5y ago
    Like counting distributed streaming data with high accuracy and low latency - or especially calculating quantiles - for example.
  - goodpoint 5y ago
    Not at all. I worked on HA and distributed systems for 10+ years and the best distibuted designs are by far the simplest.
- k8sToGo 5y ago
  Why are you so bitter about devops people?
  - Traubenfuchs 5y ago
    I am not bitter about them, personally. If I could pick one area to work in, in the field of the technical side of software engineering I would go for devops. I applaud everyone who earns their money with this dreck! But taking a step back, it's madness. I have seen mindbogglingly complex setups 10 years ago, when I had to go begging to the old websphere greybeards when I needed anything sysadmin/devops related, yet in that time it has gotten worse. 4 of the 4 kubernetes projects I have seen personally involved highly customized code for deployment and infrastructure, that wasn't comprehensible for anyone besides the relevant devops teams. From the perspective of efficiency and the company owner(s) it appeared borderline malicious.
viveksingh 5y ago
As a summary are you trying to say, its hard to debug problems on Kubernetes if you have deployed a controller/operator that is not well written. If yes, does that mean Kubernetes is complex (i agree k8s is complex).
ulzeraj 5y ago
Am I wrong when I say this type of opinion normally comes from infrastructure personal? I say that because I've never seen a standard so easy and seamless for a developer to get his code running.
- nightpool 5y ago
  Very seriously: Do yourself a favor and try out a real PaaS like Heroku sometime. Running a real, production-ready system on Heroku is truly as easy as "git push", and the concepts are way, way simpler to understand. It's not going to be the right fit for everybody, but as far as just pure devex goes, it's truly excellent.
  - zerkten 5y ago
    People have mentioned here that a Heroku-like layer would be incompatible with k8s. Based on the description of k8s as a distributed OS, I don't understand how to reconcile these things. I'd want to deploy a k8s infrastructure, but have an easy experience on top for developers that might not maximize k8s use initially, but allow devs to move down a layer eventually. The whole time the ops folks would be running k8s and limited layer above that'd keep devs in a particular lane. Isn't this feasible?
    - q845712 5y ago
      I think several companies are trying to sell exactly that. if you look up VMWare's "TAP" offering it's attempting to provide a wide umbrella of components that move in that direction. Others in the space include Redhat, Replicated, Rancher. A few individual components that try to bring back the ease of rapid/easy deploys include: Flux, Tilt, Argo, kapp-controller. They all have slightly different priorities and make different compromises, but anyways just to assure you there's quite a few people trying to build and sell the layer you're describing.
- Sebb767 5y ago
  > I say that because I've never seen a standard so easy and seamless for a developer to get his code running. You deploy using extremely long YAML files, your app interacts with a lot of complex software (secret storage, ingress controller etc) and accessing your app for debugging is very non-straightforward. Once you understand the concepts and stack it's great, but when you're coming from locally running your app or even docker, it's as opaque as it gets. In my experience, it's actually worse for developers, because infra people are forced to learn k8s as direct part of their job, while for developers it's more of a sidenote.
ikrenji 5y ago
i don't think k8s is particularly difficult to use/understand. at least from app developer perspective. maybe setting it up and operating it is a different story, but deploying apps to it is fairly simple.
nuker 5y ago
We moved all our apps to AWS ECS. Developers happy. Kube was installed by hired consultants, after they left it was real struggle. Then we moved to ECS.
- StreamBright 5y ago
  I did the same for every project where k8s was used and started with ECS where k8s was proposed. Even ECS is too complex for my taste, but who am I to dictate what clients want.
- k8sToGo 5y ago
  ECS or EKS?
  - nuker 5y ago
    ECS. EC2 and Fargate types.
- ericbarrett 5y ago
  I am a huge fan of ECS. It just works. You have to write a tiny bit of tooling, but it's the kind of glue code you'd probably write on top of kube anyway. Failures are straightforward and visible. There's no debugging layer after layer of network redirects. It's much harder for an errant config to "helpfully" punch a hole through your firewall. You get the same uptime and scalability benefits as k8s or EKS. Everything is still a container.
- d0gsg0w00f 5y ago
  I agree. I don't understand why you would use k8s on AWS. It's an abstraction on top of an abstraction doing nothing but adding complexity. If your k8s pods have to interact with S3 buckets, IAM profiles, and load balancers anyway then you can't ignore the platform. It's just somebody chasing the cloud agnosticism fallacy or doing resume driven development.
  - kgeist 5y ago
    We have to support cloud providers in other countries due to local data protection laws (not every country has an AWS data center); doesn't it make sense to make your infrastructure management more-less uniform no matter where it's hosted? In some countries, providers may have only a single availability zone (in reality), or they can get very unstable from time to time (everything is down and god knows for how long), so it's natural for us in such countries to migrate our whole production between completely different providers several times a year. As an ordinary engineer whose job sometimes involves investigating problems in production or configuring something here and there (I'm not a full-time DevOps or SRE myself), I don't have to learn the details of whatever is the current cloud provider this week in the affected country N, I access the production system through a familiar interface which looks more-less same everywhere (k8s), including our test environments, too. Am I missing something? I am pretty curious what "cloud agnosticism fallacy" means. How would you do it differently?
nova22033 5y ago
When people ask for something to be lightweight, what they really mean is "keep the things I need and remove the things I don't..." A lot of these "why you don't need kubernetes" articles should really be "do you really need containers for your 10 user app?"
- chrisandchris 5y ago
  I think one aspect your reasoning does not properly respect is that running things in container is not all about scaling, but also about easy of deployment. A 10-user-app probably doesn't need k8s, but the advantage of packing it into a container and just starting that container (on a single machine) is very nice. It's so easy to have multiple apps with different languages (or even language versions) on a single host without worrying how to install 3 different PHP versions on that machine.
  - stepbeek 5y ago
    Perhaps as a result of incorrectly classifying K8S or K8SaaS as a PaaS when it’s very far from it. Heroku offers the same - or better - devex without the operational burden.
  - kasey_junk 5y ago
    It’s worth noting that advantage (and in my belief the large adoption of containers generally) is only valuable to interpreted languages that dont allow static linking. For statically compiled binaries I’ve not worried about different versions of libraries for my binaries for 20 years (and the technology is decades older than that). I do appreciate the docker file format but largely because it’s become used for true virtualization systems like firecracker. Plain Linux “containers” largely offer me very little.
    - pjmlp 5y ago
      Worse, except for stuff like what was being done at Xerox, static linking was the only way for compiled languages. Also on desktop systems, embedding resources into the binaries exists since the early 1980's, or just deliver every in the same directory like NeXTSTEP bundles, or what is called xcopy deployment on the PC world on homage how we used to do it on MS-DOS days.
robertlagrant 5y ago
I think a reasonable design approach is to consider Kubernetes an application management system, just as Postgres is a database management system. Put your stateless pods in there, wire them together with DNS, and inject config into them. Having that philosophy, and optimising that case so it's easy for coders to deploy into it is a big win IMO.
dustinmoris 5y ago
Actually Kubernetes is not complex. The requirements of most software developers are complex. Running small self contained and isolated applications in a highly scalable manner which can scale automatically, where dead services get killed and replaced, where health checks, readiness and liveliness checks fully automate a lot of dev ops problems, where things can be load balanced publicly as well as privately in a private VPN and where complex ingress rules can be created to route traffic from one load balancer to different smaller load balancers which distribute traffic across different apps and with automatic SSL certificate management for public services via an ACME provider are VERY COMPLEX requirements. If you want to run your stuff in such an advanced fashion then Kubernetes makes it EXTREMELY SIMPLE. If you don't care about these things then don't complain about Kubernetes. Just run your shit on a single VM and deal with everything manually as and when you need it.
torginus 5y ago
Just a funny thought exercise I did recently - I looked at our sizeable kubernetes cluster of 30ish VMs, and added up the core counts. It added up to 98. You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server.
- jordanthoms 5y ago
  I guess it's all relative, our largest cluster scales up to around 3000 VCPU and there's plenty of much larger ones out there. You might be able to get better utilisation of your VMs by running a smaller number with more cores on each - I'd try 16vcpu nodes.
  - randomswede 5y ago
    At my previous workplace, one of the clusters I herded was 60-odd (physical) machines, 72 or 96 cores (honestly can't recall at this point) and 384 GB RAM (IIRC) per physical server. But, that place was a wee bit strange, in that we ran all our kubernetes in-house and started on bare metal. And, yes, as a general recommendation, I would tend towards bigger, rather than smaller, nodes. There's a trade-off to be had, but you want each node to be at LEASt as large as the largest pod you intend to deploy, and you probably want at least some spare, on top of that.
- darkwater 5y ago
  So if that host goes down you are screwed? Would not be a wise decision from your cloud provider. Although thinking about it twice, usually cloud providers response in such a case would be a shrug and that's all...
  - KptMarchewa 5y ago
    I very doubt that would happen. Even if, at the very least, control plane would be distributed, so host going down would just cause another underlying VM to spawn and everything being recreated.
codeflo 5y ago
None if that explains why it’s so, how do I put it, fiddly? A flexible tool could still have reasonable defaults. If you rent a managed K8s cluster from AWS, you’re of course fine, but why does deploying it yourself have to be so complicated that the recommendation is now to use a “Kubernetes distribution” like K3s. Why can’t it just be a daemon you install using standard mechanisms?
- lbriner 5y ago
  Probably because it was designed for scale and big companies that can afford lots of hardware don't need to invest on standalone instances.
- KptMarchewa 5y ago
  There are 3 node clusters on RPi and 15000 node bare metal clusters. For which will you optimize?
  - codeflo 5y ago
    First of all, are you sure there isn't a way to improve the ergonomics of one without harming the other? Second of all, to the extent that there is a tradeoff, this is essentially a solved problem in every other kind of daemon (from web servers to databases). Make the simple case secure and simple, while making the complicated case (which will require a lot of custom configuration and reading the documentation anyway, not matter what you do) as straightforward as possible.
- pas 5y ago
  Because it was quickly thrown together in an experimential/evolving language, in a design by committee way, by a very non-diverse group. And it's getting much better at ergonomics/UX/DX (go is also getting much better), but it's very similar to OpenStack. (Python 2, resource hog, impossibly fragile, basic setup works, but uselessly incomplete without certain critical 3rd party components - eg. without Ceph and ovs it was just too big of a mess, similarly the CNI stuff and the LoadBalancers for k8s are the important missing batteries.) That said both microk8s (snap packaged by Canonical) and kind (kubernetes in docker) are turnkey solutions. k3s is more of a vanguard :) .. also, it can be (is already) packaged into a simple RPM/DEB easily, like the GitLab Omnibus.
tardigr8 5y ago
> "you will not, in general, see that error at creation time. " I agree, this was daunting for me too. I use werf.io now (an opensource project which I am not affiliated with, I am just a user). It deploys my resources and waits to validate whether the deploy succeeded. They call it "giterminism". You can do this yourself with scripts for sure, but I have not needed to extend any of their built in functionality yet. Highly recommended for anyone using k8s with CD. https://github.com/werf/werf https://github.com/werf/werf
throwaway984393 5y ago
Kubernetes is so complex for 2 reasons: 1. A distributed system with as many design considerations as K8s will always be highly complex. 2. K8s was poorly designed by committee by people who didn't really understand how to build what they were building. It still lacks things like true multitenancy (I'm aware of the SIG) and its security model is a farce. Everything from its configuration format to its RPC to its roles and beyond are unnecessarily complex and painful. It should have been redesigned twice over by now but that will never happen at this point. The time is right for someone to write a Kubernetes replacement and disrupt the lumbering joke that is the K8s ecosystem.
- goodpoint 5y ago
  Part 1 is not true. You can implement consensus, leader election, DHTs using few simple and small libraries.
  - throwaway984393 5y ago
    Those are only 3 of like 50 different design requirements of k8s. It really is a ton of functionality when you implement it all, and how it all has to combine together is quite complex. Just trying to keep a mental model of the actual execution paths to run a distributed microservice using all of k8s is more than most people could reason about without flow charts. But that's no excuse for poor design, which is what we got
streamofdigits 5y ago
why is explaining why kubernetes is so complex, so complex? could somebody ELI-RP? (start with a collection of raspeberry pi's sitting on my desk and take it from there)
- goodpoint 5y ago
  Excellent question. Because it has plenty of unnecessary complexity.
  - streamofdigits 5y ago
    it may be. can't really tell until somebody maps it to a universe that (I think) I understand... but I suspect this seeming to be a hype of sorts within certrain tech circles (with associated professional incentives etc) there aren't many around that would have both the in-depth knowledge and the desire to really de-obfuscate and break it down
    - p_l 5y ago
      Because it's taking care of a complex problem with lots of necessary complexity (though how complex it seems depends on whether you learn it bottom up or top down, IMO)
    - goodpoint 5y ago
      > hype of sorts within certrain tech circles The downvotes on my comments and the complains are telling.
      - biorach 5y ago
        Telling you that you don't know what you're talking about.
        goodpoint 5y ago
        That's what you want to believe.
deleted 5y ago
[deleted]
ranrotx 5y ago
I personally feel like Kubernetes tries to be all the things, which creates feature and complexity bloat to be able to accommodate any possible workload you can throw at it. “Cloud Native” is a bit of a misnomer. It’s more like trying to run your own cloud versus relying on a service provider. Sure you can run a stateful, highly available database on k8s. But you’re going to have to manage the complexity of making it work on k8s. Or you could just use a managed database service of your cloud provider (which has existed even before k8s).
zimbatm 5y ago
"Kubernetes is a cluster operating system" makes sense. Some of the complexity of Kubernetes also feels like the early days of Linux where you had to basically build your own Linux distribution. Even with managed solutions like EKS, you still need to install and learn about cert-manager, how to pass the IAM credentials around, monitoring kinda sucks so you want to replace it, ...
- zozbot234 5y ago
  Note that there are other cluster operating systems which make different tradeoffs than Kubernetes. Some of these are true Single System Image (SSI), but even something as comparatively simple as Plan9 is closer to the SSI end of the scale than the less hands-on, more 'declarative'-focused k8s.
ay 5y ago
I would love to see an article from the same author about nomad, and comparison between the two…
mdtrooper 5y ago
Has Kubernetes some code of java or angular? (end of joke) I remember somebody tell me that the problem of YaCy (a free software alternative to google search) was made on java.
stillbourne 5y ago
I have an axiom I mention on almost every occasion I can when people ask for more granularity on a specific request. "A system that is infinitely configurable is infinitely complex."
fartcannon 5y ago
Complexity slows down future potential competition.
lowbloodsugar 5y ago
Kubernetes is one of those products designed to appear simple and easy to get started with, to drive adoption. Only later does one find how unavoidably complex running such a system is (usually when something awful happens and nobody knows how to fix it). In that respect I miss Mesos: it was in-your-face complex and as a result had far fewer (zero for me) surprises.
eweise 5y ago
I have barely any kubernetes experience but what struck me is that it seemed like a lot of interfaces that need to be implemented for different environments. So you need to know a lot about those specific implementations.
dekhn 5y ago
I wasn't a big fan of k8s when I first started using it (note: I was on the borg team at google when omega was being built, and know the original k8s developers from that time). When I started using it it was still pretty unstable (this was at least 5 years ago). Since then I've been using it more heavily and all I can say is this will likely be the system I spend most of my cluster work in for the next 10-20 years (after that I'll be retired!). It's enough like borg that I can translate over nearly everything, it's not broken in any fundamental way (although there are some serious bugs from 2015 that I never expect to get fixed). One thing about borg that was always controversial was the scripting language created to submit and update jobs (bcl/gcl). It's quirky but I have not seen a better system for inheriting and updating templates. Even so I'm glad the k8s authors force you to use a simple config language with a minimum (or no) logic in it.
- breakingcups 5y ago
  Could you elaborate on the serious bugs you think are unlikely to get fixed? I think they might help me understand the current limits of k8s better.
jensensbutton 5y ago
Kubernetes is complex because it was built for "The Datacenter as a Computer" http://www.cs.yale.edu/homes/yu-minlan/teach/csci599-fall12/papers/dccomputer.pdf http://www.cs.yale.edu/homes/yu-minlan/teach/csci599-fall12/... (PDF)
ted_dunning 5y ago
The posters here who are commenting on what should be (often largely what Kubernetes already is) would do well to read the paper about the Omega scheduler. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41684.pdf https://static.googleusercontent.com/media/research.google.c... It is almost the rule that none of us has the breadth of experience to understand all of the problems that need to be solved. In fact, an explicit goal of Kubernetes is that almost all users should not need to understand much of how the system actually works. Frankly, most users have neither the time, inclination or intellectual capacity to understand more than a tiny part of the systems they work on. That includes most people who say they do understand those systems completely ... a few short questions can generally expose how little they know about important aspects. That ignorance is a measure of success, btw, not failure.
- simonw 5y ago
  The problem here is the law of leaky abstractions: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a... Systems that abstract away the complexity for you work fine up to a point... but inevitably you'll end up debugging something caused by a lower layer than the one you are familiar with, at which point you need to dig in and figure out what that abstraction is doing for you. It's not possible to understand everything, but the more you understand the more productive and effective you'll be. I see my overall career partly as an ongoing quest to deepen my understanding of the various abstractions I rely on.
- bogomipz 5y ago
  Wasn't Omega the successor to Borg? I'd be curious what's the status of it today at Google?
  - dekhn 5y ago
    I believe there are several public articles what happened. Some omega features were folded back into borg, omega development pivoted to k8s development. Most of google still runs on borg/internal but more and more people are moving projects to GCP-hosted cloud. That only really applies for things that can run Outside, and for which k8s is "good enough". Borg will continue to exist, roughly in its currently for, for at least 20 more years if Google doesn't get shut down by DOJ. It would simply not be cost effective to migrate large parts of GOogle's infra off Google borg/internal. In my experience, absolutely nothing in the world matches the experience inside of google; enormous number of enormous clusters and gobs of networking within and between them, plus a heavy set of existing workloads constantly testing it, all while people constantly deploy new versions of borg, filesystems, database and applications from a single codebase. Just as importantly, there was a massive culture of cluster nerds (SRE) that ran the clusters very efficiently, but also with care and attention to a wide range of users, including ones that weren't on Big Teams. I never, ever would have been able to build and run exacycle, an idle cycle harvester that provided at least 4 significant contributions to science.
f23ojoi 5y ago
tomerbd 5y ago
Declarative = goodFunctionName(Imperative)
akagusu 5y ago
Kubernetes is so complex because vendors can make huge piles of money selling solutions to handle the complexity.

428 points by bumbledraven 5y ago

ggm 5y ago

Not mentioned in this writeup, is the added complexity of not using IPv6 inside. IPv6 has an on-demand, probabalistic-unique unroutable address assignment model called ULA and it was made for something like this. Instead of NAT and the like, this framework could have been using semantic address models in the bits of the IPv6 address space, and delivered V6 to the edge. But they went v4 only, and acquired all the growing pains of making that work.

szszrk 5y ago
If Kubernetes were IPv6 it would just add complexity for newcomers. Its adoption is actually impressive, but still not really something a junior dev or admin knows. And they do know basics of IPv4. The whole article is kind of nice, but I don't think Kubernetes itself is that hard. It's large, but after initial shock it's also very logical and well documented. I think people are afraid of k8s because it forces them to actually implement concepts that they ignored and neglected. Suddenly it occurs that they did not understand how storage works (and k8s has nothing new to offer). Or what their network admins were doing when they set up their "VIP" addresses. Watched a large webinar on "kubernetes" just yesterday. It was aimed at developers, had a huge positive response. Presenter basically did not say more then 2-3 sentences on Kubernetes itself. It was all regular concepts that were with us all the time. State/stateless, synchronous/async, tracing, monitoring. It even ignored storage completely yet still people were shocked "how much usefull kubernetes they learned". Kubernetes exposes how much we knew about system architecture, as our poor designs are exposed quickly.
- atc 5y ago
- talos2110 5y ago
  Is there a link maybe?
  - szszrk 5y ago
    Please see my other comment. Unfortunately it's Polish only.
- victorhooi 5y ago
  Second that - any chance there's a link to the webinar?
  - szszrk 5y ago
    Well, it was in Polish :) https://m.youtube.com/watch?v=7NXJHM5oetg https://m.youtube.com/watch?v=7NXJHM5oetg It's "How to implement microservice applications on Kubernetes". It's a cool run through many topics, but it barely touches Kubernetes or even clear microservices as I understand it. I find those topics quite universal, thus I chose that example.
- sascha_sl 5y ago
  Yes and no. There's a tendency in common distributions to pick complex base components to support seldomly used features at the expense of turning K8s from understandable to magic. I can tell you the main modes of operation and how everything fits together with flannel. Ask me to compare Canal to Calico or explain Cilium and I have no idea anymore. These systems also don't explain themselves well either. I have to dig into the code and docs pretty deeply to see it's BGP and eBPF, two technologies I wouldn't expect your average network-literate admin to know. And don't even get me started on Istio.
  - sparsely 5y ago
    Istio has lots of support for retrying and recovering from network errors, which is fortunate because adding it to our infra introduced a lot of transient network errors
  - hedora 5y ago
    I think this complexity is part of the business model. You either pay the cloud vendor to solve this issue, or you pay enterprise vendors for storage/network licenses, etc. J2EE was similar, from what I saw. In theory, you could run the open source stuff. In practice, IBM and Oracle made $$$ selling and supporting a proprietary version that supposedly worked out of the box.
  - xemdetia 5y ago
    I think that's the other thing that's understated too because a lot of these underlying choose-your-own-adventure pieces are also in wildly different levels of maturity when it comes to observing and debug. You can spend a while pursuing one solution only to find one of the pieces simply doesn't have the functionality or it is not achievable or the big thing for me is not able to be secured in any meaningful way or it's a TODO with no sign of completion. That's still the main thing for me that gives me pause, it ends up being very unreliable when you are trying to secure it or simply put walls up around things. Finding out that your Network plugin doesn't support the thing you need is often not obvious until you have put in a massive amount of resources and work and sometimes it's not reconcilable. The idea of k8s is good, and the model is good, but the observability of features and working through the operational costs are still really raw right now and still has a lot of room for growth.
- egeozcan 5y ago
  > still not really something a junior dev or admin knows I don't think junior devs should deal with k8s but of course, it depends on how your teams are structured. Some people think juniors should get involved in everything to make them able to choose better what they want to specialize on and I agree partially. It's just that, before you touch k8s, you better are already involved in everything, and have good communication with bigger stakeholders.
  - szszrk 5y ago
    It's very specific to the company/environment you are in. Plain Kubernetes can be clean and pleasure to work with until you hit level that "sascha_sl" is describing. On conferences I've met a few guys who considered themselves "junior/mid admins" and did all infra (k8s based) AND desktop support for devs in a 40-50 people software houses. Sounds crazy, but some people just don't complicate their lives, like larger companies love to. Some make it a silo and a complex, huge piece of internal IaaS, some just do "fly my dear bird" like on the popular comic [1] You can manage your cluster externally and have an k8s API ready for you with (some of...) popular clouds with barely any knowledge, including ingress on a public facing ip and a DNS pointing to it. A junior can surely do it. "Does it have backups? YES. So it's production ready". [1] https://www.bluelock.com/wp-content/uploads/2017/06/enterprise-vs-startup-journey-to-cloud.png https://www.bluelock.com/wp-content/uploads/2017/06/enterpri...
- prmoustache 5y ago
  I don't see vanilla kurnetes as complex. It is the whole ecosystem around it that is complex, especially as there are no 2 setups done the same way. Also each third party operator has its own set of pros/cons/limitations/learning curve. This is the reason when you follow a tutorial about how to publish application X using helm on your local kubernetes minikube/k3s whatever setup on your own laptop usually works fine but just doesn't necessarily translate quickly to deploy a production ready working app of your own on your company's managed k8s setup.
- raesene9 5y ago
  Kubernetes Dual-stack is enabled by default... https://kubernetes.io/docs/concepts/services-networking/dual-stack/ https://kubernetes.io/docs/concepts/services-networking/dual...
jmillikin 5y ago
Kubernetes isn't IPv4-only. It makes very few assumptions about the network, and can easily be configured to use IPv6 (either directly or in an overlay). I wrote a small tutorial on exactly this: https://john-millikin.com/stateless-kubernetes-overlay-networks-with-ipv6 https://john-millikin.com/stateless-kubernetes-overlay-netwo...
mschuster91 5y ago
> But they went v4 only, and acquired all the growing pains of making that work. Because the major clouds didn't speak IPv6 well enough, back in the time - Kubernetes dates back to 2014 after all. And from what I've read on various threads here on HN and on Reddit, all three major clouds still have weird bugs and IPv4-only services...
- ggm 5y ago
  Yes. Another thing that they made (in my opinion) poor choices on.
raesene9 5y ago
Kubernetes isn't IPv4 only. https://kubernetes.io/docs/concepts/services-networking/dual-stack/ https://kubernetes.io/docs/concepts/services-networking/dual... Dual stack is stable and has been around a while. Also you don't have to use NAT even with IPv4, it's just that most CNIs implement Kubernetes networking with non-addressable pod addresses. It's perfectly possible to make service and pod addresses routable from outside the cluster.
- motoboi 5y ago
  > Also you don't have to use NAT even with IPv4, it's just that most CNIs implement Kubernetes networking with non-addressable pod addresses. It's perfectly possible to make service and pod addresses routable from outside the cluster. This is the default on Google Cloud now. Also an option on Azure. At first I found it to be a strange decision, but then I realized the beauty of it, having pod as first class citizens of the cloud.
- bakies 5y ago
  Yes! At home I have the pod IP space routed and can connect to pods with out any port-forwarding.
lyind 5y ago
We use Kubernetes IPv6-only for some time now. Using Cilium CNI (software defined network) plugin. Works great after find workarounds for some minor issues. Much less hassle than with tight/inflexbile IPv4 address space.
withinboredom 5y ago
I also love how the ipv6 docs say to give each pod a /64. Like wtaf is a pod going to do with that many ip addresses?
- BlueTemplar 5y ago
  Might be for ease of use (fitting MAC addresses with spare bits) and/or security, like residential subnets are supposed to be one (/64) suffix each (with from /56 to /48 per customer).

pastrami_panda 5y ago

Didn't K8 add an autopilot or similar feature to make it easier to grok?

tecleandor 5y ago
Google's Kubernetes as a service (GKE) has a "simpler" autopilot mode: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview https://cloud.google.com/kubernetes-engine/docs/concepts/aut...

tuananh 5y ago

off topic but buttondown.email looks very good for doing newsletter

ayewo 5y ago
Agreed. In fact, I signed up last week mainly because it is one of the few tools that makes it easy to add footnotes to your writing (although as a reader, the anchor links for footnotes don't work reliably in my email client). Until I discovered that Substack also supports footnotes :).

nathell 5y ago

(Disclaimer: I have never used k8s, so cannot assess the factual accuracy of this piece.) If Kubernetes hasn’t yet incorporated this article into their documentation somewhere prominent, they should. This is an excellent, and crucial, piece of documentation: providing a mental model and explaining key design decisions. All complex systems should have a document like this one. As a newcomer, this is what I want to read first. Bravo, author. I hope you get rewarded.

thundergolfer 5y ago
The article is only a few hours old, so K8s devs would have to be very quick on the uptake to have it incorporated. But agree that it provides a great mental mode for K8s
pid-1 5y ago
I would add one more thing: K8s is complex because oci containers are complex. Running a process in isolation in Linux means dealing with many different, resource specific APIs. Unfortunately, any Linux container orchestrator will inherit the resulting complexity.
- motoboi 5y ago
  Actually running the container is a small part of kubernetes.
  - waynesonfire 5y ago
    Agree, this is abundently clear when attempting to get similar value on a freebsd system using jails.
- randomswede 5y ago
  Kubernetes is complex because coordinating potentially hundreds och machines with potentially tens of thousands of containers so that it all flows smoothly, expediently, and allows change is a complex problem.
5y ago

cturner 5y ago

""" One could imagine a very imperative “cluster operating system,” like the above, which exposed primitives like “allocate 5 CPUs worth of compute” or “create a new virtual network,” which in turn backed onto configuration changes either in the system’s internal abstractions or into calls into the EC2 API (or other underlying cloud provider). """ There is a name for this: Single System Image. There is no need to write the operating system layer - Linux works well. I built a SSI. Each grid can host several apps, 'gridapps'. As priorities change during the day, an operator can rebalance resources using a bash-like shell. Each gridapp has (1) core nodes, which need to keep running, and (2) scale nodes which can be started and stopped. Nodes can be configured so they are taskset-bound to an individual processor, or sharing a processor between several nodes. There are equivalent primitives for memory and persistent storage. Commercial, 1000 cores. The author contrasts this with K8s by saying that K8s is declarative. This is not the important distinction. My system had declarative configuration, and did a fitting at startup. The key difference is motive. The design above is the natural choice when you build a grid computer from first-principles. K8s is designed to solve a different problem: scaling up the traditional model of programming. Here, a developer writes an algorithm on their workstation. The deployment circumstances and interactions with other systems are an afterthought. Often this developer does not deploy the code themselves, this is done by another team who don't understand the internals of the code.

pjmlp 5y ago
We don't need Linux, a type 1 hypervisor for Cloud infrastructure is good enough.
- cturner 5y ago
  Question about drivers. If you had DMA-oriented/kernel-bypass network cards or wanted to resource-manage GPUs, would a type 1 hypervisor be able to support this?
  - pjmlp 5y ago
    I don't see why not, type 1 hypervisors are the modern version of what started on System/360, naturally depending on specific one the APIs for such stuff will be different.

lbriner 5y ago

I think it is related to but not the same as the OP but to me, something that you don't think of much is that this was invented by Google for Google-scale issues. What does this mean? Lots of things that would be relatively easy for Google make it harder for much smaller companies. Logging? Google can build a log aggregator. Hardware? Google have 100s of 1000s of servers. Documentation? Originally, certain knowledge could be kept in small teams who worked on it, the documentation then had to be generated after-the-fact, which is perhaps why the docs are quite factual and not so good at introducing some of the concepts. You want GUI tools for setting up e.g. rbac and certs? Great but Google don't because they have people who can write the relevant automation scripts in their sleep so don't need what us mere mortals might prefer when new to it. I do like it though and use it for some of our systems.

motoboi 5y ago
Although people love to think of Kubernetes as “the thing google runs on that was open sourced”, it is actually a simplified version, written from scratch, based on the same principles of the tool google actually runs on. On used to run on, because it seems that there is no public information about the new platform after Borg. As a comparison, Mesos was built with the same goal (open source version inspired on Borg) and made some design choices Kubernetes haven’t. UPDATE: - I don’t really know why Google spent the effort on building Kubernetes, but I believe it was for Google Cloud. And it needed to be open sourced to form a community.
- ayewo 5y ago
  Your last sentence with respect to Apache Mesos led me to this previous HN discussion: Apache Mesos to be moved to Attic [1]. 1: https://news.ycombinator.com/item?id=26713082 https://news.ycombinator.com/item?id=26713082
- menage 5y ago
  > On used to run on, because it seems that there is no public information about the new platform after Borg. I left Google quite a few years ago, but my impression is that Borg is still going strong for many of Google's services. There was an attempt to replace it with a new system named Omega, but that ended up being more of an experiment - some of the findings of which were rolled back into future Borg development.

Splizard 5y ago

Let me SSH into a cluster and navigate the configuration database like a virtual filesystem. Make it feel more like a traditional operating system. That will make Kubernetes more approachable.

withinboredom 5y ago
I was thinking the exact same thing. Something like /proc or /dev.
pjmlp 5y ago
Without looking into it, I bet there are companies on CNCF selling exactly that.
wbl 5y ago
Plan 9 was made for this.
tedd4u 5y ago
Try k9s https://github.com/derailed/k9s https://github.com/derailed/k9s
motoboi 5y ago
What you just said is fun, because that is a design choice in Kubernetes. You DO have a virtual hierarchical configuration. It’s just that it’s not a txt file, it’s a json object. The whole Kubernetes things is actually that: a single json defining a supercomputer. Changes to the json become changes in the supercomputer. The whole complexity aims just that: make that task easy. You have a lot of tools to interact with that json, but you actually only need a http client, because it is exposed as an API.
throwaway984393 5y ago
What we need is a real distributed OS.

smitty1e 5y ago

I explain qualitativley to people that the 'cloud' provider is doing a lot on the control plane below the hypervisor. K8s is a 'puff' within the 'cloud', and a bunch of additional control plane activity going on below the 'cloud cover' of the hypervisor line is now yours. Happy days have arrived with the added IP managent, scaling, DNS, &c requirements.

Traubenfuchs 5y ago

There is no reason for kubernetes not being just as simple as heroku or its many alternatives, none. The only reason I‘d believe is that it‘s a conspiracy for long lasting job security for anyone in devops. They managed to shit out an application server even more complex than enterprise websphere, which is both fascinating and abhorrent. Same for terraform and all the other overengineered cloud crap companies pay their devops wizards for.

Sebb767 5y ago
A lot of easy things suddenly become very complex if you need to deal with high availability and/or a lot of distributed computers.
- onlyrealcuzzo 5y ago
  Like counting distributed streaming data with high accuracy and low latency - or especially calculating quantiles - for example.
- goodpoint 5y ago
  Not at all. I worked on HA and distributed systems for 10+ years and the best distibuted designs are by far the simplest.
k8sToGo 5y ago
Why are you so bitter about devops people?
- Traubenfuchs 5y ago
  I am not bitter about them, personally. If I could pick one area to work in, in the field of the technical side of software engineering I would go for devops. I applaud everyone who earns their money with this dreck! But taking a step back, it's madness. I have seen mindbogglingly complex setups 10 years ago, when I had to go begging to the old websphere greybeards when I needed anything sysadmin/devops related, yet in that time it has gotten worse. 4 of the 4 kubernetes projects I have seen personally involved highly customized code for deployment and infrastructure, that wasn't comprehensible for anyone besides the relevant devops teams. From the perspective of efficiency and the company owner(s) it appeared borderline malicious.

viveksingh 5y ago

As a summary are you trying to say, its hard to debug problems on Kubernetes if you have deployed a controller/operator that is not well written. If yes, does that mean Kubernetes is complex (i agree k8s is complex).

ulzeraj 5y ago

Am I wrong when I say this type of opinion normally comes from infrastructure personal? I say that because I've never seen a standard so easy and seamless for a developer to get his code running.

nightpool 5y ago
Very seriously: Do yourself a favor and try out a real PaaS like Heroku sometime. Running a real, production-ready system on Heroku is truly as easy as "git push", and the concepts are way, way simpler to understand. It's not going to be the right fit for everybody, but as far as just pure devex goes, it's truly excellent.
- zerkten 5y ago
  People have mentioned here that a Heroku-like layer would be incompatible with k8s. Based on the description of k8s as a distributed OS, I don't understand how to reconcile these things. I'd want to deploy a k8s infrastructure, but have an easy experience on top for developers that might not maximize k8s use initially, but allow devs to move down a layer eventually. The whole time the ops folks would be running k8s and limited layer above that'd keep devs in a particular lane. Isn't this feasible?
  - q845712 5y ago
    I think several companies are trying to sell exactly that. if you look up VMWare's "TAP" offering it's attempting to provide a wide umbrella of components that move in that direction. Others in the space include Redhat, Replicated, Rancher. A few individual components that try to bring back the ease of rapid/easy deploys include: Flux, Tilt, Argo, kapp-controller. They all have slightly different priorities and make different compromises, but anyways just to assure you there's quite a few people trying to build and sell the layer you're describing.
Sebb767 5y ago
> I say that because I've never seen a standard so easy and seamless for a developer to get his code running. You deploy using extremely long YAML files, your app interacts with a lot of complex software (secret storage, ingress controller etc) and accessing your app for debugging is very non-straightforward. Once you understand the concepts and stack it's great, but when you're coming from locally running your app or even docker, it's as opaque as it gets. In my experience, it's actually worse for developers, because infra people are forced to learn k8s as direct part of their job, while for developers it's more of a sidenote.

ikrenji 5y ago

i don't think k8s is particularly difficult to use/understand. at least from app developer perspective. maybe setting it up and operating it is a different story, but deploying apps to it is fairly simple.

nuker 5y ago

We moved all our apps to AWS ECS. Developers happy. Kube was installed by hired consultants, after they left it was real struggle. Then we moved to ECS.

StreamBright 5y ago
I did the same for every project where k8s was used and started with ECS where k8s was proposed. Even ECS is too complex for my taste, but who am I to dictate what clients want.
k8sToGo 5y ago
ECS or EKS?
- nuker 5y ago
  ECS. EC2 and Fargate types.
ericbarrett 5y ago
I am a huge fan of ECS. It just works. You have to write a tiny bit of tooling, but it's the kind of glue code you'd probably write on top of kube anyway. Failures are straightforward and visible. There's no debugging layer after layer of network redirects. It's much harder for an errant config to "helpfully" punch a hole through your firewall. You get the same uptime and scalability benefits as k8s or EKS. Everything is still a container.
d0gsg0w00f 5y ago
I agree. I don't understand why you would use k8s on AWS. It's an abstraction on top of an abstraction doing nothing but adding complexity. If your k8s pods have to interact with S3 buckets, IAM profiles, and load balancers anyway then you can't ignore the platform. It's just somebody chasing the cloud agnosticism fallacy or doing resume driven development.
- kgeist 5y ago
  We have to support cloud providers in other countries due to local data protection laws (not every country has an AWS data center); doesn't it make sense to make your infrastructure management more-less uniform no matter where it's hosted? In some countries, providers may have only a single availability zone (in reality), or they can get very unstable from time to time (everything is down and god knows for how long), so it's natural for us in such countries to migrate our whole production between completely different providers several times a year. As an ordinary engineer whose job sometimes involves investigating problems in production or configuring something here and there (I'm not a full-time DevOps or SRE myself), I don't have to learn the details of whatever is the current cloud provider this week in the affected country N, I access the production system through a familiar interface which looks more-less same everywhere (k8s), including our test environments, too. Am I missing something? I am pretty curious what "cloud agnosticism fallacy" means. How would you do it differently?

nova22033 5y ago

When people ask for something to be lightweight, what they really mean is "keep the things I need and remove the things I don't..." A lot of these "why you don't need kubernetes" articles should really be "do you really need containers for your 10 user app?"

chrisandchris 5y ago
I think one aspect your reasoning does not properly respect is that running things in container is not all about scaling, but also about easy of deployment. A 10-user-app probably doesn't need k8s, but the advantage of packing it into a container and just starting that container (on a single machine) is very nice. It's so easy to have multiple apps with different languages (or even language versions) on a single host without worrying how to install 3 different PHP versions on that machine.
- stepbeek 5y ago
  Perhaps as a result of incorrectly classifying K8S or K8SaaS as a PaaS when it’s very far from it. Heroku offers the same - or better - devex without the operational burden.
- kasey_junk 5y ago
  It’s worth noting that advantage (and in my belief the large adoption of containers generally) is only valuable to interpreted languages that dont allow static linking. For statically compiled binaries I’ve not worried about different versions of libraries for my binaries for 20 years (and the technology is decades older than that). I do appreciate the docker file format but largely because it’s become used for true virtualization systems like firecracker. Plain Linux “containers” largely offer me very little.
  - pjmlp 5y ago
    Worse, except for stuff like what was being done at Xerox, static linking was the only way for compiled languages. Also on desktop systems, embedding resources into the binaries exists since the early 1980's, or just deliver every in the same directory like NeXTSTEP bundles, or what is called xcopy deployment on the PC world on homage how we used to do it on MS-DOS days.

robertlagrant 5y ago

I think a reasonable design approach is to consider Kubernetes an application management system, just as Postgres is a database management system. Put your stateless pods in there, wire them together with DNS, and inject config into them. Having that philosophy, and optimising that case so it's easy for coders to deploy into it is a big win IMO.

dustinmoris 5y ago

Actually Kubernetes is not complex. The requirements of most software developers are complex. Running small self contained and isolated applications in a highly scalable manner which can scale automatically, where dead services get killed and replaced, where health checks, readiness and liveliness checks fully automate a lot of dev ops problems, where things can be load balanced publicly as well as privately in a private VPN and where complex ingress rules can be created to route traffic from one load balancer to different smaller load balancers which distribute traffic across different apps and with automatic SSL certificate management for public services via an ACME provider are VERY COMPLEX requirements. If you want to run your stuff in such an advanced fashion then Kubernetes makes it EXTREMELY SIMPLE. If you don't care about these things then don't complain about Kubernetes. Just run your shit on a single VM and deal with everything manually as and when you need it.

torginus 5y ago

Just a funny thought exercise I did recently - I looked at our sizeable kubernetes cluster of 30ish VMs, and added up the core counts. It added up to 98. You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server.

jordanthoms 5y ago
I guess it's all relative, our largest cluster scales up to around 3000 VCPU and there's plenty of much larger ones out there. You might be able to get better utilisation of your VMs by running a smaller number with more cores on each - I'd try 16vcpu nodes.
- randomswede 5y ago
  At my previous workplace, one of the clusters I herded was 60-odd (physical) machines, 72 or 96 cores (honestly can't recall at this point) and 384 GB RAM (IIRC) per physical server. But, that place was a wee bit strange, in that we ran all our kubernetes in-house and started on bare metal. And, yes, as a general recommendation, I would tend towards bigger, rather than smaller, nodes. There's a trade-off to be had, but you want each node to be at LEASt as large as the largest pod you intend to deploy, and you probably want at least some spare, on top of that.
darkwater 5y ago
So if that host goes down you are screwed? Would not be a wise decision from your cloud provider. Although thinking about it twice, usually cloud providers response in such a case would be a shrug and that's all...
- KptMarchewa 5y ago
  I very doubt that would happen. Even if, at the very least, control plane would be distributed, so host going down would just cause another underlying VM to spawn and everything being recreated.

codeflo 5y ago

None if that explains why it’s so, how do I put it, fiddly? A flexible tool could still have reasonable defaults. If you rent a managed K8s cluster from AWS, you’re of course fine, but why does deploying it yourself have to be so complicated that the recommendation is now to use a “Kubernetes distribution” like K3s. Why can’t it just be a daemon you install using standard mechanisms?

lbriner 5y ago
Probably because it was designed for scale and big companies that can afford lots of hardware don't need to invest on standalone instances.
KptMarchewa 5y ago
There are 3 node clusters on RPi and 15000 node bare metal clusters. For which will you optimize?
- codeflo 5y ago
  First of all, are you sure there isn't a way to improve the ergonomics of one without harming the other? Second of all, to the extent that there is a tradeoff, this is essentially a solved problem in every other kind of daemon (from web servers to databases). Make the simple case secure and simple, while making the complicated case (which will require a lot of custom configuration and reading the documentation anyway, not matter what you do) as straightforward as possible.
pas 5y ago
Because it was quickly thrown together in an experimential/evolving language, in a design by committee way, by a very non-diverse group. And it's getting much better at ergonomics/UX/DX (go is also getting much better), but it's very similar to OpenStack. (Python 2, resource hog, impossibly fragile, basic setup works, but uselessly incomplete without certain critical 3rd party components - eg. without Ceph and ovs it was just too big of a mess, similarly the CNI stuff and the LoadBalancers for k8s are the important missing batteries.) That said both microk8s (snap packaged by Canonical) and kind (kubernetes in docker) are turnkey solutions. k3s is more of a vanguard :) .. also, it can be (is already) packaged into a simple RPM/DEB easily, like the GitLab Omnibus.

tardigr8 5y ago

> "you will not, in general, see that error at creation time. " I agree, this was daunting for me too. I use werf.io now (an opensource project which I am not affiliated with, I am just a user). It deploys my resources and waits to validate whether the deploy succeeded. They call it "giterminism". You can do this yourself with scripts for sure, but I have not needed to extend any of their built in functionality yet. Highly recommended for anyone using k8s with CD. https://github.com/werf/werf https://github.com/werf/werf

throwaway984393 5y ago

Kubernetes is so complex for 2 reasons: 1. A distributed system with as many design considerations as K8s will always be highly complex. 2. K8s was poorly designed by committee by people who didn't really understand how to build what they were building. It still lacks things like true multitenancy (I'm aware of the SIG) and its security model is a farce. Everything from its configuration format to its RPC to its roles and beyond are unnecessarily complex and painful. It should have been redesigned twice over by now but that will never happen at this point. The time is right for someone to write a Kubernetes replacement and disrupt the lumbering joke that is the K8s ecosystem.

goodpoint 5y ago
Part 1 is not true. You can implement consensus, leader election, DHTs using few simple and small libraries.
- throwaway984393 5y ago
  Those are only 3 of like 50 different design requirements of k8s. It really is a ton of functionality when you implement it all, and how it all has to combine together is quite complex. Just trying to keep a mental model of the actual execution paths to run a distributed microservice using all of k8s is more than most people could reason about without flow charts. But that's no excuse for poor design, which is what we got

streamofdigits 5y ago

why is explaining why kubernetes is so complex, so complex? could somebody ELI-RP? (start with a collection of raspeberry pi's sitting on my desk and take it from there)

goodpoint 5y ago
Excellent question. Because it has plenty of unnecessary complexity.
- streamofdigits 5y ago
  it may be. can't really tell until somebody maps it to a universe that (I think) I understand... but I suspect this seeming to be a hype of sorts within certrain tech circles (with associated professional incentives etc) there aren't many around that would have both the in-depth knowledge and the desire to really de-obfuscate and break it down
  - p_l 5y ago
    Because it's taking care of a complex problem with lots of necessary complexity (though how complex it seems depends on whether you learn it bottom up or top down, IMO)
  - goodpoint 5y ago
    > hype of sorts within certrain tech circles The downvotes on my comments and the complains are telling.
    - biorach 5y ago
      Telling you that you don't know what you're talking about.
      - goodpoint 5y ago
        That's what you want to believe.

deleted 5y ago

[deleted]

ranrotx 5y ago

I personally feel like Kubernetes tries to be all the things, which creates feature and complexity bloat to be able to accommodate any possible workload you can throw at it. “Cloud Native” is a bit of a misnomer. It’s more like trying to run your own cloud versus relying on a service provider. Sure you can run a stateful, highly available database on k8s. But you’re going to have to manage the complexity of making it work on k8s. Or you could just use a managed database service of your cloud provider (which has existed even before k8s).

zimbatm 5y ago

"Kubernetes is a cluster operating system" makes sense. Some of the complexity of Kubernetes also feels like the early days of Linux where you had to basically build your own Linux distribution. Even with managed solutions like EKS, you still need to install and learn about cert-manager, how to pass the IAM credentials around, monitoring kinda sucks so you want to replace it, ...

zozbot234 5y ago
Note that there are other cluster operating systems which make different tradeoffs than Kubernetes. Some of these are true Single System Image (SSI), but even something as comparatively simple as Plan9 is closer to the SSI end of the scale than the less hands-on, more 'declarative'-focused k8s.

ay 5y ago

I would love to see an article from the same author about nomad, and comparison between the two…

mdtrooper 5y ago

Has Kubernetes some code of java or angular? (end of joke) I remember somebody tell me that the problem of YaCy (a free software alternative to google search) was made on java.

stillbourne 5y ago

I have an axiom I mention on almost every occasion I can when people ask for more granularity on a specific request. "A system that is infinitely configurable is infinitely complex."

fartcannon 5y ago

Complexity slows down future potential competition.

lowbloodsugar 5y ago

Kubernetes is one of those products designed to appear simple and easy to get started with, to drive adoption. Only later does one find how unavoidably complex running such a system is (usually when something awful happens and nobody knows how to fix it). In that respect I miss Mesos: it was in-your-face complex and as a result had far fewer (zero for me) surprises.

eweise 5y ago

I have barely any kubernetes experience but what struck me is that it seemed like a lot of interfaces that need to be implemented for different environments. So you need to know a lot about those specific implementations.

dekhn 5y ago

I wasn't a big fan of k8s when I first started using it (note: I was on the borg team at google when omega was being built, and know the original k8s developers from that time). When I started using it it was still pretty unstable (this was at least 5 years ago). Since then I've been using it more heavily and all I can say is this will likely be the system I spend most of my cluster work in for the next 10-20 years (after that I'll be retired!). It's enough like borg that I can translate over nearly everything, it's not broken in any fundamental way (although there are some serious bugs from 2015 that I never expect to get fixed). One thing about borg that was always controversial was the scripting language created to submit and update jobs (bcl/gcl). It's quirky but I have not seen a better system for inheriting and updating templates. Even so I'm glad the k8s authors force you to use a simple config language with a minimum (or no) logic in it.

breakingcups 5y ago
Could you elaborate on the serious bugs you think are unlikely to get fixed? I think they might help me understand the current limits of k8s better.

jensensbutton 5y ago

Kubernetes is complex because it was built for "The Datacenter as a Computer" http://www.cs.yale.edu/homes/yu-minlan/teach/csci599-fall12/papers/dccomputer.pdf http://www.cs.yale.edu/homes/yu-minlan/teach/csci599-fall12/... (PDF)

ted_dunning 5y ago

The posters here who are commenting on what should be (often largely what Kubernetes already is) would do well to read the paper about the Omega scheduler. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41684.pdf https://static.googleusercontent.com/media/research.google.c... It is almost the rule that none of us has the breadth of experience to understand all of the problems that need to be solved. In fact, an explicit goal of Kubernetes is that almost all users should not need to understand much of how the system actually works. Frankly, most users have neither the time, inclination or intellectual capacity to understand more than a tiny part of the systems they work on. That includes most people who say they do understand those systems completely ... a few short questions can generally expose how little they know about important aspects. That ignorance is a measure of success, btw, not failure.

simonw 5y ago
The problem here is the law of leaky abstractions: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a... Systems that abstract away the complexity for you work fine up to a point... but inevitably you'll end up debugging something caused by a lower layer than the one you are familiar with, at which point you need to dig in and figure out what that abstraction is doing for you. It's not possible to understand everything, but the more you understand the more productive and effective you'll be. I see my overall career partly as an ongoing quest to deepen my understanding of the various abstractions I rely on.
bogomipz 5y ago
Wasn't Omega the successor to Borg? I'd be curious what's the status of it today at Google?
- dekhn 5y ago
  I believe there are several public articles what happened. Some omega features were folded back into borg, omega development pivoted to k8s development. Most of google still runs on borg/internal but more and more people are moving projects to GCP-hosted cloud. That only really applies for things that can run Outside, and for which k8s is "good enough". Borg will continue to exist, roughly in its currently for, for at least 20 more years if Google doesn't get shut down by DOJ. It would simply not be cost effective to migrate large parts of GOogle's infra off Google borg/internal. In my experience, absolutely nothing in the world matches the experience inside of google; enormous number of enormous clusters and gobs of networking within and between them, plus a heavy set of existing workloads constantly testing it, all while people constantly deploy new versions of borg, filesystems, database and applications from a single codebase. Just as importantly, there was a massive culture of cluster nerds (SRE) that ran the clusters very efficiently, but also with care and attention to a wide range of users, including ones that weren't on Big Teams. I never, ever would have been able to build and run exacycle, an idle cycle harvester that provided at least 4 significant contributions to science.

f23ojoi 5y ago

tomerbd 5y ago

Declarative = goodFunctionName(Imperative)

akagusu 5y ago

Kubernetes is so complex because vendors can make huge piles of money selling solutions to handle the complexity.