73 ms·

Kubernetes is our generation's Multics

• christophilus 5y ago
  Kubernetes gets a lot of shade, and rightfully so. It’s a tough problem. I do hope we get a Ken Thompson or Rich Hickey-esque solution at some point.
  • cogman10 5y ago
    I see the shade thrown at k8s... but honestly I don't know how much of it is truly deserved. k8s is complex not unnecessarily, but because k8s is solving a large host of problems. It isn't JUST solving the problem of "what should be running where". It's solving problems like "how many instances should be where? How do I know what is good and what isn't? How do I route from instance A to instance b? How do I flag when a problem happens? How do I fix problems when they happen? How do I provide access to a shared resource or filesystem?" It's doing a whole host of things that are often ignored by shade throwers. I'm open to any solution that's actually simpler, but I'll bet you that by the time you've reached feature parity, you end up with the same complex mess. The main critique I'd throw at k8s isn't that it's complex, it's that there are too many options to do the same thing.
    • novok 5y ago
      K8 is the semi truck of software, great for semi scale things, but often used when a van would just do fine.
      • cogman10 5y ago
        To me, usefulness is less to do with scale and more to do with number of distinct services. If you have just a single monolith app (such as a wordpress app) then sure, k8s is overkill. Even if you have 1000 instances of that app. It's once you start having something like 20+ distinct services that k8s starts paying for itself.
        • lanstin 5y ago
          Especially with 10 distinct development teams that all have someone smart enough to crank out some YAML with their specific requirements.
      • loudlambda 5y ago
        Kubernetes is an aircraft carrier, where most people just need a skiff.
    • giantrobot 5y ago
      I think part of the shade throwing is k8s has a high lower bound of scale/complexity "entry fee" where is actually makes sense. If your scale/complexity envelope is below that lower bound, you're fighting k8s, wasting time, or wasting resources. Unfortunately unless you've got a lot of k8s experience that scale/complexity lower bound isn't super obvious. It's also possible to have your scale/complexity accelerate from "k8s isn't worthwhile" to "oh shit get me some k8s" pretty quickly without obvious signs. That just compounds the TMTOWTDI choice paralysis problems. So you get people that choose k8s when it doesn't make sense and have a bad time and then throw shade. They didn't know ahead of time it wouldn't make sense and only learned through the experience. There's a lot of projects like k8s that don't advertise their sharp edges or entry fee very well.
      • throwaway894345 5y ago
        > I think part of the shade throwing is k8s has a high lower bound of scale/complexity "entry fee" where is actually makes sense. If your scale/complexity envelope is below that lower bound, you're fighting k8s, wasting time, or wasting resources. Maybe compared to Heroku or similar, but compared to a world where you're managing more than a couple of VMs I think Kubernetes becomes compelling quickly. Specifically, when people think about VMs they seem to forget all of the stuff that goes into getting VMs working which largely comes with cloud-provider managed Kubernetes (especially if you install a couple of handy operators like cert-manager and external-dns): instance profiles, AMIs, auto-scaling groups, key management, cert management, DNS records, init scripts, infra as code, ssh configuration, log exfiltration, monitoring, process management, etc. And then there's training new employees to understand your bespoke system versus hiring employees who know Kubernetes or training them with the ample training material. Similarly, when you have a problem with your bespoke system, how much work will it be to Google it versus a standard Kubernetes error? Also, Kubernetes is really new and it is getting better at a rapid pace, so when you're making the "Kubernetes vs X" calculation, consider the trend: where will each technology be in a few years. Consider how little work you would have to do to get the benefits from Kubernetes vs building those improvements yourself on your bespoke system.
        • lanstin 5y ago
    • dolni 5y ago
      > how many instances should be where? Are you referring to instances of your application, or EC2 instances? If instances of your application, in my experience it doesn't really do much for you unless you are willing to waste compute resources. It takes a lot of dailing in to effectively colocate multiple pods and maximize your resource utilization. If you're referring to EC2 instances, well AWS autoscaling does that for you. Amazon and other cloud providers have the advantage of years of tuning their virtual machine deployment strategies to provide maximum insulation from disruptive neighbors. If you are running your own Kubernetes installation, you have to figure it out yourself. > How do I know what is good and what isn't? Autoscaling w/ a load balancer does this trivially with a health check, and it's also self-healing. > How do I route from instance A to instance b? You don't have to know or care about this if you're in a simple VPC. If you are in multiple VPCs or a more complex single VPC setup, you have to figure it out anyway because Kubernetes isn't magic. > How do I flag when a problem happens? Probably a dedicated service that does some monitoring, which as far as I know is still standard practice for the industry. Kubernetes doesn't make that go away. > How do I fix problems when they happen? This is such a generic question that I'm not sure how you felt it could be included. Kubernetes isn't magic, your stuff doesn't always just magically work because Kubernetes is running underneath it. > How do I provide access to a shared resource or filesystem? Amazon EFS is one way. It works fine. Ideally you are not using EFS and prefer something like S3, if that meets your needs. > It's doing a whole host of things that are often ignored by shade throwers. I don't think they're ignored, I think that you assume they are because they are because those things aren't talked about. They aren't talked about because they aren't an issue with Kubernetes. The problem with Kubernetes is that it is a massively complex system that needs to be understood by its administrators. The problem it solves overlaps nearly entirely with existing solutions that it depends on. And it introduces its own set of issues via complexity and the breakneck pace of development. You don't get to just ignore the underlying cloud provider technology that Kubernetes is interfacing with just because it abstracts those away. You have to be able to diagnose and respond to cloud provider issues _in addition_ to those that might be Kubernetes-centric. So yes, Kubernetes does solve some problems. Do the problems it solves outweigh the problems it introduces? I am not sure about that. My experience to Kubernetes is limited to troubleshooting issues with Kubernetes ~1.6, which we got rid of because we regularly ran into annoying problems. Things like: * We scaled up and then back down, and now there are multiple nodes running 1 pod and wasting most of their compute resources. * Kubernetes would try to add routes to a route table that was full, and attempts to route traffic to new pods would fail. * The local disk of a node would fill up because of one bad actor and impact multiple services. At my workplace, we build AMIs that bake-in their Docker image and run the Docker container when the instance launches. There are some additional things we had to take on because of that, but the total complexity is far less than what Kubernetes brings. Additionally, we have the side benefit of being insulated from Docker Hub outages.
    • deleted 5y ago
      [deleted]
  • encryptluks2 5y ago
    I think a large part of the problem is that systems like Kubernetes are designed to be extensible with a plugin architecture in mind. Simple applications usually have one way of doing things but they are really good at it. This begs to question if there is a wrong or right way of doing things and if a single system can adapt fast enough to the rapidly changing underlying strategies, protocols, and languages to always be at the forefront of what is considered best practices in all levels of development and deployment. These unified approaches usually manifest themselves as each cloud providers best practice playbooks, but each public cloud is different. Unless something like Kuberenetes can build a unified approach across all cloud providers or self hosting solutions then it will always be overly complex because it will always be changing for each provider to maximize their interests in adding their unique services.
  • throwaway894345 5y ago
    Having used Kubernetes for a while, I'm of the opinion that it's not so much complex as it is foreign, and when we learn Kubernetes we're confronted with a bunch of new concepts all at once even though each of the concepts are pretty simple. For example, people are used to Ansible or Terraform managing their changes, and the "controllers continuously reconciling" takes a bit to wrap one's head around. And then there are all of the different kinds of resources and the general UX problem of managing errors ("I created an ingress but I can't talk to my service" is a kind of error that requires experience to understand how to debug because the UX is so bad, similarly all of the different pod state errors). It's not fundamentally complex, however. The bits that are legitimately complex seem to involve setting up a Kubernetes distribution (configuring an ingress controller, load balancer provider, persistent volume providers, etc) which are mostly taken care of for you by your cloud provider. I also think this complexity will be resolved with open source distributions (think "Linux distributions", but for Kubernetes)--we already have some of these but they're half-baked at this point (e.g., k3s has local storage providers but that's not a serious persistence solution). I can imagine a world where a distribution comes with out-of-the-box support for not only the low level stuff (load balancers, ingress controllers, persistence, etc) but also higher level stuff like auto-rotating certs and DNS. I think this will come in a few years but it will take a while for it to be fleshed out. Beyond that, a lot of the apparent "complexity" is just ecosystem churn--we have this new way of doing things and it empowers a lot of new patterns and practices and technologies and the industry needs time and experience to sort out what works and what doesn't work. To the extent I think this could be simplified, I think it will mostly be shoring up conventions, building "distributions" that come with the right things and encourage the right practices. I think in time when we have to worry less about packaging legacy monolith applications, we might be able to move away from containers and toward something more like unikernels (you don't need to ship a whole userland with every application now that we're starting to write applications that don't assume they're deployed onto a particular Linux distribution). But for now Kubernetes is the bridge between old school monoliths (and importantly, the culture, practices, and org model for building and operating these monoliths) and the new devops / microservices / etc world.
• nickjj 5y ago
  I'd be curious what a better alternative looks like. I'm a huge fan of keeping things simple (vertically scaling 1 server with Docker Compose and scaling horizontally only when it's necessary) but having learned and used Kubernetes recently for a project I think it's pretty good. I haven't come across too many other tools that were so well thought out while also guiding you into how to break down the components of "deploying". The idea of a pod, deployment, service, ingress, job, etc. are super well thought out and are flexible enough to let you deploy many types of things but the abstractions are good enough that you can also abstract away a ton of complexity once you've learned the fundamentals. For example you can write about 15 lines of straight forward YAML configuration to deploy any type of stateless web app once you set up a decently tricked out Helm chart.. That's complete with running DB migrations in a sane way, updating public DNS records, SSL certs, CI / CD, having live-preview pull requests that get deployed to a sub-domain, zero downtime deployments and more.
  • ljm 5y ago
    > once you set up a decently tricked out Helm chart I don't disagree but this condition is doing a hell of a lot of work. To be fair, you don't need to do much to run a service on a toy k8s project. It just gets complicated when you layer on all production grade stuff like load balancers, service meshes, access control, CI pipelines, o11y, etc. etc.
    • edoceo 5y ago
      o11y = observability
      • tovej 5y ago
        You couldn't create a parody of this naming convention that's more outlandish than the way it's actually being used.
        • handrous 5y ago
          o11y? In my head it sounds like it's a move in "Tony Hawk: Pro K8er"
• flowerlad 5y ago
  The alternatives to Kubernetes are even more complex. Kubernetes takes a few weeks to learn. To learn alternatives, it takes years, and applications built on alternatives will be tied to one cloud. See prior discussion here: https://news.ycombinator.com/item?id=23463467 https://news.ycombinator.com/item?id=23463467 You'd have to learn AWS autoscaling group (proprietary to AWS), Elastic Load Balancer (proprietary to AWS) or HAProxy, Blue-green deployment, or phased rollout, Consul, Systemd, pingdom, Cloudwatch, etc. etc.
  • dolni 5y ago
    Kubernetes uses all those underlying AWS technologies anyway (or at least an equivalently complex thing). You still have to be prepared to diagnose issues with them to effectively administrate Kubernetes.
    • flowerlad 5y ago
      That hasn't been my experience. I use Kubernetes on Google cloud (because they have the best implementation of K8s), and I have never had to learn any Google-proprietary things.
    • giantrobot 5y ago
      At least with building to k8s you can shift to another cloud provider if those problems end up too difficult to diagnose or fix. Moving providers with a k8s system can be a weeks long project rather than a years long project which can easily make the difference between surviving and closing the doors. It's not a panacea but it at least doesn't make your system dependent on a single provider.
      • dolni 5y ago
        If you can literally pick up and shift to another cloud provider just by moving Kubernetes somewhere else, you are spending mountains of engineering time reinventing a bunch of different wheels. Are you saying you don't use any of your cloud vendor's supporting services, like CloudWatch, EFS, S3, DynamoDB, Lambda, SQS, SNS? If you're running on plain EC2 and have any kind of sane build process, moving your compute stuff is the easy part. It's all of the surrounding crap that is a giant pain (the aforementioned services + whatever security policies you have around those).
• ransom1538 5y ago
  Eh. I think people over complicate k8s in their head. Create a bunch of docker files that let your code run, write a bunch of yaml files on how you want your containers to interact, get an endpoint: the end.
  • klysm 5y ago
    I mean people over complicated software engineering in their head. Write a bunch of files of code, write some other files: the end. /s
    • ransom1538 5y ago
      Why the "/s"? -- sounds right.
  • sgt 5y ago
    ... That's also a little bit like saying it's super simple to develop an app using framework X because a "TODO list" type of app could be written in 50 loc.
• exdsq 5y ago
  Two amazing quotes that really resonate with me: > The industry is full of engineers who are experts in weirdly named "technologies" (which are really just products and libraries) but have no idea how the actual technologies (e.g. TCP/IP, file systems, memory hierarchy etc.) work. I don't know what to think when I meet engineers who know how to setup an ELB on AWS but don't quite understand what a socket is... > Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP.
  • rantwasp 5y ago
    this is bound to happen. the more complicated the stack that you use becomes, the less details you understand about the lower levels. who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process? All of these were table stakes at some point in time. The key is not to understand all layers perfectly. The key is to know when to stop adding layers.
    • abathur 5y ago
      And maybe to learn the smell of a leaking layer?
    • ohgodplsno 5y ago
      While I will not pretend to be an expert at either of those, having at least a minimal understanding of all of these is crucial if you want to pretend to be a software engineer. If you can't write a library, or figure out why your process isn't working, you're not an engineer, you're a plumber, or a code monkey. Not to say that's bad, but considering the sheer amount of mediocre devs at FAANG calling themselves engineers, it just really shines a terrible light on our profession.
      • puffyvolvo 5y ago
        abstractions layers exist for this reason. as much of a sham as the 7-layer networking model is, it's the reason you can spin up an http server without knowing tcp internals, and you can write a webapp without caring (much) about if its being served over https, http/2, or SPDY.
• jaaron 5y ago
  I feel like we've already seen some alternatives and the industry, thus far, is still orienting towards k8s. Hashicorp's stack, using Nomad as an orchestrator, is much simpler and more composable. I've long been a fan of Mesos' architecture, which I also think is more composable than the k8s stack. I just find it surprising an article that is calling for an evolution of the cluster management architecture fails to investigate the existing alternatives and why they haven't caught on.
  • verdverm 5y ago
    We had someone explore K8s vs Nomad and they said K8s because nomad docs are bad. They got much further with K8s in the same timeboxed spike
    • bradstewart 5y ago
      Setting up the right parameters/eval criteria to exercise inside of a few week timebox (I'm assuming this wasn't a many month task) is extremely difficult to do for a complex system like this. At least, to me it is--maybe more ops focused folks can do it quicker. Getting _something_ up and running quickly isn't necessarily a good indicator of how well a set of tools will work for you over time, in production work loads.
      • verdverm 5y ago
        It was more about migrating the existing microservices than some example app, runs in docker compare today. Getting the respective platforms up was not the issue. I don't think weeks were spent, but they were able to migrate a complex application to K8s in less than a week. Couldn't get it running in Nomad, which was tried first due to its supposed simplicity over K8s.
    • mlnj 5y ago
      For me when exploring K8s vs Nomad, Nomad looked like a clear choice. That was until I had to get Nomad + Consul running. I found it all really difficult to get running in a satisfactory manner. I never even touched the whole Vault part of the setup because it was all overwhelming. On the other side K8s was a steep learning curve with lots of options and 'terms' to learn but never was a point into the whole exploration where I was stuck. The docs are great. the community is great and the number of examples available allows us to mix n match lots of different approaches.
• tyingq 5y ago
  It looks to be getting more complex too. I understand the sales pitch for a service mesh like Istio, but now we're layering something fairly complicated on top of K8S. Similar for other aspects like bolt on secrets managers, logging, deployment, etc, run through even more abstractions.
• etaioinshrdlu 5y ago
  As a Kubernetes outsider, I get confused why so much new jargon had to be introduced. As well as so many little new projects coupled to Kubernetes with varying degrees of interoperability. It makes it hard to get a grip on what Kube really is for newcomers. It also has all the hallmarks of a high-churn product where you need to piece together your solution from a variety of lower-quality information sources (tutorials, QA sites) rather than a single source of foolproof documentation.
  • sidlls 5y ago
    > I get confused why so much new jargon had to be introduced. Consider the source of the project for your answer (mainly, but not entirely, bored engineers who are too arrogant to think anybody has solved their problem before). > It also has all the hallmarks of a high-churn product where you need to piece together your solution from a variety of lower-quality information sources (tutorials, QA sites) rather than a single source of foolproof documentation. This describes 99% of open source libraries used.The documentation looks good because auto doc tools produce a prolific amount of boilerplate documentation. In reality the result is documentation that's very shallow, and often just a re-statement of the APIs. The actual usage documentation of these projects is generally terrible, with few exceptions.
    • nhooyr 5y ago
      I find it's low quality libraries that tend to have poor documentation. Perhaps that's 99% of open source libraries.
    • joshuamorton 5y ago
      > Consider the source of the project for your answer (mainly, but not entirely, bored engineers who are too arrogant to think anybody has solved their problem before). This seems both wrong and contrary to the article (which mentions that k8s is a descendant of Borg, and in fact if memory serves many of the k8s authors were borg maintainers). So they clearly were aware that people had solved their problem before, because they maintained the tool that had solved the problem for close to a decade.
    • 5y ago
• commanderjroc 5y ago
  This feels like a post ranting against SystemD written from someone who likes init. I understand that K8 does many things but its also how you look at the problem. K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail. Arguably, this is one problem that is made up of smaller problems that are solved by smaller services just like SystemD works. Sometimes I wonder if the Perlis-Thompson Principle and the Unix Philosophy have become a way to force a legalistic view of software development or are just out-dated.
  • dolni 5y ago
    I don't find the comparison to systemd to be convincing here. The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts. They're reduced to unit files which are often just 10-15 lines. systemd is designed to replace old stuff. The result of Kubernetes for the average administrator is a massively complex system with its own unique concepts. It needs to be well understood if you want to be able to administrate it effectively. Updates come fast and loose, and updates are going to impact an entire cluster. Kubernetes, unlike systemd, is designed to be built _on top of_ existing technologies you'd be using anyway (cloud provider autoscaling, load balancing, storage). So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds.
    • 0xEFF 5y ago
      Kubernetes removes the complexity of keeping a process (service) available. There’s a lot to unpack in that sentence, which is to say there’s a lot of complexity it removes. Agree it does add as well. I’m not convinced k8s is a net increase in complexity after everything is accounted for. Authentication, authorization, availability, monitoring, logging, deployment tooling, auto scaling, abstracting the underlying infrastructure, etc…
      • dolni 5y ago
        > Kubernetes removes the complexity of keeping a process (service) available. Does it really do that if it you just use it to provision an AWS load balancer, which can do health checks and terminate unhealthy instances for you? No. Sure, you could run some other ingress controller but now you have _yet another_ thing to manage.
• dekhn 5y ago
  I have borg experience and my experience with k8s was extremely negative. Most of my time was spent diagnosing self-inflicted probmems by the k8s framework. I've been trying nomad lately and it's a bit more direct.
  • jedberg 5y ago
    I think that's because Borg comes with a team of engineers who keep it running and make it easy. I've had a similar experience with Cassandra. Using Cassandra at Netflix was a joy because it always just worked. But there was also a team of engineers who made sure that was the case. Running it elsewhere was always fraught with peril.
    • dekhn 5y ago
      yes several of the big benefits are: the people who run borg (and the ecosystem) are well run (for the most part). And, the ability to find them in chat and get them to fix things for you (or explain some sharp edge).
  • jrockway 5y ago
    I have borg experience and I think Kubernetes is great. Before borg, I would basically never touch production -- I would let someone else handle all that because it was always a pain. When I left Google, I had to start releasing software (because every other developer is also in that "let someone else handle it" mindset), and Kubernetes removed a lot of the pain. Write a manifest. Change the version. Apply. Your new shit is running. If it crashes, traffic is still directed to the working replicas. Everyone on my team can release their code to any environment with a single click. Nobody has ever ssh'd to production. It just works. I do understand people's complaints, however. Setting up "the rest" of the system involves making a lot of decisions. Observability requires application support, and you have to set up the infrastructure yourself. People generally aren't willing to do that, and so are upset when their favorite application doesn't work their favorite observability stack. (I remember being upset that my traces didn't propagate from Envoy to Grafana, because Envoy uses the Zipkin propagation protocol and Grafana uses Jaeger. However, Grafana is open source and I just added that feature. Took about 15 minutes and they released it a few days later, so... the option is available to people that demand perfection.) Auth is another issue that has been punted on. Maybe your cloud provider has something. Maybe you bought something. Maybe the app you want to run supports OIDC. To me, the dream of the container world is that applications don't have to focus on these things -- there is just persistent authentication intrinsic to the environment, and your app can collect signals and make a decision if absolutely necessary. But that's not the way it worked out -- BeyondCorp style authentication proxies lost to OIDC. So if you write an application, your team will be spending the first month wiring that in, and the second month documenting all the quirks with Okta, Auth0, Google, Github, Gitlab, Bitbucket, and whatever other OIDC upstreams exist. Big disaster. (I wrote https://github.com/jrockway/jsso2 https://github.com/jrockway/jsso2 and so this isn't a problem for me personally. I can run any service I want in my Kubernetes cluster, and authenticate to it with my FaceID on my phone, or a touch of my Yubikey on my desktop. Applications that want my identity can read the signed header with extra information and verify it against a public key. But, self-hosting auth is not a moneymaking business, so OIDC is here to stay, wasting thousands of hours of software engineering time a day.) Ingress is the worst of Kubernetes' APIs. My customers run into Ingress problems every day, because we use gRPC and keeping HTTP/2 streams intact from client to backend is not something it handles well. I have completely written it off -- it is underspecified to the point of causing harm, and I'm shocked when I hear about people using it in production. I just use Envoy and have an xDS layer to integrate with Kubernetes, and it does exactly what it should do, and no more. (I would like some DNS IaC though.) Many things associated with Kubernetes are imperfect, like Gitops. A lot of people have trouble with the stack that pushes software to production, and there should be some sort of standard here. (I use ShipIt, a Go program to edit manifests https://github.com/pachyderm/version-bump https://github.com/pachyderm/version-bump, and ArgoCD, and am very happy. But it was real engineering work to set that up, and releasing new versions of in-house code is a big problem that there should be a simple solution to.) Most of these things are not problems brought about by Kubernetes, of course. If you just have a Linux box, you still have to configure auth and observability. But also, your website goes down when the power supply in the computer dies. So I think Kubernetes is an improvement. The thing that will kill Kubernetes, though, is Helm. I'm out of time to write this comment but I promise a thorough analysis and rant in the future ;)
• wizwit999 5y ago
  I agree a lot with his premise, that Kubernetes is too complex, but not at all with his alternative to go even lower level. And the alternative of doing everything yourself isn't too much better either, you need to learn all sorts of cloud concepts. The better alternative is a higher level abstraction that takes care of all of this for you, so an average engineer building an API does not need to worry about all these low level details, kind of like how serverless completely removed the need to deal with instances (I'm building this).
  • mountainriver 5y ago
    That sounds like knative
    • wizwit999 5y ago
      I haven't heard of that. Took a look and it still seems too low level. I think we need to think much bigger in this space. Btw were not approaching this from a Kubernetes angle at all.
• zozbot234 5y ago
  > Kubernetes is our generation's Multics Prove it. Create something simpler, more elegant and more principled that does the same job. (While you're at it, do the same for systemd which is often criticized for the same reasons.) Even a limited proof of concept would be helpful. Plan9 and Inferno/Limbo were built as successors to *NIX to address process/environment isolation ("containerization") and distributed computing use cases from the ground up, but even these don't even come close to providing a viable solution for everything that Kubernetes must be concerned with.
  • lallysingh 5y ago
    The successor will probably be a more integrated platform where it provides a lot of stuff you've got to use sidecars, etc for. Probably a language with good IPC (designed for real distributed systems that handle failover), some unified auth library, and built-in metrics and logging. A lot of real-life k8s complexity is trying to accommodate many supplemental systems for that stuff. Otherwise it's a job scheduler and haproxy.
  • up_and_up 5y ago
    https://www.nomadproject.io/ https://www.nomadproject.io/
    • gizdan 5y ago
      Nomad also doesn't have a lot of feature that are built into kubernetes, features that otherwise require other hashicorp tools. So now you have a vault cluster, a consul cluster, a nomad cluster, then hcl to manage it all, probably a terraform enterprise cluster. So what have you gained? Besides the same amount of complexities with fewer features.
      • AlexCoventry 5y ago
        I think Nomad sounds like the direction the OP blog post is proposing to move in: a set of largely independent tools which can each address some aspect of the problem kubernetes is trying to solve.
• debarshri 5y ago
  One of the most relevant and amazing blogs I have read in recent times. I have been working for a firm that have been onboarding multiple small scale startup or lifestyle businesses to kubernetes. My opinion is that if you have an ruby on rails or python app, you don't really need kubernetes. It is like bringing bazooka to a knife fight. However, I do think kubernetes has some good practice embedded in them, which I will always cherish. If you are not operating at huge scale, both operations or/and teams, it actually comes at a high cost of productivity and tech debt. I wish there was an easier tech that would bridge going from VMs to bunch of VMs, bunch of containers to kubernetes.
• powera 5y ago
  Eh. Kubernetes is complex, but I think a lot of that is that computing is complex, and Kubernetes doesn't hide it. Your UNIX system runs many daemons you don't have to care about. Whereas something like lockserver configuration is still a thing you have to care about if you're running Kubernetes.
• waynesonfire 5y ago
  my problem with k8s, is that you learn OS concepts, and then k8s / docker shits all over them.
  • chubot 5y ago
    Yes, this is a core part of the design issue and argument I'm making. The new concepts are leaky abstractions -- they wrap the old ones badly. You still have to understand both to understand the system. Networking in k8s seems to really suffer from this. And the new concepts and old concepts don't compose. They create combinatorial problems, i.e. O(M*N) amounts of glue code.
    • nikau 5y ago
      It's a double whammy, you get the complexity of Kubernetes, and then you get to exec into a docker image that has been stripped of any useful debugging tools under the guise of security. Its even better when its a busybox based image for that linksys router/80s unix troubleshooting experience.
• joe_the_user 5y ago
  Whatever Kubernetes flaws, the analogy is clearly wrong. Multics was never a success and never had wide deployment so Unix never had to compete with it. Once an OS is widely deployed, efforts to get rid of it have a different dynamic (see the history of desktop computing, etc). Especially, getting rid of any deployed, working system (os, application, language, chip-instruction-set, etc) in the name of simplicity is inherently difficult. Everyone agrees things should be stripped down to a bare minimum but no one agrees on what that bare minimum is.
  • loudlambda 5y ago
    Agreed; I think a better analogy for Kubernetes is XML. So many wasted meetings about where to split up namespaces and should every last thing be an attribute or a subtag; none of that added business value. JSON took all those decisions off the table. And yes, huge industrials validly complained that JSON didn't cover X or Y or Z, but for most users JSON is a much better solution then XML. Kubernetes reminds me a lot of XML; there are too many decision points adding unnecessary complexity for the average user's needs. Too many foot guns. Too many unintuitive things. People keep on describing it as "declarative", which seems to be about as true as saying that Java is a functional language. Hopefully someday we'll have something actually declarative, and much more intuitive, something more like AWS's CDK.
    • parasubvert 5y ago
      How is Kubernetes not declarative? I don’t disagree about the exposed complexity, that’s a fundamental decision Kubernetes made about openness and extensibility. Everything is on a level playing field, there are no private APIs.
      • loudlambda 5y ago
        As I recall, running "kubectl edit deployment..." doesn't do anything except edit the definition of the config. Instead, to have it take effect you seem to have to manually kill pods, and the new pods will come up with the edited config. If it were declarative, it should detect what needs to be changed, and automatically update accordingly. Same thing with editing a config. It's possible it was the funnel my local DevOps forced on me (and lacking needed permissions at every turn), but my experience was that if you removed deployments, configs, etc on the next deployment, nothing would be cleaned up and you had to manually remove. Again, that's not declarative. In my experience Terraform and CDK are much more declarative; where you never issue commands to delete a pod or a load balancer or similar. Instead you describe what you want, and their engine figures out what it needs to add or remove or change to get to that state.
• honkycat 5y ago
  People love to pooh-pooh "complicated" things like unit tests, type systems, Kubernetes, GraphQL, etc. Things that are solving a specific problem for LARGE SCALE ENTERPRISE users. I will quote myself here: A problem does not cease to exist just because you decided to ignore it. Without Kubernetes, you still need to: - Install software onto your machines - Start services - Configure your virtual machines to listen on specific ports - have a load balancer directing traffic to and watching the health of those ports - a system to re-start processes when they exit - something to take the logs of your systems and ship them to a centralized place so you can analyze them. - A place to store secrets and provide those secrets to your services. - A system to replace outdated services with newer versions ( for either security updates, or feature updates ). - A system to direct traffic to allow your services to communicate with one another. ( Service discovery ) - A way to add additional instances to a running service and tell the load balancer about them - A way to remove instances when they are no longer needed due to decreased load. So sure, you don't need Kubernetes at an enterprise organization! Just write all of that yourself! Great use of your time, instead of concentrating on writing features that will make your organization more money.
  • bajsejohannes 5y ago
    > solving a specific problem The problem to me is that Kubernetes is not solving a specific problem, but a whole slew of problems. And some of them it's solving really poorly. For example, you can't really have downtime-free deploys in kubernetes (set a longish timer from SIGTERM to increase the chance that there's no downtime). Instead I'd rather solve each problem in a good way. It's not that hard. I'm not implementing it from scratch, but with good tools that exists outside of kubernetes and actually solve a specific problem.
    • hnjst 5y ago
      You can actually renew/upgrade your whole cluster with no downtime if you care enough to tackle the annoying bits that cost a few minutes in YOLO mode.
    • ferdowsi 5y ago
      Why can you not have downtime-free deploys? You tell your applications to drain connections and gracefully exit on SIGTERM. https://pkg.go.dev/net/http#Server.Shutdown https://pkg.go.dev/net/http#Server.Shutdown If your server is incapable of gracefully exiting, that's not a K8s problem.
• slackfan 5y ago
  Kubernetes is fantastic if you're running global-scale cloud platforms, ie, you are literally Google. Over my past five years working with it, there has been not a single customer that had a workload appropriate for kubernetes, and it was 100% cargo-cult programming and tool selection.
  • dilyevsky 5y ago
    Your case is def not the norm. We’re not google sized but we are taking a big advantage of k8s running dozens of services on it - from live video transcoding to log pipelines.
• anonygler 5y ago
  I dislike the deification of Ken Thompson. He's great, but let's not pretend that he'd somehow will a superior solution into existence. The economics and scale of this era are vastly different. Borg (and thus, Kubernetes) grew out of an environment where 1 in a million happens every second. Edge cases make everything incredibly complex, and Borg has solved them all.
  • jeffbee 5y ago
    Much as I am a fan of borg, I think it succeeds mostly by ignoring edge cases, not solving them. k8s looks complicated because people have, in my opinion, weird and dumb use cases that are fundamentally hard to support. Borg and its developers don't want to hear about your weird, dumb use case and within Google there is the structure to say "don't do that" which cannot exist outside a hierarchical organization.
    • throwdbaaway 5y ago
      Interesting. Perhaps k8s is succeeding in the real world because it is the only one that tries to support all the weird and dumb use cases? > Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET – All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That’s probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can’t spend writing new features. > Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP. Instead of seeing k8s as the equivalent of "cover fire" or Windows XP, a more apt comparison is probably Microsoft Office, with all kinds of features to support all the weird and dumb use cases.
• fungiblecog 5y ago
  If Antoine de Saint-Exupery was right that: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." then IT as an industry is heading further and further away from perfection at an exponentially accelerating rate. The only example I can think of where a modern community is actively seeking to simplify things is Clojure. Rich Hickey is very clear on the problem of building more and more complicated stuff and is actively trying to create software by composing genuinely simpler parts.
  • harperlee 5y ago
    Jonathan Blow has also been vocal on that regard.
  • TameAntelope 5y ago
    I'd argue that perfection achievement is not a linear process. Sometimes you have to add way too many things before you can remove all of the useless things. Nobody is puppeteering some grand master plan, we're on a journey of discovery. When we're honest with ourselves, we realize nobody knows what will stick and what won't.
    • benjaminjosephw 5y ago
      Absolutely, but dogma and "best-practices" anchor design discussions around today's norms. People get very defensive about tools they've invested in and that kind of dogma stunts imagination for different and better solutions. Discovery is very rarely an accidental process so we can't take for granted that it will be inevitable. I think it's important to recognize that most people are not interested in discovery at all. Practitioners are often not explorers, and that's okay. They may find incremental improvements through their practice, but paradigm shifting innovation comes from those willing to swim against the stream of popular opinion. Discovery has to be an intentional pursuit of those brave enough to imagine a future beyond Multics/Kubernetes/etc despite the torrent of opinionated naysayers telling them they are foolish for even trying.
      • 5y ago
• aneutron 5y ago
  Okay, right off the bat, the author is already giving himself answers: > Essentially, this means that it [k8s] will have fewer concepts and be more compositional. Well, that's already the case ! At its base, k8s is literally a while loop that converges resources to wanted states. You CAN strip it down to your liking. However, as it is usually distributed, it would be useless to distribute it with nothing but the scheduler and the API ... I do get the author's point. At a certain point it becomes bloated. But I find that when used correctly, it is adequately complex for the problems it solves.
• fnord77 5y ago
  I really love how kubernetes decouples compute resources from actual servers. It works pretty well and handles all kinds of sys-ops-y things automatically. It really cuts down on work for big deployments. actually, it has shown me what sorts of dev-ops work is completely unneeded.
• rektide 5y ago
  It's called "Images and Feelings", but I quite dislike using a the Cloud Native Computing Foundation's quite busy map of services/offerings as evidence against Kubernetes. That lots of people have adopted this, and built different tools & systems around it & to help it is not a downside. I really enjoy the Oil Blog, & was really looking forward when I clicked the link to having some good real criticism. But it feels to me like most of the criticism I see: highly emotional, really averse/afraid/reactionary. It wants something easier simpler, which is so common. I cannot emphasize enough, just do it anyways. There's a lot of arguments from both sides about trying to assess what level of complexity you need, about trying to right size what you roll with. This outlook of fear & doubt & skepticism I think does a huge disservice. A can do, jump in, eager attitude, at many levels of scale, is a huge boon, and it will build skills & familiarity you will almost certainly be able to continue to use & enjoy for a long time. Trying to do less is harder, much harder, than doing the right/good/better job: you will endlessly hunt for solutions, for better ways, and there will be fields of possibilities you must select from, must build & assemble yourself. Be thankful. Be thankful you have something integrative, be thankful you have common cloud software you can enjoy that is cross-vendor, be thankful there's so many different concerns that are managed under this tend. The build/deploy pipeline is still a bit rough, and you'll have to pick/build it out. Kubernetes manifests are a bit big in size, true, but it's really not a problem, it really is there for basically good purpose & some refactoring wouldn't really change what it is. There's some things that could be better. But getting started is surprisingly easy, surprisingly not heavy. There's a weird emotional war going on, it's easy to be convinced to be scared, to join in with reactionary behaviors, but I really have seen nothing nearly so well composed, nothing that fits together so many different pieces well, and Kubernetes makes it fantastically easy imo to throw up a couple containers & have them just run, behind a load balancer, talking to a database, which coverages a huge amount of our use cases.
• misiti3780 5y ago
  This post brings up a good question - how does one get better at low-level programming? What are some good resources?
• hnjst 5y ago
  I'm pretty biased since I gave k8s trainings and operate several kubes for my company and clients. I'll take two pretty different contexts to illustrate why for me k8s makes sense. 1- I'm part of the cloud infrastructure team (99% AWS, a bit of Azure) for a pretty large private bank. We are in charge of security and conformity of the whole platform while trying to let teams be as autonomous as possible. The core services we provide are a self-hosted Gitlab along with ~100 CI runners (Atlantis and Gitlab-CI, that many for segregation), SSO infrastructure and a few other little things. Team of 5, I don't really see a better way to run this kind of workload with the required SLA. The whole thing is fully provisioned and configured via Terraform along with it's dependencies and we have a staging env that is identical (and the ability to pop another at will or to recreate this one). Plenty of benefits like almost 0 downtime upgrades (workloads and cluster), on-the-shelf charts for plenty of apps, observability, resources optimization (~100 runners mostly idle on a few nodes), etc. 2- Single VM projects (my small company infrastructure and home server) for which I'm using k3s. Same benefits in terms of observability, robustness (at least while the host stays up...), IaC, resources usage. Stable minimalists hardened host OS with the ability to run whatever makes sense inside k3s. I had to setup similarly small infrastructures for other projects recently with the constraint of relying on more classic tools so that it's easier for the next ops to take over, I end up rebuilding a fraction of k8s/k3s features with much more efforts (did that with docker and directly on the host OS for several projects). Maybe that's because I know my hammer well enough for screws to look like nails but from my perspective once the tool is not an obstacle k8s standardized and made available a pretty impressive and useful set of features, at large scale but arguably also for smaller setups.
  • pram 5y ago
    99% AWS? You can do Gitlab runners and pretty much everything else with ECS+Fargate. You wouldn't even need to maintain any nodes, clusters, etc!
• gonab 5y ago
  No one has used the words "docker swarm" on the comment section Fill in the words Kubernetes is to Multics as ____ is to docker swarm
• talideon 5y ago
  ...except people actually use K8S.
• wwwaldo 5y ago
  Hi Andy: if you see this, I'm the other 4d polygon renderer! I read the kubernetes whitepaper after RC and ended up spending a lot of the last year on it. Maybe if I had asked you about working with Borg I could have saved myself some trouble. Glad to see you're still very active!
  • chubot 5y ago
    Hi :) Yeah I think it's an interesting topic, and I'm not saying anyone should necessarily be doing something different. But if it "feels wrong", then that's not too surprising to me :) I'd be interested in hearing about any k8s experiences.
• deleted 5y ago
  [deleted]
• deleted 5y ago
  [deleted]
• 1vuio0pswjnm7 5y ago
  I like this title so much I am finally going to give this shell a try. One thing I notice right away is readline. Could editline also be an option. (There's two "editlines", the NetBSD one and an older one at https://github.com/troglobit/editline https://github.com/troglobit/editline) Next thing I notice is the use of ANSI codes by default. Could that be a compile-time option or do we have to edit the source to remove it. TBH I think the graphical web browser is the current generation's Multics. Something that is overly complex, corporatised, and capable of being replaced by something simpler. I am not steeped in Kubernetes or its reason for being but it sounds like it is filling a void of shell know-how amongst its audience. Or perhaps it is addressing a common dislike of the shell by some group of developers. I am not a developer and I love the shell. It is one thing that generally does not change much from year to year. I can safely create things with it (same way people have made build systems with it) that last forever. These things just keep running from one decade to the next no matter what the current "trends" are. Usually smaller and faster, too.
  • parasubvert 5y ago
    Kubernetes is designed similar to the shell: the APIs are a uniform interface, designed for stabilization, while resources are composable and extensible through the it. If you use the stable APIs, your code will run for decades. My hypothetical deployment from 2016 will not need touching (beyond image updates for CVEs) to keep running in 2026 or 2036.
• deleted 5y ago
  [deleted]
• deleted 5y ago
  [deleted]
• asciimov 5y ago
  Anyone want to fill me in on what this "Perlis-Thompson Principle" is?
  • JohnL4 5y ago
    I had to Google it and scroll a blog post.
  • JohnL4 5y ago
    http://www.oilshell.org/blog/2021/07/blog-backlog-2.html http://www.oilshell.org/blog/2021/07/blog-backlog-2.html
  • chubot 5y ago
    I still have to explain it properly, but there is a pretty good sketch on a recent blog post, linked from this comment. (You will probably end up chasing a lot of comment threads, but it's mostly there.) https://news.ycombinator.com/item?id=27914632 https://news.ycombinator.com/item?id=27914632 It's an argument about avoiding O(M*N) glue code. O(M*N) amounts of code are expensive to write, and contain O(M*N) numbers of bugs.
• overgard 5y ago
  Is it really that complex compared to an operating system like Unix though? I mean there's nothing simple about Unix. To me the question is, is it solving a problem that people have in a reasonably simple way? And it seems like it definitely does. I think the hate comes from people using it where it's not appropriate, but then, just don't use it in the wrong place, like anything of this nature. And honestly its complexity is way overblown. There's like 10 important concepts and most of what you do is run "kubectl apply -f somefile.yaml". I mean, services are DNS entries, deployments are a collection of pods, pods are a self contained server. None of these things are hard?
  • cjalmeida 5y ago
    What’s complex about *nix? All you need to understand are device files, POSIX permissions and ACLs, cgroups, tcp/udp sockets, nginx/haproxy, thread/process scheduling, (virtual) memory, PAM, dbus, syslog, pipes, unix sockets, 30 filesystem options, nfs, userspace vs. kernel space, sysvinit or 10 flavors of systemd files, iptables/ufw, networkmanager, ssh, selinux, chroot, flatpack, snaps, rpm, deb, ansible/chef/puppet. Oh deploying on the cloud? Cloudformation/AzureRM as well. Pretty easy. No damn complex k8s needed.
    • nikau 5y ago
      The irony in your comment is tools like networkmanager, snaps, systemd are kubernetes like and severely disliked by experienced unix admins due to the needless complexity and usability of them.
• zmmmmm 5y ago
  I feel like k8s sits in the same space as git. One of those tools that is ridiculously complex, obtuse, un-userfriendly but at the same time worth sucking it all up because the win from consolidating your knowledge into something that is an industry standard is far greater than whatever particular things one doesn't like about how it works. It is a fascinating dynamic however that generates these outcomes where a large numbers of people collectively settle on something that the majority of them seem to hate.
• mastrsushi 5y ago
  >“Our generations Multics” What, because they’re both complex? Multics was never successful enough to be used much outside of Honeywell. They’re both considered complex, but there are so many other examples of industry relevant technologies he could’ve used. Products that’ve actually gained communities and a user space enough to gain insight. I get it, you hate using GooberBoobies or whatever.
• Animats 5y ago
  I'm not sure that "fewer concepts" is a win. "Everything is a file" went too far with Linux, where you get status from the kernel by reading what appears to be various text files. But that runs into all the complexities of maintaining the file illusion. What if you read it in small blocks? Does it change while being read? If not, what if you read some of it and then just hold the file handle. Are you tying up kernel memory? Holding important locks? Or what? Orchestration has a political and business problem, too. How does Amazon feel about something that runs most jobs on your own bare metal servers and rents extra resources from AWS only during overload situations? This appears to be the financially optimal strategy for compute-bound work such as game servers. Renting bare iron 24/7 at AWS prices is not cost effective.
  • jiggawatts 5y ago
    > "Everything is a file" went too far with Linux Having had a play with a few variants on this theme, I think kernel based abstractions are the mistake here. It's too low level and too constrained by the low-level details of the API, as you've said yourself. If you look at something like PowerShell, it has a variant of this abstraction that is implemented in user mode. Within the PowerShell process, there are provider plugins (DLLs) that implement various logical filesystems like "environment variables", "certificates", "IIS sites", etc... These don't all implement the full filesystem APIs! Instead they have various subsets. E.g.: for some providers only implement atomic reads and writes, which is what you want for something like kernel parameters, but not generic data files.
• ablekh 5y ago
  I think that all this boils down to a rather simple dilemma for modern cloud-native infrastructural platforms [in terms of developer experience, i.e., external APIs etc., not internal architecture; and this is not even limited to this class of systems - it is general concept for all software systems]: a) universal, highly configurable & complex (K8s) OR b) highly opinionated and [relatively] simple (e.g., Nomad/Waypoint, Heroku, Apollo, CapRover, Dokku, Porter, AWS Elastic Beanstalk, Digital Ocean's App Platform, Fly, Render). Obviously, there exists a middle-ground category as well: relatively simple, but still opinionated and moderately (???) or highly (e.g., OpenShift) configurable platforms. Thus, the optimal choice depends on relevant team's or organization's priorities with respect to those attributes (configurability, complexity, level & scope of opinionation) as well as level of organizational standardization for IT environments, economic factors, vendor lock-in considerations and, perhaps, something else that I forgot to mention).
• filereaper 5y ago
  Sure, you don't have to use k8s. You can roll your own solutions to what it solves. Your own custom built solution will work, but what in 5 years? 10 years? When it all becomes legacy what then? Will you find the talent who'll want to fix your esoteric environment, just like those COBOL devs? Will anyone respond to your job posts to fix your snowflake environment. Will you pay above average wages to fix your snowflake ways of solving problems that k8s standardized? I bet your C-Level is thinking this. What's to say they won't rip out all of your awesomeness and replace it with standard k8s down the line as its dominating the marketshare. When you're laid off in the next recession, is your amazing problem-solving on your snowflake environment going to help you when everyone else is fully well versed with k8s?
  • chrismsimpson 5y ago
    Whoa dude, ease up on the cool aid
    • pm90 5y ago
      Personally, I think this is an extremely mild version of the dire situation that most teams working with legacy systems often find themselves in.
• musicale 5y ago
  No, Multics was easier to understand, easier to manage, and more reliable. However Multics didn't offer automatic/elastic cloud scaling, which seems to be the main selling point of modern, usually very complicated, container orchestration systems, nor was it designed for building distributed systems. However, if modern Linux had a Multics-style ring architecture, it could replace many of the uses for virtualization and containers.
• skissane 5y ago
  One point where the analogy fails, is that Multics was never particularly popular. Although it was historically influential (especially but not purely through its influence on Unix), it was only ever a small player in the market. It was positioned as an operating system for high-end multi-million dollar mainframes, but in that market IBM was king (with thousands of sites), Multics wasn't even near being second place (with a mere 80 sites at its peak). Even for its vendor, GE/Honeywell, it was an also-ran – Honeywell ended up preferring GCOS as the solution for that market, which is part of why it killed Multics off. GCOS was no doubt technically inferior, but it was a simpler system which made more frugal use of system resources. By contrast, k8s is wildly popular. I have no idea how many installations of it exist in the world, but it probably numbers into the millions.
• MichaelMoser123 5y ago
  i thought that kubernetes is our generations jcl (job control language on ibm mainframes) ; There is a remote similarity in how we are writing descriptors for tasks and then submit it for execution and wait till the mainframe has considered our specification. (suddenly feeling old because of this comparison ...)
  • pram 5y ago
    Yup lol, I've had this same thought. It's like neo-Tuxedo which is basically a mainframe TPS for UNIX https://en.wikipedia.org/wiki/Tuxedo_(software) https://en.wikipedia.org/wiki/Tuxedo_(software)
    • MichaelMoser123 5y ago
      it's funny when you think of it, most of all this distributed system magic was already there on the old mainframe, in some form. And it was there for ages...
• rossjudson 5y ago
  This whole article is, well, a little silly. It says that Kubernetes will disappear and be replaced by something simpler, because it's very difficult to create reliable systems that use it. But...there are tons of reliable systems at Google, all using Borg, and that has a lot of features Kubernetes doesn't have. Stripping down Kubernetes doesn't reduce complexity. It just shifts it.
  • chubot 5y ago
    I don't agree. I worked at Google for over 10 years, during the time when SREs started to make as much or more money than SWEs. There's a reason for that. I also disagree that the systems are reliable. From the outside most the stateless services are fast and reliable; the stateful ones less so. From the inside, no: Internal services were unreliable and slow. (This could have changed in the last 5 years, but there was a clear trend in one direction in my time there.) There were many more internal services on Borg than external ones.
• brandonbloom 5y ago
  Related: https://www.youtube.com/watch?v=3Ea3pkTCYx4 https://www.youtube.com/watch?v=3Ea3pkTCYx4 Key insight can be summarized as "code the perimeter"
  • chubot 5y ago
    (author here) Yes exactly! This is what I'm calling the Perlis-Thompson principle, although it still needs to be fully formed and explained. There are obvious objections to it (which I have some answers to). Sketch of the argument here, with links: http://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts http://www.oilshell.org/blog/2021/07/blog-backlog-1.html#con... Here's my comment which links the "Unix vs. Google" video (and I very much agree based on my first hand experience with Google's incoherent architecture, which executives started to pay attention to in various shake-ups.) https://lobste.rs/s/euswuc/glue_dark_matter_software#c_sppff7 https://lobste.rs/s/euswuc/glue_dark_matter_software#c_sppff... It links to my comment about the closely related "narrow waist" idea in networks and operating systems. That is a closely related concept regarding scaling your "codebase" and interoperability. I have been looking up the history of this idea. I found a paper co-authored by Eric Brewer which credits it to Kleinrock: http://bnrg.eecs.berkeley.edu/~randy/Papers/InternetServices.pdf http://bnrg.eecs.berkeley.edu/~randy/Papers/InternetServices... (was this ever published? I can't find a date or citations) But I'm not done with all the research. I'm not sure if it's worth it to write all this, but I think it's interesting I will learn something by explaining it clearly and going through all the objections. I'm definitely interested in the input of others. I have about 10 different resources where people are getting at this same scaling idea, but I can use more arguments / examples / viewpoints.
• PaulHoule 5y ago
  I wish that were so. Multics made a big splash in the literature but in terms of use it was an obscure os on an obscure mainframe. It had nothing on TOPS-20 or VM/CMS. Unfortunately many of us are suffering with Kube.
• rightisleft 5y ago
  Going to post a lovely update for docker swarm here - Swarm simplifies/reduces the possibility space compared to K8, but i consider that a feature not a drawback. With Mirantis actively hiring and extending support for SwarmKit, it should be considered a viable 'batteries included' alternative to K8: https://github.com/docker/roadmap/issues/175#issuecomment-826312525 https://github.com/docker/roadmap/issues/175#issuecomment-82...
• dannyrosen 5y ago
  Consider looking into Fuchsia's component framework for thoughts on what a distributed application looks like inside an operating system. https://fuchsia.dev/fuchsia-src/concepts/components/v2/introduction https://fuchsia.dev/fuchsia-src/concepts/components/v2/intro...
• cturner 5y ago
  There is a trap in distributed system design - seeking to scale-up from a single-host perspective. An example - we have apache and want to scale it up, so we put it in a container and generate its configuration so we can run several of them in parallel. This leads to unnecessarily heavy systems - you do not need a container to host a server socket. Industry puts algorithms and Big O on a pedestal. Most software projects start as someone building algorithms, with deployment and interactions only getting late attention. This is a bit like building the kitchen and bathroom before laying the foundations. Algorithm centric design creates mathematically elegant algorithms that move gigabytes of io across the network for every minor transaction. Teams wrap commodity resource schedulers around carefully tuned worker nodes, and discover their performance is awful because the scheduler can’t deal in the domain language of the big picture problem. I think it is interesting that the culture of Big O interviews and k8s both came out of Google.
  • zbentley 5y ago
    Do you have any examples/ideas of what a non algorithm-first approach might look like?
    • cturner 5y ago
      Not sure if this is helpful, there are some notes at cthulix.com.
• throwaway984393 5y ago
  I think it will end up with a "simple" Distributed OS in the same way we have internal combustion engines: they're very hard to build, complicated to repair, moderately easy to maintain, very easy to use. Here's the things I think we need in order to make a "simple" Distributed OS: Cutting edge tech. If developers don't want to use it, it dies, period. It needs to be trendy. Novel interaction of different versions of different software components. The model we use today is 40+ years old and does not scale past a single system. We have to make it easy for different versions of software to interact in any way, without making people think hard about it or use hacks. (There are solutions for this already but nobody uses them; we need a trendy blog post and some new code conventions to make them take off) Novel network stack. Distributed systems have been twisting themselves into pretzels for decades to get Component A to talk to Component B over a network. You can have upwards of a dozen different components in between, all just dedicated to getting two components to talk to each other. The thing holding this up is the lack of integration between all the layers, and along hops. Distributed Tracing. You can't troubleshoot a distributed system effectively without it. Lack of debugging tools means the systems won't be used seriously and the effort will die on the vine. Distributed Computing Health Metrics as a higher level abstraction than "is this host-specific resource running out". Basically this requires a gossip network of health metrics and some fancy math to estimate probabilities of health. Distributed Shared Memory for Threaded Applications. Yes, I went there. Building distributed systems will continue to be a pain without it. We have to make these systems stupid easy to program and use; if it takes a PhD or a two-pizza team of amateurs to program for it, it's just not gonna take off. (applies to the "Images and Feelings" part of OP) Versioned Immutable Operating Models. Basically, distributed systems today are not immutable, because various layers of the "stack" that makes them up are not immutable or version-controlled. To reliably operate even a non-distributed system, you need this. It's especially important for SaaS, PaaS, IaaS, etc. We have built whole ecosystems of tools because many parts of a distributed system simply have bad operational models. You can start with building such a model for regular-old software, and each layer of software (and hardware!) around it should also develop such a model. A complete stack with that model will be very determinate, easy to operate, & easy to reason about. I estimate this will make 50% of the current distributed computing ecosystem redundant. Federation, Encryption, Fine-Grained Access Control by default. We need any component to be able to talk to any component in a secure manner, again without jumping through a lot of hoops. Distributed Control and Data Plane separation by default. This is both a novel I/O model, and a novel control plane for all components. Resource Reservation. Software needs to specify the kind and amount of resources it will need before it even runs. This is necessary to prevent the inevitable resource exhaustion churn, ex. when competing pods spin up and die in a loop. Distributed Networking Safety Conventions. The best practice stuff to prevent network storms on crowded resources. Throttling, backoff, jitter, quotas, etc. Distributed Scheduler. Simple idea, difficult implementation. A generic scheduler that is smart enough to schedule all kinds of weird things across distributed systems. Almost all of these things already exist, but that's not the hard part! The hard part is combining them all together in a way people want to use. The only way that's gonna happen is if we start up another research project ala Plan9.
• whalesalad 5y ago
  After reading the title I worried this was going to be yet another k8s bashing post. Pleasantly surprised to see this take because it’s a refreshing look at kube and I strongly agree. I think it’s the absolute best way to deploy large systems today, especially if you’re a polyglot organization. But it can be tough to grok without lots of labbing and experimentation - it’s hard to approach. We are really at the infancy of containerization. Kube is a springboard for doing the next big thing.
• wskish 5y ago
  K8s abstracts away much more complexity than it exposes, which is the hallmark of a great api. History will surely view it amongst the greatest api’s of all time.
• solatic 5y ago
  > A distributed OS that follows the Perlis-Thompson Principle would have fewer concepts. Kubernetes is a relatively simple system with few concepts. You have manifests stored in etcd, behind the API server, and various controllers that act on these manifests. Some controllers (Deployment, StatefulSet, etc.) come standard out of the box, some are custom and added later. The basic unit of computation is a Pod, and DNS is provided with Services. Cluster administrators need to worry about the networking and storage layers, not cluster users. Honestly, that's pretty much it! Really not so complicated. Now, does that help you write a manifest for the Deployment controller? No, and neither does it help you autoscale the Deployment via writing a manifest for the HorizontalPodAutoscaler controller, or setting up a load balancer by writing a manifest for the Ingress controller. But I wouldn't call the UNIX model complex because Linux distributions and package managers add complexity.
• mcv 5y ago
  Most of this stuff is completely over my head, and I'm certainly no kubernetes expert, but I'm working on a project that's deployed with kubernetes, and one of the steps in our process is running our e2e tests, also in a separate kubernetes deploy. These tests (using Cypress) have proven to be extremely flakey on the server. Locally they work fine, though. I was wondering if Cypress is simply crap, but this article makes me wonder if kubernetes might be the real culprit here.
  • speedgoose 5y ago
    Kubernetes for sure. But it will force you to write more relisient software. Since we migrated to kubernetes, we had to implement automatic retry strategies in every network exchange, http requests, database transactions, because the managed kubernetes of a major cloud provider is a train wreck.
• pjmlp 5y ago
  Well, given that Multics was much more secure than UNIX ever was, and written on a proper systems programming language that everyone (except UNIX folks) is trying to get back to, probably isn't that bad after all.
  • bruce343434 5y ago
    > proper systems programming language that everyone (except UNIX folks) is trying to get back to Wikipedia: Written in PL/I, Assembly language ????
    • pjmlp 5y ago
      I advise you to learn about the safety capabilities regarding strings, arrays, pointer manipulation and references, numerics and enumerations in PL/I versus C. Additionally, you can go over to Multicians and read the security assessemt reports of Multics vs UNIX done by DoD, back in the day.
• gerbilly 5y ago
  If you're enjoying your Kubernetes, then have at it, but in my opinion it sounds like Stockholm syndrome. The thing is so complicated that even the guys who wrote it probably can't figure it out. I myself would rather sew together .BAT files, CORBA and COBOL into a shambling software frankenstein before I'd even consider using Kubernetes and get sucked into that mess. But seriously, 99 percent of us, even on HN, don't have the problems that kubernetes is trying to solve. Why do we put ourselves through this when we should know just looking at the thing that it's just going to be a nightmare when things go wrong?
  • deleted 5y ago
    [deleted]
  • froh 5y ago
    .JCL files, not .BAT files and: yes.
• benlivengood 5y ago
  So the basic problem with *nix is its permission model. If we had truly separable security/privilege/resource domains then Linux wouldn't have needed containers and simple processes and threads could have sufficed in place of Borg/docker/Kubernetes. There's a simpler and more powerful security model; capabilities. Capabilities fix 90% of the problems with *nix. There's currently no simple resource model. Everything is an ad-hoc human-driven heuristic for allocating resources to processes and threads, and is a really difficult problem to solve formally because it has to go beyond algorithmic complexity and care about the constant factors as well. The other *nix problem is "files". Files were a compromise between usability and precision but very few things are merely files. Devices and sockets sure aren't. There's a reason the 'file' utility exists; nothing is really just a file. Text files are actually text files + a context-free grammar (hopefully) and parser somewhere, or they're human-readable text (but probably with markup, so again a parser somewhere). Plenty of object models have come and gone; they haven't simplified computers (much less distributed computers), so we'll need some theory more powerful than anything we've had in the past to express relationships between computation, storage, networks, and identities.
  • dijit 5y ago
    Containers never solved the permission model. They solved the packaging and idempotency problem. I really dislike when people assume containers give them security, it’s the wrong thing to think about. Containers allowed us to deploy reproducibly, that’s powerful.
    • foxhill 5y ago
      > I really dislike when people assume containers give them security, it’s the wrong thing to think about. to be fair, there is lots of published text around suggesting that this _is_ the case. many junior to semi-experienced engineers i've known have at some point thought it's plausible to "ssh into" a container. they're seen as light-weight VMs, not as what they are - processes. > Containers allowed us to deploy reproducibly, that’s powerful. and it was done in the most "to bake an apple pie from scratch, you must first create the universe" approach.
• mixxit 5y ago
  i just want cloud agnostic faas with first class triggers and outputs something like lambda and azure functions without feeling locked in
• jillesvangurp 5y ago
  The problem is the devops culture that has burdened development teams with having to juggle a lot of complexity. The solution is having some separation of concerns. Development teams should not have to spend a lot of time on devops. That's something that should just work that you buy from someone. You pay for the privilege of doing more interesting things. Kubernetes becomes a problem when you have people who are not operations people with many years of experience with this stuff trying to do this while learning how to do it at the same time. The related problem is that having people spend time on this is orders of magnitudes more expensive than it is to run an actual cluster, which is also not cheap. A week of devops time easily equates months/years of cloud hosting time for a modestly sized setup using e.g. Google Cloud Run. And lets face it, it's never just a week. Many teams have full time dev ops people costing 100-200$K/year, each. Great if you are running a business generating millions of revenue. Not so great if you are running a project that has yet to generate a single dollar of revenue and is a long time away from actually getting there. That describes most startups out there. I actually managed to stay below the Cloud Run freemium layer for a while making it close to free. Took me 2 minutes to setup CI/CD. Comes with logging, auto scaling, alerting, etc. Best of all, it freed me up to do more interesting things. Technically I'm using Kubernetes. Except of course I'm not. I spent zero time fiddling with kubernetes specific config. All I did was tell Google Cloud run to go create me a CI/CD pipeline from this git repository and scale it. 3 minute job to click together. Service was up and running right after the build succeeded. Great stuff. That's how devops should be: spend a minimum of time on it in exchange for acceptable results.
  • mgkimsal 5y ago
    > Great if you are running a business generating millions of revenue. It's not even great in that situation. Millions in profit, perhaps, but that $200k+ would probably better be spent elsewhere - enhancing functionality, increasing sales, support, etc.
  • parasubvert 5y ago
    "Development teams should not have to spend a lot of time on devops. That's something that should just work that you buy from someone." This is the fundamental disagreement. DevOps was a reaction to developers that build software that was nearly impossible to operate because they treated Ops as servants that paid to do the dirty work, rather than peers with a set of valuable skills that cover a scope beyond what many Dev teams have. And it was a reaction to Ops being ground down into becoming the "department of no", when really they should be at the table with the development team as a way towards a collaborative reality check. A model where one team gets to completely ignore the complexities of operational reality is a broken, inhumane, and unsustainable model. That said, it's also unsustainable to expose all complexity to dev teams that don't have the skills or incentive to manage this. Progressive disclosure and composable abstractions are the tool to remedy this. Kubernetes was never intended to be exposed directly to app developers, it was a system developer's platform toolkit. Exposing it is misunderstanding + laziness on the part of some operations teams. The intent was always to build higher PaaS-like abstractions such as Knative (which is what Google Cloud Run is based on).
• ilovefood 5y ago
  We have both Nomad (Consul + Vault + Nomad) and Kubernetes (hosted and on prem) running, both excel at different things. I love Nomad's flexibility and ease of use, a simple hcl file and I (and all the devs) can debug and understand what is going with the deployment without wasting a whole sprint, debugging and understanding the systems is trivial. However I agree parts of the documentation should be fixed and can confuse people who want to start up and it's also relatively "new" insofar that there is a small but growing community around it. I love Kubernetes because of the community, if there's a Helm chart for a service, it's going to work in 80% of the cases. If however there are bugs in the helm chart, or something is quite not on the beaten path, then good luck. Most of the time wasted on Kubernetes was the inexperience of the operators and also the esoteric bugs that can happen now and then. Building on top of things that have been done before is a great way to win time and flexibility but it shouldn't be an excuse to not understand them (helm charts as an example). In both cases, you always need an ops team to take care of the clusters. For Nomad, 2/3 people are enough. For Kubernetes you will need 5+ people depending on the size and locality of the cluster, if you want to do things right, that is. If your dev team is managing them it's already game over and just a question of time until you made yourself more real problems than you initially had. What bugs me the most however is the cargo culting around the tools serving as a "beating around the bush" technique to not do actual work. They're just that, tools, if you have to deploy a rails or django app with an sqlite database just do it on metal with a two liner "ci/cd" and grow from there. If it gets bigger, sure, go for Kubernetes to manage the deployments and auto scale, but be damn sure that you can debug anything that goes wrong within minutes/hours. If things go wrong and there's no hit on your googled error code you essentially fall from your highest level of abstraction and are at the mercy of consultants that will both waste your time in writing requirements and waste your money by taking too much time than was initially planned and agreed upon (my experience, sample size N=6).
• parasubvert 5y ago
  Claiming Kubernetes is Multics , and that the UNIX equivalent is around the corner, is worthless claim without actual data or argument to back it up. To me, Kubernetes is the new UNIX, centered around a small number of core ideas: controller loops, Pods, level-triggered events, and a fully open, well-standardized, and declarative, and extensible RESTful API. Kubernetes has its complexities - just like UNIX, because it's trying to solve two big problems: shifting the fundamental unit of computation into an immutable / ephemeral unit (rather than mutable), i.e. the Pod, and having a single open API for controlling almost every aspect of IT using control systems theory as the philosophy. The various clouds and predecessor cloud orchestrators (Azure ARM, AWS Cloud Formation, etc) are (to me) the infinitely complicated beasts. This article didn't have an argument beyond "I don't understand it, and therefore I don't like it". He just linked to a few rants about the complexity of the CNCF ecosystem (which is like complaining that "IT is complicated" - it is a reflection of reality, not Kubernetes), and extended cranky rant / thought exercise by the MetalLB dude. The latter is the closest to an actual argument against Kubernetes, but there’s a LOT of things to disagree with in that post. THAT would be an interesting debate. The biggest issue with Kubernetes is the insularity of the culture to reject anything that doesn't think like Kubernetes (as defined by whoever might be running any given SIG). That is also its greatest strength. But if it doesn't compromise this vision in some respects, such as developer experience, it will be self-limiting.
• gerbilly 5y ago
  More like our genenration’s EJBs (https://www.oracle.com/java/technologies/enterprise-javabeans-technology.html https://www.oracle.com/java/technologies/enterprise-javabean...)
• soobrosa 5y ago
  Add the two cents of http://adamierymenko.com/ports.html http://adamierymenko.com/ports.html "Since we chose the path of virtualization and containerization we've allowed the multi-tenancy facilities in Unix to atrophy and it would take a little bit of work to bring them back into form."
• nwmcsween 5y ago
  I thought the same as well but then I went down the docker/container route for making something similar to k8s it turned out to be just reimplementing k8s badly. The reason k8s is so complicated is the horrible infatuation with walls of YAML and CRDS, think of YAML and CRDS as XML and XMLNS for those of us who lived through XML.