10 ms·

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstract

• andrewstuart2 5y ago
  From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective. Most of the complexity can be ignored (until you want it). It's the same reason I still like to use postgres when I can versus NoSQL until I know I need the one feature I may not be able to achieve with postgres: automatic sharding for massive global scale. The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. It's also the same reason bash is great for tasks and small automated programs, but kind of terrible for more complex needs. The primitives are super simple, easy to grok, and crazy productive at that scale, but other languages give tools to more easily handle the complexities that start to emerge when things get more complicated.
  • xtracto 5y ago
    > The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. But that's just basically a calculated tradeoff of Postgres (and several CP databases) trading Availability for Consistency.
    • kbenson 5y ago
      Probably less calculated and more "that's what's available to offer stably right now that we can feasibly deliver, so that's what we'll do." Distributed RDBMS were not exactly cheap or common in open source a couple decades back. I don't think there was much of a choice to make.
      • spookthesunset 5y ago
        I mean it is a trade off though. You cannot beat the speed of light. The further apart your database servers are, the more lag you get between them. If you want a transactional, consistent datastore you are gonna have to put a lock on something while writes happen. And if you want consistency it means those locks need to be on all systems in the cluster. And the entire cluster needs to hold that lock until the transaction ends. If your DB’s are 100ms apart… that is a pretty large, non negotiable overhead on all transactions. If you toss out being fully consistent as a requirement, things get much easier in replication-land. In that case you just fucking write locally and let that change propagate out. The complexity then becomes sorting out what happens when writes to the same record on different nodes conflict… but that is a solvable problem. There will be trade offs in the solution, but it isn’t going against the laws of physics.
        • kbenson 5y ago
          Yes, but my point was there wasn't really a choice to make at that time, therefore no trade off. Even if I won $100 in the lotto today and had the money in hand, I wouldn't describe my choice which house I bought years ago as a calculated trade off between what I bought and some $10 million dollar mansion. That wasn't a feasible choice at that time. Neither was making a distributed RDBMS as an open source project decades ago, IMO.
          • lazide 5y ago
            Wasn’t MySQL (pre-Oracle) an open source distributed RDBMS decades ago? At least I remember running it using replication in early 2000’s
  • brightball 5y ago
    At the moment you need massive sharding options with Postgres, you've got a lot of options. I'd assume by the time you get there, you can probably budget for them pretty easily as well. Citus is specifically designed for it. Timescale appears to have the use case covered too. Ultimately though, sharding is one of the easier ways to scale a database out. NoSQL just does it by eliminating joins. You can do that pretty easily with almost any solution, PG or otherwise.
  • specialp 5y ago
    It doesn't take long working with Nomad to hit the cases where you need to augment it. Now I know some of people enjoy being able to plug and play the various layers that you get in the complicated kitchen sink Kubernetes. We already had something that just ran containers and that was Mesos. They had the opinion that all the stuff like service discovery could be implemented and handled by other services like Marathon. But it did not work well. Most people that was to deploy containers in an orchestrated manner want service discovery. At least these parts are being provided by the same company (Hashicorp) so it probably won't suffer the same lack of coordination between separate projects that Mesos did. The benefit to the kitchen sink opinionated framework that K8s does is that your deployments and descriptors are not very difficult to understand and can be shared widely. I do not think the comparison of massively sharded NoSQL to Postgres is the same because most people will not need massive sharding, but almost everyone is going to need the service discovery and other things like secrets management that K8s provides.
    • wg0 5y ago
      One of the things that I don't like about Nomad is HCL. It is a language that is mainly limited to HashiCorp tools and there's no wider adoption outside at least not to my knowledge. From the documentation: > Nomad HCL is parsed in the command line and sent to Nomad in JSON format via the HTTP API. So why not just JSON or even JSON at all and not MsgPack or just straight up HCL because that's over and over introduced as being machine readable and human friendly both at the same time?
      • orthecreedence 5y ago
        To my understanding, you can write most (all?) of the job/config files in JSON if you wish. At my company, we have a ton of HCL files because in the beginning it was easier to hand-write them that way, but we're now getting to the point where we're going to be templating them and going to switch to JSON. In other words, I believe HCL is optional.
      • gizdan 5y ago
        JSON was designed for machine readability, HCL was designed for human readability. HCL requires a lot more code to parse and many more resources to keep in memory vs JSON. I think it completely makes sense to do it this way. K8s is the same. On the server it does everything in JSON. Your YAML gets converted prior to sending to K8s.
  • spookthesunset 5y ago
    > From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective. I’ve been doing software for 20 years in one form or another. One of the things I’ve learned is the simpler and more polished something seems to the user, it is almost always because there is a hell of a lot of complexity under the covers to make it that way. Making something that handles the 80% is easy. Every step closer to 100% becomes non-linear in a hurry. All that polish and ease of use took months of “fit & finish” timeboxes. It took tons of input from UX and product. It involved making very hard decisions so you, the user, don’t have to. A good example is TurboTax online. People love to hate on their business practices (for good cause) but their UX handles like 98% of your common tax scenarios in an incredibly easy to use, highly polished way. Robinhood does a pretty good job too, in my opinion—there is a lot of polish in that app that abstracts away some pretty complex stuff.
• throwaway923840 5y ago
  They're both complex. But one of them has 10 times the components than the other, and requires you to use them. One of them is very difficult to install - so much so that there are a dozen different projects intended just to get it running. While the other is a single binary. And while one of them is built around containers (and all of the complexity that comes with interacting with them / between them), the other one doesn't have to use containers at all.
  • wg0 5y ago
    k0s is a single binary.
  • dividedbyzero 5y ago
    Which is which?
    • jaaron 5y ago
      From parent's comment: k8s = 10x the components & difficult to install. Nomad = single binary, works with but doesn't require containers.
    • steeleduncan 5y ago
      I believe that the one that requires containers is Kubernetes. Nomad doesn't require containers, it has a number of execution backends, some of which are container engines, some of which aren't. Nomad is the single binary one, however this is a little disingenuous as Nomad alone has far fewer features than Kubernetes. You would need to install Nomad+Consul+Vault to match the featureset of Kubernetes, at which point there is less of a difference. Notwithstanding that, Kubernetes is very much harder to install on bare metal than Nomad, and realistically almost everyone without a dedicated operations team using Kubernetes does so via a managed Kubernetes service from a cloud provider.
  • gizdan 5y ago
    > But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?
• candiddevmike 5y ago
  Agreed. With k3s and friends, nomad is more complicated and requires other components (Consul, some kind of ingress) to match what you get out of the box.
• websap 5y ago
  I couldn't disagree with this more. I manage and run Kubernetes clusters as part of my job, and I can tell you that configuring, running and installing these clusters is no small feat. I don't know much about Nomad, but I would highly encourage most users to not think K8S is simple by any standard. Multiple cloud providers now provide ways to run your code directly, or your containers directly. Unless you have a reason to manage that complexity, save yourself some heartburn and make your developers happy that they just need to worry about running their code and not what happens when a seemingly innocuous change might cause a cascading failure. This has some great account of war stories - https://k8s.af/ https://k8s.af/
  • llama052 5y ago
    Honestly if you use a managed kubernetes provider it's pretty simple once you nail the concepts. Though you'll get hit with cloud provider issues every now and then but it's really not terrible. I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.
    • zie 5y ago
      Sure, you pay someone else to keep k8s alive, it's not so bad, but it's expensive to do that. You generally need a full-time team of people to keep a k8s deployment alive if you are running it yourself. I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.
      • k8sToGo 5y ago
        You can also use something like Rancher or k3s to keep it alive part-time.
        • InvertedRhodium 5y ago
          RKE2 is the closest thing we've found to a turnkey, on prem, production ready Kubernetes distribution. Though we were last looking about a year ago so I concede there might be better or comparable options now.
• jeffnappi 5y ago
  > Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. Are you referring to actually spinning up and operating your own clusters here or utilizing managed k8s (e.g. GKE/EKS)? In my understanding, sure - using the managed services and deploying simple use cases on it might not be that big of a deal, but running and maintaining your own k8s cluster is likely far more of a challenge [1] than Nomad as I understand it. [1] https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
  • zaat 5y ago
    Kubernetes the hard way is an educational resource, meant to server as a resource for those interested in deep diving into the platform, similar to Linux from scratch. Like the statement on the label: > Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster. > The results of this tutorial should not be viewed as production ready, and may receive limited support from the community, but don't let that stop you from learning! If you want to spin up cluster that you actually want to use you'll pick one of the many available free or paid distros, and spinning up something like k3s with rancher, microk8s or even the pretty vanilla option of kubeadmin is pretty simple.
• jbluepolarbear 5y ago
  It’s wouldn't have that reputation if it was easy. There’s too many things that can go wrong for it to be easy.
• richwater 5y ago
  Running a production ready, high availability Kubernetes cluster with proper authn, authz, and resource controls is about the farthest thing from "simple" that I can imagine.
• cfigueir 5y ago
  The comparison is a bit misleading. As someone that has used both nomad and k8s at scale -- - Nomad is a scheduler. Clean and focused. It is very fast. I was an early user and encountered a number of bumps, but that's software. The people at Hashicorp are super sharp and lovely. - K8s is a lot more. It includes a scheduler, but in the simplest sense, it is a complete control-plane based on the control-loop pattern. You have an API, a scheduler, a db, various controllers, etc. Forget for a moment that most people use it to orchestrate containers -- it's really designed to orchestrate anything. Its API is extensible, you can add and compose controllers -- there are many possibilities once you wrap your head around it. This is all opinionated and includes a lot of capability. It's just very different. You can stitch together nomad, consul, vault, and various glue to create a container orchestration system... but when you start wanting to manage the control-planes as though they are the "kind" (the container for example) with meta-control-planes, and you start wanting to orchestrate network, storage, and other dependencies... all while doing this in a multi-tenant environment, then things get interesting. -charles.
  • still_grokking 5y ago
    I'm not sure it's a good idea to use Google's "monolithic cloud operation system". Such a monoculture has the same issues as MS Windows had, even for the same reasons. The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.
    • m00x 5y ago
      K8s itself is divided in multiple parts, where you can customize to your own liking, and you can swap parts if you'd like as long as the APIs are similar. It's very much built the UNIX way.
      • still_grokking 5y ago
        Where can I find those alternative elements that can be swapped? If this is true there should be a lot of them, right?
• mcdevilkiller 5y ago
  Definitely the contrary in my experience
• HeavyStorm 5y ago
  Profoundly disagree. In fact, your statements are not true, since before you can run a pod you will most probably need to create a deployment. And then you'll need a service to expose your workload. Anything but the most trivial workloads will also lead you into questions of how to mount volumes, cijfure) use configmaps and secrets, etc. And that's not even touching the cluster configuration, which you can skip over if you are using a cloud provider that can provision it for you.