5 ms·

One of the things I hear the most about Kubernetes is the complexity. Honestly, it's not that complex from an user perspective. It's not much more complicated t

One of the things I hear the most about Kubernetes is the complexity. Honestly, it's not that complex from an user perspective. It's not much more complicated than Docker, really. It's also super nice to just write two yaml files and have everything running nicely with rollouts, auto-scaling, node pinning etc. It just works and I never have to even think about the internals (although I took the time to learn most of them). Adding: heck, I'll run a single node k3s on my computer (which takes one (1) command) instead of using docker-compose just because I like it better than compose.

ninkendo 4y ago
Yeah most of the (IMO) legitimate complaints about the difficulty of k8s are about how hard it is to correctly set up and administer. (Networking setup, making ingress work, cloud service integration, etc.) Especially when you get into permissions/namespaces and trying to get that right for a decent size organization and trying to configure things for least privilege. Simply using it isn’t that hard IMO.
- steveBK123 4y ago
  Right but this is the irony I think. If you need the scale provided by k8s, you are probably in a large org. If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed. Setting up a science project at home in your toy AWS account I'm sure is no big deal. But this is true of every tech- "works on my box" is always easier than "doing this in enterprise". The measure of a technology is how easy it is to use in the real production environment.
  - trallnag 4y ago
    Being in a large organization does not equal having a dedicated team for running Kubernetes clusters. For example Daimler (the company behind brands like Mercedes) has hundreds of Kubernetes clusters across hundreds of small product teams.
- jasonjayr 4y ago
  IMHO, the biggest annoyance and difficulty with K8s, is figuring out the helm chart ecosystem with the crazy templated YAML. Some of those documents can get quite convoluted. One family of charts (say, from Bitami) will have one set of conventions for values.yaml, and other families will have a slightly different way of organizing this.
  - akvadrako 4y ago
    You can just not use helm. Kustomize, which is built-in, does almost everything needed and is much simpler.
    - itintheory 4y ago
      But then you have an entirely different challenge if you want to use a project that only provides a chart as a deployment package / docs.
      - jpdb 4y ago
        Try checking out kpt, it does packages and hydration. https://kpt.dev/ https://kpt.dev/
danpalmer 4y ago
Kubernetes is complex if you don't need what it provides. However if you do need what it provides in _some way_, Kubernetes provides a pretty nice, coherent way of doing it.
- steveBK123 4y ago
  The problem is a lot of people read about k8s and decided stuff that could be done with a queue, 3 micro services, and 4 cores of CPU running on a RasPi somehow needs ... k8s.
  - danpalmer 4y ago
    This might be true, but one of the under-appreciated benefits of Kubernetes is the control around deployments. 3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc. It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.
bravetraveler 4y ago
I find consuming K8s to be like this - it's great, provides facilities for common everyday things I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it. It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking. My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.
- adra 4y ago
  I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.
  - ilyt 4y ago
    You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit. The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
MuffinFlavored 4y ago
Could you share those 2 YAML files? I feel like it's not hard to do k8s YAML the wrong way and miss out on the auto-rollout, auto-scaling, etc. features
- trallnag 4y ago
  You are implying that he needs all these automations.
ilyt 4y ago
It moves most/all of the complexity on ops side. or in "whatever automation thingy you use to deploy it". Yes, easy on dev, but compared to "just docker" or "a VM with some systemd service running your app" it is hideously complex

ninkendo 4y ago

Yeah most of the (IMO) legitimate complaints about the difficulty of k8s are about how hard it is to correctly set up and administer. (Networking setup, making ingress work, cloud service integration, etc.) Especially when you get into permissions/namespaces and trying to get that right for a decent size organization and trying to configure things for least privilege. Simply using it isn’t that hard IMO.

steveBK123 4y ago
Right but this is the irony I think. If you need the scale provided by k8s, you are probably in a large org. If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed. Setting up a science project at home in your toy AWS account I'm sure is no big deal. But this is true of every tech- "works on my box" is always easier than "doing this in enterprise". The measure of a technology is how easy it is to use in the real production environment.
- trallnag 4y ago
  Being in a large organization does not equal having a dedicated team for running Kubernetes clusters. For example Daimler (the company behind brands like Mercedes) has hundreds of Kubernetes clusters across hundreds of small product teams.
jasonjayr 4y ago
IMHO, the biggest annoyance and difficulty with K8s, is figuring out the helm chart ecosystem with the crazy templated YAML. Some of those documents can get quite convoluted. One family of charts (say, from Bitami) will have one set of conventions for values.yaml, and other families will have a slightly different way of organizing this.
- akvadrako 4y ago
  You can just not use helm. Kustomize, which is built-in, does almost everything needed and is much simpler.
  - itintheory 4y ago
    But then you have an entirely different challenge if you want to use a project that only provides a chart as a deployment package / docs.
    - jpdb 4y ago
      Try checking out kpt, it does packages and hydration. https://kpt.dev/ https://kpt.dev/

danpalmer 4y ago

Kubernetes is complex if you don't need what it provides. However if you do need what it provides in _some way_, Kubernetes provides a pretty nice, coherent way of doing it.

steveBK123 4y ago
The problem is a lot of people read about k8s and decided stuff that could be done with a queue, 3 micro services, and 4 cores of CPU running on a RasPi somehow needs ... k8s.
- danpalmer 4y ago
  This might be true, but one of the under-appreciated benefits of Kubernetes is the control around deployments. 3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc. It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.

bravetraveler 4y ago

I find consuming K8s to be like this - it's great, provides facilities for common everyday things I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it. It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking. My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.

adra 4y ago
I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.
- ilyt 4y ago
  You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit. The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.

MuffinFlavored 4y ago

Could you share those 2 YAML files? I feel like it's not hard to do k8s YAML the wrong way and miss out on the auto-rollout, auto-scaling, etc. features

trallnag 4y ago
You are implying that he needs all these automations.

ilyt 4y ago

It moves most/all of the complexity on ops side. or in "whatever automation thingy you use to deploy it". Yes, easy on dev, but compared to "just docker" or "a VM with some systemd service running your app" it is hideously complex