4 ms·

> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed dec

> Kubernetes is complex and I think they are partially right Kubernetes is a distributed centralized operating system which itself depends on a distributed decentralized database, and has a varying network topology, permissions system, plugins, scheduler, storage, and much more, depending on how & where it was built, and runs applications as independent containerized environments (often deeply dependent on Linux kernel features) which can all have their own base operating systems. All of which must be maintained, upgraded, patched, and secured, separately, and frequently. Kubernetes is literally the most complex single system that almost anyone in the world will ever use. It is the Katamari Damacy of the cloud. > It allows dev teams to not worry about all these things; all they must do is to write a simple YAML file. cackles, then sobs > More importantly, teams no longer need to ask DevOps/infra folks to add DNS entry and create a Load Balancer just to expose a service. more sobbing > Should you use Kubernetes? Should you change a tire with a crowbar, a can of WD40, and a lighter? Given an alternative, the alternative is usually better, but sometimes you don't have an alternative.

dekhn 4y ago
if you think k8s is the most complex system anyone in the world will ever use: 50K googler using borg beg to disagree.
- dilyevsky 4y ago
  Actually most devs should have a much easier time with borg than k8s bc there’s an army of sres/devs running it for them. I was one of those running it but even I didn’t appreciate the complexity of running airtight kubernetes distro at the beginning. That’s on onprem tho, if you don’t know how to network just use the cloud thing
- foobiekr 4y ago
  Is borg as operationally broken as k8s? Also, I would guess borg directly fits the google applications. k8s is not really a great match for many others.
  - dekhn 4y ago
    I'm not sure what you mean by operationally broken. While I would agree that k8s is not a great match for many, I wouldn't say it's fundamentally broken. It still has many rough edges. What's amazing about borg is the number of clusters, their sizes, and the support teams that keep them healthy. k8s does not have that.
- master_crab 4y ago
  This is a bad argument. You are basically admitting that companies need google-level resources to hire dozens (hundreds?) of infra engineers to maintain a K8s-type infra.
  - dekhn 4y ago
    Where did I say that k8s needs that? I just said borg is more complex than k8s, so your argument doesn't even hold. Now, turning things around: companies need to hire more infra engineers, that's true.
    - paulddraper 4y ago
      > companies need to hire more infra engineers Unless they don't.
hartator 4y ago
> t allows dev teams to not worry about all these things; all they must do is to write a simple YAML file. A "simple" YAML file.
paulddraper 4y ago
> a simple YAML file Oooooof
Rantenki 4y ago
OMFG so much this. I worked at a large company that deployed it's own Kubernetes stack, on a VERY large number of physical hosts. The theory was that the K8S would simplify our devops story enough that we could iterate quickly and scale linearly. In reality, the K8S team ended up being literally 10x larger than the team building the application we were deploying on it. In addition, K8S introduced entirely new categories of failure mode (ahem; CNI updates/restarts/bedsh*tting, operator/custom resource failures, and tons of other ego driven footguns). The worst part? The application itself ran fine on a single dev workstation, but also on any random assortment of VMs. Just pass the consul details as environment variables. I am not saying everybody on K8S is in the same boat, but I think that far more people are planning on becoming a unicorn cloud service than have any hope of becoming a unicorn cloud service. TL;DR: If your hosting solution requires more maintenance than the application itself, you made a boo-boo.
- physicles 4y ago
  Using k8s is not remotely the same thing as maintaining your own k8s stack. One is easy (get your feet wet in an afternoon), the other is hard (maybe after a couple months of full-time study you can pull it off). The vast majority of teams that have enough crap to run to warrant using k8s should not be maintaining their own k8s stack. In fact, it’s entirely possible that running k8s is so hard that the only players that can do it reliably are the big cloud companies.

dekhn 4y ago

if you think k8s is the most complex system anyone in the world will ever use: 50K googler using borg beg to disagree.

dilyevsky 4y ago
Actually most devs should have a much easier time with borg than k8s bc there’s an army of sres/devs running it for them. I was one of those running it but even I didn’t appreciate the complexity of running airtight kubernetes distro at the beginning. That’s on onprem tho, if you don’t know how to network just use the cloud thing
foobiekr 4y ago
Is borg as operationally broken as k8s? Also, I would guess borg directly fits the google applications. k8s is not really a great match for many others.
- dekhn 4y ago
  I'm not sure what you mean by operationally broken. While I would agree that k8s is not a great match for many, I wouldn't say it's fundamentally broken. It still has many rough edges. What's amazing about borg is the number of clusters, their sizes, and the support teams that keep them healthy. k8s does not have that.
master_crab 4y ago
This is a bad argument. You are basically admitting that companies need google-level resources to hire dozens (hundreds?) of infra engineers to maintain a K8s-type infra.
- dekhn 4y ago
  Where did I say that k8s needs that? I just said borg is more complex than k8s, so your argument doesn't even hold. Now, turning things around: companies need to hire more infra engineers, that's true.
  - paulddraper 4y ago
    > companies need to hire more infra engineers Unless they don't.

hartator 4y ago

> t allows dev teams to not worry about all these things; all they must do is to write a simple YAML file. A "simple" YAML file.

paulddraper 4y ago

> a simple YAML file Oooooof

Rantenki 4y ago

OMFG so much this. I worked at a large company that deployed it's own Kubernetes stack, on a VERY large number of physical hosts. The theory was that the K8S would simplify our devops story enough that we could iterate quickly and scale linearly. In reality, the K8S team ended up being literally 10x larger than the team building the application we were deploying on it. In addition, K8S introduced entirely new categories of failure mode (ahem; CNI updates/restarts/bedsh*tting, operator/custom resource failures, and tons of other ego driven footguns). The worst part? The application itself ran fine on a single dev workstation, but also on any random assortment of VMs. Just pass the consul details as environment variables. I am not saying everybody on K8S is in the same boat, but I think that far more people are planning on becoming a unicorn cloud service than have any hope of becoming a unicorn cloud service. TL;DR: If your hosting solution requires more maintenance than the application itself, you made a boo-boo.

physicles 4y ago
Using k8s is not remotely the same thing as maintaining your own k8s stack. One is easy (get your feet wet in an afternoon), the other is hard (maybe after a couple months of full-time study you can pull it off). The vast majority of teams that have enough crap to run to warrant using k8s should not be maintaining their own k8s stack. In fact, it’s entirely possible that running k8s is so hard that the only players that can do it reliably are the big cloud companies.