2 ms·

I find consuming K8s to be like this - it's great, provides facilities for common everyday things I find building K8s (well) to be wildly different. CNIs, sto

I find consuming K8s to be like this - it's great, provides facilities for common everyday things I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it. It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking. My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.

adra 4y ago
I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.
- ilyt 4y ago
  You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit. The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
  - GauntletWizard 4y ago
    That, and not having to pay for VMware's licensing. Big factor in the adoption.
itintheory 4y ago
> For a 'solution', you have to build a lot of it. This is true even in the 'managed' k8s world (like EKS). They'll automatically configure EBS storage for you, but that's not sufficient for non-trivial workloads. Same with networking, logging, or metrics. I like that EKS means I don't have to worry about etcd backups and restores, but it's not a production ready platform by any stretch.
mmh0000 4y ago
This is why you should never run plain-old-Kubernetes in prod (or in dev, if you want my opinion). Stock Kubernetes basically ignores security and, then, as you said, it requires a bunch of additional work to config. There are several companies that take Kubernetes and polish it up, the two biggest examples are Red Hat OpenShift[0] (which I'm most familiar with, because I manage it as my day job), and Rancher[1]. Also, note OpenShift is a paid product, but Red Hat also releases OKD[2] for free, which is OpenShift without support. OpenShift is Kubernetes with great default security policies. Storage, networking, logging, and more are all configured out of the box. I would recommend anyone who is using or looking to use Kubernetes to strongly consider OpenShift or OKD. [0] https://developers.redhat.com/products/openshift/overview https://developers.redhat.com/products/openshift/overview [2] https://www.okd.io/ https://www.okd.io/ [1] https://www.rancher.com/ https://www.rancher.com/
- Turbots 4y ago
  [dead]

adra 4y ago

I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.

ilyt 4y ago
You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit. The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
- GauntletWizard 4y ago
  That, and not having to pay for VMware's licensing. Big factor in the adoption.

itintheory 4y ago

> For a 'solution', you have to build a lot of it. This is true even in the 'managed' k8s world (like EKS). They'll automatically configure EBS storage for you, but that's not sufficient for non-trivial workloads. Same with networking, logging, or metrics. I like that EKS means I don't have to worry about etcd backups and restores, but it's not a production ready platform by any stretch.

mmh0000 4y ago

This is why you should never run plain-old-Kubernetes in prod (or in dev, if you want my opinion). Stock Kubernetes basically ignores security and, then, as you said, it requires a bunch of additional work to config. There are several companies that take Kubernetes and polish it up, the two biggest examples are Red Hat OpenShift[0] (which I'm most familiar with, because I manage it as my day job), and Rancher[1]. Also, note OpenShift is a paid product, but Red Hat also releases OKD[2] for free, which is OpenShift without support. OpenShift is Kubernetes with great default security policies. Storage, networking, logging, and more are all configured out of the box. I would recommend anyone who is using or looking to use Kubernetes to strongly consider OpenShift or OKD. [0] https://developers.redhat.com/products/openshift/overview https://developers.redhat.com/products/openshift/overview [2] https://www.okd.io/ https://www.okd.io/ [1] https://www.rancher.com/ https://www.rancher.com/

Turbots 4y ago
[dead]