4 ms·

May be this a hated take but just wondering - The place that pretty much invented cluster orchestration and reinvented it as k8s is having problems upgrading it

by wg0 3y ago

May be this a hated take but just wondering - The place that pretty much invented cluster orchestration and reinvented it as k8s is having problems upgrading it. What chance a bunch of poor sys admins stand running bunch of k8s clusters for a mid size company I wonder. Every time I think of deploying (self managed, have done full stack) it for something mission critical, this upgrade scenario simply makes me rethink it altogether. And even managed k8s has no guarantees and if managed is to be the option, nothing beats ECS in simplicity and smooth operation at certain scales. PS: Full stack K8s means ingress controllers, DNS auto registration, GitOps , logging, monitoring, CI/CD and all the bells and wistles including a management UI behind oauth etc.

talonx 3y ago
What bugs me most about GKE is that incidents seem to span a whole of datacenters simultaneously (source: see zones affected by previous incidents), and you are often left with no option except wait for a resolution. Often there are issues like networking problems affecting almost the entire global GKE control plane. Whatever happened to resilience by isolation?
OhSoHumble 3y ago
What's really unfortunate is that it feels like not choosing k8s is career suicide for SREs/DevOps/platform engineering. I chose Nomad for my current role and it works fantastic. I used ECS in my last role and it was also fantastic. I'm currently gently looking for a job and the market is brutal. I'm not even getting responses back from applications and I have ten years of experience. Every job posting I see requires deep k8s knowledge and hands-on experience with some ancillary k8s project like ArgoCD or Flux. I'm actually thinking about downplaying non-k8s experience on my resume and just floating the k8s that I do have straight to the top because, at the very least, it'll get my resume looked at. If I was going to go back in time, I'd pick k8s instead of Nomad or ECS even if they weren't the best contextual choices because, hey, I gotta' put food on the table.
- ricc 3y ago
  Quick tip: You can still put your target keywords where you have zero to minimal experience in your resume/CV, just put it in an honest statement describing how much experience you have. In your case, you could write, for example, "X years of experience in HashiCorp Nomad and AWS ECS (Kubernetes alternatives for container orchestration)". That should put your resume/CV through the keyword filter without lying about it.
harikb 3y ago
An equally hated contrary take - may be it is our (as users of k8s) desire to do in-place upgrades without downtime.. Sometimes a clean install of fresh cluster and switch over is so much simpler if we can just take a few minutes downtime to cut over. Complex requirements (sometimes) end up with complex solutions.
- wg0 3y ago
  It is simply not feasible if you have stateful workloads for some reason. Or at least isn't as straight forward. Booting up a cluster is fine but the real struggle starts from that point onwards.
- xeromal 3y ago
  I've never used K8s in my life but if you have to deal with downtime, I'd question the use of K8s when I can do it the old and busted way.
  - harikb 3y ago
    I am not sure what "old and busted way" you are referring to, but k8s does have value, particularly when we are not upgrading k8s itself It would be good to think of the occasional downtime during a significant upgrade (once in 6 months?) separate from the ability to automatically replace failed nodes, scale up and down, having all the k8s magic.
    - PH95VuimJjqBqy 3y ago
      I would imagine using a load balancer in front of multiple servers across several datacenters coupled with a stateless design that allows traffic to flow freely across either of them. there are other variations of this, but this seems the most obvious. blue/green deploys and the like.
- llama052 3y ago
  I think it's definitely overlooked. You should always have the ability to spin off a new cluster and run your workloads on that. At a minimum as a disaster recovery scenario.
moondev 3y ago
Check out cluster-api. Complete zero downtime cluster upgrades is as simple as bumping the version and machine image name in the cluster manifest - or ClusterClass if you run classy clusters
- markstos 3y ago
  And why is that expected to be more reliable than Google’s service?
Freedom2 3y ago
While I see your point and agree, there's also the view that the people who invented the tech are simply different than the kind of people who Google hires nowadays - pure algo focused LeetCoders with zero-to-little operational experience and little interest in even learning how to run and maintain software.
eitally 3y ago
It used to be straightforward when Google Cloud just offered everyone the same version of GKE essentially everywhere. It was one of not-too-many services that are vanilla and identical in pretty much every region. But, newer offerings have fractured things a bit (sovereign cloud, Google Distributed Cloud, FedRAMP, Tencent, etc) and I can see why it would become problematic to adequately test and keep everything in sync while also not breaking interfaces and/or downstream services. Also, Google used to allow customers essentially 0 say in when upgrades to services were applied. Now, as part of many vendor security audit processes, it's a hard stop requirement that customers are able to test new revs before applying to prod ... and GKE, like GCE, is one of those critical services.
0xbadcafebee 3y ago
The design of K8s is ridiculous. The fact that you're forced to upgrade every 9 months is even more absurd. It's "opinionated", but the "opinions" are those of one of the largest companies in the world, who hire the most engineers in the world, and everyone else is supposed to operate like that now. K8s is a great example of a "tech jobs program": technology that is absurdly complicated to the point that you have to pay someone lots of money to make it keep working, but you can't afford not to pay for this, because everyone has decided to be stuck on it and now there's no alternative (and probably never will be)
- wg0 3y ago
  The biggest foot gun and most complicated part IMHO is networking. It is a blackbox with many competing implementations and each implementation even differing in implementation details within it's own versions. In theory, it seems reasonable to say that K8s networking is an implemention detail which you shouldn't bother about but in practice, it's going to bite you because of some misconfigured component somewhere someday and you would have no idea what to make of thousands of iptable rules or BGP routes or even least inspectable eBPF mini programs. Paid support, consulting.
  - Too 3y ago
    This. Most application developers don’t give a rats ass about how 2 nodes are connected as long as the pods can speak to each other. Throw these guys in a devops team and give them the task to setup a 2 node k8s cluster and guess how much consideration they will pay to networking. Correct, zero. Except you can’t, because you have to choose a CNI plugin or choose a distribution that has made an opinionated choice for you. To lower the barrier of entry. A default installation must become a lot more opinionated with much less options and more sane and safe default.
- sangnoir 3y ago
  > K8s is a great example of a "tech jobs program" No, it is not. K8s is an example of how it's impossible to be a jack of all trades and retain simplicity. If a tool seeks to be all things to everyone, then there's no sweeping the inherent complexity under the carpet. Full-coverage solutions to complex domains are themselves complex by necessity - see SAP, Salesforce, Oracle, Windows backwards compatibility, or try writing a parser/stemmer for a spoken language that evolved naturally :)

by wg0 3y ago

talonx 3y ago

What bugs me most about GKE is that incidents seem to span a whole of datacenters simultaneously (source: see zones affected by previous incidents), and you are often left with no option except wait for a resolution. Often there are issues like networking problems affecting almost the entire global GKE control plane. Whatever happened to resilience by isolation?

OhSoHumble 3y ago

What's really unfortunate is that it feels like not choosing k8s is career suicide for SREs/DevOps/platform engineering. I chose Nomad for my current role and it works fantastic. I used ECS in my last role and it was also fantastic. I'm currently gently looking for a job and the market is brutal. I'm not even getting responses back from applications and I have ten years of experience. Every job posting I see requires deep k8s knowledge and hands-on experience with some ancillary k8s project like ArgoCD or Flux. I'm actually thinking about downplaying non-k8s experience on my resume and just floating the k8s that I do have straight to the top because, at the very least, it'll get my resume looked at. If I was going to go back in time, I'd pick k8s instead of Nomad or ECS even if they weren't the best contextual choices because, hey, I gotta' put food on the table.

ricc 3y ago
Quick tip: You can still put your target keywords where you have zero to minimal experience in your resume/CV, just put it in an honest statement describing how much experience you have. In your case, you could write, for example, "X years of experience in HashiCorp Nomad and AWS ECS (Kubernetes alternatives for container orchestration)". That should put your resume/CV through the keyword filter without lying about it.

harikb 3y ago

An equally hated contrary take - may be it is our (as users of k8s) desire to do in-place upgrades without downtime.. Sometimes a clean install of fresh cluster and switch over is so much simpler if we can just take a few minutes downtime to cut over. Complex requirements (sometimes) end up with complex solutions.

wg0 3y ago
It is simply not feasible if you have stateful workloads for some reason. Or at least isn't as straight forward. Booting up a cluster is fine but the real struggle starts from that point onwards.
xeromal 3y ago
I've never used K8s in my life but if you have to deal with downtime, I'd question the use of K8s when I can do it the old and busted way.
- harikb 3y ago
  I am not sure what "old and busted way" you are referring to, but k8s does have value, particularly when we are not upgrading k8s itself It would be good to think of the occasional downtime during a significant upgrade (once in 6 months?) separate from the ability to automatically replace failed nodes, scale up and down, having all the k8s magic.
  - PH95VuimJjqBqy 3y ago
    I would imagine using a load balancer in front of multiple servers across several datacenters coupled with a stateless design that allows traffic to flow freely across either of them. there are other variations of this, but this seems the most obvious. blue/green deploys and the like.
llama052 3y ago
I think it's definitely overlooked. You should always have the ability to spin off a new cluster and run your workloads on that. At a minimum as a disaster recovery scenario.

moondev 3y ago

Check out cluster-api. Complete zero downtime cluster upgrades is as simple as bumping the version and machine image name in the cluster manifest - or ClusterClass if you run classy clusters

markstos 3y ago
And why is that expected to be more reliable than Google’s service?

Freedom2 3y ago

While I see your point and agree, there's also the view that the people who invented the tech are simply different than the kind of people who Google hires nowadays - pure algo focused LeetCoders with zero-to-little operational experience and little interest in even learning how to run and maintain software.

eitally 3y ago

It used to be straightforward when Google Cloud just offered everyone the same version of GKE essentially everywhere. It was one of not-too-many services that are vanilla and identical in pretty much every region. But, newer offerings have fractured things a bit (sovereign cloud, Google Distributed Cloud, FedRAMP, Tencent, etc) and I can see why it would become problematic to adequately test and keep everything in sync while also not breaking interfaces and/or downstream services. Also, Google used to allow customers essentially 0 say in when upgrades to services were applied. Now, as part of many vendor security audit processes, it's a hard stop requirement that customers are able to test new revs before applying to prod ... and GKE, like GCE, is one of those critical services.

0xbadcafebee 3y ago

The design of K8s is ridiculous. The fact that you're forced to upgrade every 9 months is even more absurd. It's "opinionated", but the "opinions" are those of one of the largest companies in the world, who hire the most engineers in the world, and everyone else is supposed to operate like that now. K8s is a great example of a "tech jobs program": technology that is absurdly complicated to the point that you have to pay someone lots of money to make it keep working, but you can't afford not to pay for this, because everyone has decided to be stuck on it and now there's no alternative (and probably never will be)

wg0 3y ago
The biggest foot gun and most complicated part IMHO is networking. It is a blackbox with many competing implementations and each implementation even differing in implementation details within it's own versions. In theory, it seems reasonable to say that K8s networking is an implemention detail which you shouldn't bother about but in practice, it's going to bite you because of some misconfigured component somewhere someday and you would have no idea what to make of thousands of iptable rules or BGP routes or even least inspectable eBPF mini programs. Paid support, consulting.
- Too 3y ago
  This. Most application developers don’t give a rats ass about how 2 nodes are connected as long as the pods can speak to each other. Throw these guys in a devops team and give them the task to setup a 2 node k8s cluster and guess how much consideration they will pay to networking. Correct, zero. Except you can’t, because you have to choose a CNI plugin or choose a distribution that has made an opinionated choice for you. To lower the barrier of entry. A default installation must become a lot more opinionated with much less options and more sane and safe default.
sangnoir 3y ago
> K8s is a great example of a "tech jobs program" No, it is not. K8s is an example of how it's impossible to be a jack of all trades and retain simplicity. If a tool seeks to be all things to everyone, then there's no sweeping the inherent complexity under the carpet. Full-coverage solutions to complex domains are themselves complex by necessity - see SAP, Salesforce, Oracle, Windows backwards compatibility, or try writing a parser/stemmer for a spoken language that evolved naturally :)