3 ms·

You aren't a real K8s admin until your self-managed cluster crashes hard and you have to spend 3 days trying to recover/rebuild it. Just dealing with the certs

You aren't a real K8s admin until your self-managed cluster crashes hard and you have to spend 3 days trying to recover/rebuild it. Just dealing with the certs once they start expiring is a nightmare. To avoid chicken-and-egg, your critical services (Drone, Vault, Bind) need to live outside of K8s in something stupid simple, like an ASG or a hot/cold EC2 pair. I've mostly come to think of K8s as a development tool. It makes it quick and easy for devs to mock up a software architecture and run it anywhere, compared to trying to adopt a single cloud vendor's SaaS tools, and giving devs all the Cloud access needed to control it. Give them access to a semi-locked-down K8s cluster instead and they can build pretty much whatever they need without asking anyone for anything. For production, it's kind of crap, but usable. It doesn't have any of the operational intelligence you'd want a resilient production system to have, doesn't have real version control, isn't immutable, and makes it very hard to identify and fix problems. A production alternative to K8s should be much more stripped-down, like Fargate, with more useful operational features, and other aspects handled by external projects.

throwboatyface 3y ago
Honestly in this day and age rolling your own k8s cluster is negligent. I've worked at multiple companies using EKS, AKS, GKE, and we haven't had 10% of the issues I see people complaining about.
- jauntywundrkind 3y ago
  Once your team has upgrades down, everything is pretty rote. This submission (Urbit, lol) seemed particularly incompetent at managing cert rotation. The other capital lesson here? Have backups. The team couldnt restore a bunch of their services effectively, cause they didn't have the manifests. Sure, a managed provider may have less disruptions/avoid some fuckups, but the whole point of Kubernetes is Promise Theory, is Desired State Mamagememt. If you can re-state your asks, put the manifests back, most shit should just work again, easy as that. The team had seemingly no operational system so their whole cluster was a vast special pet. They fucked up. Don't do that.
  - nyolfen 3y ago
    this is actually a separate project from urbit, called urb-it https://urb-it.webflow.io/ https://urb-it.webflow.io/
  - anotherhue 3y ago
    Different Urbit.
- dilyevsky 3y ago
  I've picked my fair share of outages on managed k8s solutions. The difference there is once it's hosed, your fate is 100% in the hands of cloud support and well... good luck with that one. The cloud apologists in this thread will ofc try to shame you for not buying into their marketing
  - catchnear4321 3y ago
    if your fate is in the hands of one of the cloud gods, what right does anyone have to blame you for what transpires? mere mortals are not privy to all of the internal downstream impacts from that public-facing service outage. it would be like shouting into the void and expecting an answer, and, more, liking it. no, it is easier to recognize one’s place, pay the tithes, and enjoy one god’s blessings and curses alike. do not stray and attempt to please two, it will only end in misery. (three is right out.)
nvarsj 3y ago
It's kind of the modus operandi of Kubernetes since inception. The core model is okay, but ops was always a barely constructed afterthought. And the network stack (kube-proxy) was literally a summer of code project. I'm thinking a lot of that was by design - both Redhat and Google had incentives to get you onto their value-add to get an actual production ready system. It also created an entire cottage industry, although much of this has faded as everyone moved to purely managed solutions. Because anything else is absolutely insane.
- treflop 3y ago
  I’m not sure if it’s intentional. I don’t find the other container orchestrators that much better either. No one ever cares about making tooling in any software project. You’re always using something by a dead-ass random third-party. Microsoft is probably the company where I am actually using Microsoft-made tools to manage Microsoft-made products. And maybe Adobe back in the day.
vundercind 3y ago
In the bad old days of self-managing some servers with a few libvirt VMs and such, I’d have considered a 3-day outage such a shockingly bad outcome that I’d have totally reconsidered what I was doing. And k8s is supposed to make that situation better, but these multi-day outage stories are… common? Why are we adding all this complexity and cost if the result is consumer-PC-tower-in-a-closet-with-no-IAC uptime (or worse)?
- aliasxneo 3y ago
  I've been running Kubernetes in production for two years and have never experienced anything remotely close to this. The worst is a node dies every now and then and, on a rare occasion, a workload doesn't happily migrate. Of course, my experience is in no way authoritative, but referencing this type of incident as common is pretty foreign to me and may be mostly relegated to self-managed clusters.
  - JohnMakin 3y ago
    > may be mostly relegated to self-managed clusters. Foreign to me too, but not surprising people report issues as common. there are a lot of footguns in kubernetes that come from a lack of understanding. You can build a robust kubernetes cluster that hosts an application that’s nearly impossible to bring offline without an act of god, it just takes some know-how and a tiny bit of effort/experience.
  - deleted 3y ago
    [deleted]
  - goosejuice 3y ago
    GKE since 2017 here. Healthcare. I think we had one major outage that involved the cluster itself. It resolved itself and we never discovered what caused it. That was in the early days, so I recall very little. Now I'm using Fly.io. They both have their advantages. Folks tend to make kubernetes sound way more difficult than it is. It can be overkill but it can also solve so many challenges out of the box. At least when it's managed. It'll cost you though.
- 3y ago
brightball 3y ago
Personally, I’m a big fan for QA review sites. Deploy multiple low traffic full site clones to a cluster and spin them up and down as needed. Manual review, automated scans, etc. It’s great for that use case IMO. In production I always want dedicated resources though.
ikiris 3y ago
What's drone?
- dabber 3y ago
  https://www.drone.io/ https://www.drone.io/ > Automate Software Build and Testing Drone is a self-service Continuous Integration platform for busy development teams.
  - 0xbadcafebee 3y ago
    Simplest possible CI tool that exists, as far as I'm aware. Gives you just barely everything you need, everything is stupid simple, and it just works. There's an OSS fork in development (https://woodpecker-ci.org/ https://woodpecker-ci.org/) but it's far behind in terms of features and stability.

throwboatyface 3y ago

Honestly in this day and age rolling your own k8s cluster is negligent. I've worked at multiple companies using EKS, AKS, GKE, and we haven't had 10% of the issues I see people complaining about.

jauntywundrkind 3y ago
Once your team has upgrades down, everything is pretty rote. This submission (Urbit, lol) seemed particularly incompetent at managing cert rotation. The other capital lesson here? Have backups. The team couldnt restore a bunch of their services effectively, cause they didn't have the manifests. Sure, a managed provider may have less disruptions/avoid some fuckups, but the whole point of Kubernetes is Promise Theory, is Desired State Mamagememt. If you can re-state your asks, put the manifests back, most shit should just work again, easy as that. The team had seemingly no operational system so their whole cluster was a vast special pet. They fucked up. Don't do that.
- nyolfen 3y ago
  this is actually a separate project from urbit, called urb-it https://urb-it.webflow.io/ https://urb-it.webflow.io/
- anotherhue 3y ago
  Different Urbit.
dilyevsky 3y ago
I've picked my fair share of outages on managed k8s solutions. The difference there is once it's hosed, your fate is 100% in the hands of cloud support and well... good luck with that one. The cloud apologists in this thread will ofc try to shame you for not buying into their marketing
- catchnear4321 3y ago
  if your fate is in the hands of one of the cloud gods, what right does anyone have to blame you for what transpires? mere mortals are not privy to all of the internal downstream impacts from that public-facing service outage. it would be like shouting into the void and expecting an answer, and, more, liking it. no, it is easier to recognize one’s place, pay the tithes, and enjoy one god’s blessings and curses alike. do not stray and attempt to please two, it will only end in misery. (three is right out.)

nvarsj 3y ago

It's kind of the modus operandi of Kubernetes since inception. The core model is okay, but ops was always a barely constructed afterthought. And the network stack (kube-proxy) was literally a summer of code project. I'm thinking a lot of that was by design - both Redhat and Google had incentives to get you onto their value-add to get an actual production ready system. It also created an entire cottage industry, although much of this has faded as everyone moved to purely managed solutions. Because anything else is absolutely insane.

treflop 3y ago
I’m not sure if it’s intentional. I don’t find the other container orchestrators that much better either. No one ever cares about making tooling in any software project. You’re always using something by a dead-ass random third-party. Microsoft is probably the company where I am actually using Microsoft-made tools to manage Microsoft-made products. And maybe Adobe back in the day.

vundercind 3y ago

In the bad old days of self-managing some servers with a few libvirt VMs and such, I’d have considered a 3-day outage such a shockingly bad outcome that I’d have totally reconsidered what I was doing. And k8s is supposed to make that situation better, but these multi-day outage stories are… common? Why are we adding all this complexity and cost if the result is consumer-PC-tower-in-a-closet-with-no-IAC uptime (or worse)?

aliasxneo 3y ago
I've been running Kubernetes in production for two years and have never experienced anything remotely close to this. The worst is a node dies every now and then and, on a rare occasion, a workload doesn't happily migrate. Of course, my experience is in no way authoritative, but referencing this type of incident as common is pretty foreign to me and may be mostly relegated to self-managed clusters.
- JohnMakin 3y ago
  > may be mostly relegated to self-managed clusters. Foreign to me too, but not surprising people report issues as common. there are a lot of footguns in kubernetes that come from a lack of understanding. You can build a robust kubernetes cluster that hosts an application that’s nearly impossible to bring offline without an act of god, it just takes some know-how and a tiny bit of effort/experience.
- deleted 3y ago
  [deleted]
- goosejuice 3y ago
  GKE since 2017 here. Healthcare. I think we had one major outage that involved the cluster itself. It resolved itself and we never discovered what caused it. That was in the early days, so I recall very little. Now I'm using Fly.io. They both have their advantages. Folks tend to make kubernetes sound way more difficult than it is. It can be overkill but it can also solve so many challenges out of the box. At least when it's managed. It'll cost you though.
3y ago

brightball 3y ago

Personally, I’m a big fan for QA review sites. Deploy multiple low traffic full site clones to a cluster and spin them up and down as needed. Manual review, automated scans, etc. It’s great for that use case IMO. In production I always want dedicated resources though.

ikiris 3y ago

What's drone?

dabber 3y ago
https://www.drone.io/ https://www.drone.io/ > Automate Software Build and Testing Drone is a self-service Continuous Integration platform for busy development teams.
- 0xbadcafebee 3y ago
  Simplest possible CI tool that exists, as far as I'm aware. Gives you just barely everything you need, everything is stupid simple, and it just works. There's an OSS fork in development (https://woodpecker-ci.org/ https://woodpecker-ci.org/) but it's far behind in terms of features and stability.