5 ms·

It was 01:30. About to go to bed. Next day would fly with my daughter to holidays. I checked the apps of a bunch of clients. None of them loaded. I was like wh

It was 01:30. About to go to bed. Next day would fly with my daughter to holidays. I checked the apps of a bunch of clients. None of them loaded. I was like what... I checked the server. Everything down. I'd been running Kubernetes on Digital Ocean. And Digital Ocean forced a Kubernetes update that was incompatitable with mine at night. Took me 8 hours to fix it. No sleep. Ended up moving it back to a good old VPS. And throw away K8s. Now to be fair, I had been getting warnings with deployments. But I was used to that, Kubernetes has 10 updates per week. I dont have time to update K8S or my helm files every week. So yeah it was my fault, but I was used to good old VPS hosting. There is an old php application I've build 5 years ago with Laravel, that never needs anything. I did some updates and patches, but it always just works. Im used to running node on apache or nginx, even though a bit less stable, still almost never crashes. Kubernetes there is always something. I guess there are reasons to choose it, but it's not stability. I ended up taking the plane, and my daughter was super kind and patient. But no more kubernetes for me.

alyandon 4y ago
My experience so far is that k8s itself is relatively stable. It's when you start using vendor specific addons/plugins to actually do stuff like provision PVs, modify Citrix LB settings, etc outside of k8s everything quickly becomes a burning tire fire. :-/
jhoelzel 4y ago
I feel your pain and i would lie if a k8s update never caught me off guard, but there are still policies you can set for the autoupdate of k3s or trigger them manually at will. have a look at k3s and maybe you will like kubernetes more again. There is no magic to it. Have a faulty node? spin up a new one. And if your hoster is kind enough, there will even be apis with cloudinit for you to do that. Im not trying to imply that you need a managed k3s with this post, but rather trying to show how easy kubernetes can be if you leave the big clouds and try not to overcomplicate things.
bithavoc 4y ago
DigitalOcean also cordons Nodes for no reason at random times. Currently moving to EKS exactly for this reason
- atmosx 4y ago
  Can you share more details please? How did you find out it was DO, did you open a support ticket?
  - bithavoc 4y ago
    well, I'm the only one with access to the cluster and I didn't do it, with Lens IDE I barely check the Nodes tab, I checked the Nodes because I noticed no app pods were scheduling in such node and I found it cordoned.
- taspeotis 4y ago
  In late 2021 we had EKS nodes shit the bed for no apparent reason. Max out CPU and couldn't SSH in. They'd get recreated eventually by EC2 (multiple hours) despite being tainted in k8s pretty quickly. We ran with N+2 nodes to deal with it. After dutifully updating EKS releases and AMIs it seems to have fixed itself last year. EKS seems to be fine generally, but it would be nice if they didn't release new AMIs with missing commits?? [1] and especially [2] [1] https://github.com/aws/containers-roadmap/issues/319 https://github.com/aws/containers-roadmap/issues/319 [2] https://github.com/awslabs/amazon-eks-ami/issues/278#issuecomment-498693960 https://github.com/awslabs/amazon-eks-ami/issues/278#issueco...
davidkuennen 4y ago
Sounds like Digital Ocean was the problem, not Kubernetes. Been running managed Kubernetes in GCP for years without any issues. They are bugging me to update for some time now, but I don't think they would force an update on me.
- figassis 4y ago
  Something is always the problem. With a VPS, you don’t usually have to worry about DO being a problem. You pay your bills, your server stays up. Maybe if you don’t update you might get hacked, but killed by provider update is not a failure mode of VPSs. I use k3s on bare metal. I’ve tried to run k8s so many times before and they all failed at some point in ways that didn’t inspire confidence. Mostly etcd became a mess, so this time I’m running k3s with a Postgres datastore that I can actually look at, reason about, backup and independently make HA, with a plus of not having to run arcane commands to manage the cluster that runs my cluster. I once tried nomad because it claimed to be simpler. I stopped when I realized I’d also have to use multiple clusters to run a cluster (nomad+consul). I use k3s because I build apps for multiple clients and I like the flexibility that it gives me in running servers like cattle. My point is that these orchestrators need to start decreasing the enormous amount of risk they add to any project.
- tinco 4y ago
  I hope you're right, I haven't had time to look at the same warning for 6 months now. There was a hard deadline on it in February and it came and went. The backwards incompatible Kubernetes api updates are a PITA. I know they said "beta", but if you literally have a version field in your manifest format, why ever deprecate a way of configuring something? They could just have operators translate the configs automatically on the fly.
  - Rantenki 4y ago
    >They could just have operators translate the configs automatically on the fly How do they test that across the infinite number of permutations of configurations and deployments of K8S in the field though? It'll work for some people on the happy path, but it's really hard to maintain over time. Worse, it'll break randomly at some point in the future that is hard to predict, instead of at some publicly announced point in time where the breaking change is deployed (how it happened this time).
b33j0r 4y ago
Yep. My problem with a managed cloud is that kubernetes drivers are vendor specific. I hit this landmine too when the ingress I was using got a breaking update (as far as I could tell). Still worked on my test cluster. It was a throw away site, so I didn’t even bother fixing it. And, I honestly can’t tell you what went wrong. Just deploying docker compose, often with ansible, is the most reliable for me at most scales.
- ilyt 4y ago
  Yeah, that's the annoying part. You get around 80% of parts that just work anywhere and rest is entirely dependant on what's underneath k8s or on what cloud it is running. When going in I assumed there are some kind of standard way to implement ingresses/loadbalancers but no, it's just different plugins each with different syntax and features.
amq 4y ago
In my experience, helm is the worst offender. Somehow, every second chart update has a breaking change preventing the update, defaults that don't mind wiping the persistent volume and a discontinued somewhat consistent 'stable' central repo make me seriously regret using helm charts for anything that is not ephemeral.
whalesalad 4y ago
Don’t throw the baby out with the bathwater.
woopwoop24 4y ago
there is a good talk from kelsey hightower on when it makes sense to ue k8s and the number was like 20 servers. i think with what you are describing i would not even think about using k8s.

alyandon 4y ago

My experience so far is that k8s itself is relatively stable. It's when you start using vendor specific addons/plugins to actually do stuff like provision PVs, modify Citrix LB settings, etc outside of k8s everything quickly becomes a burning tire fire. :-/

jhoelzel 4y ago

I feel your pain and i would lie if a k8s update never caught me off guard, but there are still policies you can set for the autoupdate of k3s or trigger them manually at will. have a look at k3s and maybe you will like kubernetes more again. There is no magic to it. Have a faulty node? spin up a new one. And if your hoster is kind enough, there will even be apis with cloudinit for you to do that. Im not trying to imply that you need a managed k3s with this post, but rather trying to show how easy kubernetes can be if you leave the big clouds and try not to overcomplicate things.

bithavoc 4y ago

DigitalOcean also cordons Nodes for no reason at random times. Currently moving to EKS exactly for this reason

atmosx 4y ago
Can you share more details please? How did you find out it was DO, did you open a support ticket?
- bithavoc 4y ago
  well, I'm the only one with access to the cluster and I didn't do it, with Lens IDE I barely check the Nodes tab, I checked the Nodes because I noticed no app pods were scheduling in such node and I found it cordoned.
taspeotis 4y ago
In late 2021 we had EKS nodes shit the bed for no apparent reason. Max out CPU and couldn't SSH in. They'd get recreated eventually by EC2 (multiple hours) despite being tainted in k8s pretty quickly. We ran with N+2 nodes to deal with it. After dutifully updating EKS releases and AMIs it seems to have fixed itself last year. EKS seems to be fine generally, but it would be nice if they didn't release new AMIs with missing commits?? [1] and especially [2] [1] https://github.com/aws/containers-roadmap/issues/319 https://github.com/aws/containers-roadmap/issues/319 [2] https://github.com/awslabs/amazon-eks-ami/issues/278#issuecomment-498693960 https://github.com/awslabs/amazon-eks-ami/issues/278#issueco...

davidkuennen 4y ago

Sounds like Digital Ocean was the problem, not Kubernetes. Been running managed Kubernetes in GCP for years without any issues. They are bugging me to update for some time now, but I don't think they would force an update on me.

figassis 4y ago
Something is always the problem. With a VPS, you don’t usually have to worry about DO being a problem. You pay your bills, your server stays up. Maybe if you don’t update you might get hacked, but killed by provider update is not a failure mode of VPSs. I use k3s on bare metal. I’ve tried to run k8s so many times before and they all failed at some point in ways that didn’t inspire confidence. Mostly etcd became a mess, so this time I’m running k3s with a Postgres datastore that I can actually look at, reason about, backup and independently make HA, with a plus of not having to run arcane commands to manage the cluster that runs my cluster. I once tried nomad because it claimed to be simpler. I stopped when I realized I’d also have to use multiple clusters to run a cluster (nomad+consul). I use k3s because I build apps for multiple clients and I like the flexibility that it gives me in running servers like cattle. My point is that these orchestrators need to start decreasing the enormous amount of risk they add to any project.
tinco 4y ago
I hope you're right, I haven't had time to look at the same warning for 6 months now. There was a hard deadline on it in February and it came and went. The backwards incompatible Kubernetes api updates are a PITA. I know they said "beta", but if you literally have a version field in your manifest format, why ever deprecate a way of configuring something? They could just have operators translate the configs automatically on the fly.
- Rantenki 4y ago
  >They could just have operators translate the configs automatically on the fly How do they test that across the infinite number of permutations of configurations and deployments of K8S in the field though? It'll work for some people on the happy path, but it's really hard to maintain over time. Worse, it'll break randomly at some point in the future that is hard to predict, instead of at some publicly announced point in time where the breaking change is deployed (how it happened this time).

b33j0r 4y ago

Yep. My problem with a managed cloud is that kubernetes drivers are vendor specific. I hit this landmine too when the ingress I was using got a breaking update (as far as I could tell). Still worked on my test cluster. It was a throw away site, so I didn’t even bother fixing it. And, I honestly can’t tell you what went wrong. Just deploying docker compose, often with ansible, is the most reliable for me at most scales.

ilyt 4y ago
Yeah, that's the annoying part. You get around 80% of parts that just work anywhere and rest is entirely dependant on what's underneath k8s or on what cloud it is running. When going in I assumed there are some kind of standard way to implement ingresses/loadbalancers but no, it's just different plugins each with different syntax and features.

amq 4y ago

In my experience, helm is the worst offender. Somehow, every second chart update has a breaking change preventing the update, defaults that don't mind wiping the persistent volume and a discontinued somewhat consistent 'stable' central repo make me seriously regret using helm charts for anything that is not ephemeral.

whalesalad 4y ago

Don’t throw the baby out with the bathwater.

woopwoop24 4y ago

there is a good talk from kelsey hightower on when it makes sense to ue k8s and the number was like 20 servers. i think with what you are describing i would not even think about using k8s.