5 ms·

I made the mistake of falling for the k8s hype a few years back for running all of my indie hacker businesses. Big mistake. Overnight, the cluster config files

I made the mistake of falling for the k8s hype a few years back for running all of my indie hacker businesses. Big mistake. Overnight, the cluster config files I used were no longer supported by the k8s version DigitalOcean auto upgraded my cluster to and _boom_. Every single business was offline. Made the switch to some simple bash scripts for bootstrapping/monitoring/scaling and systemd for starting/restarting apps (nodejs). I'll never look back.

eddd-ddde 2y ago
So either digital ocean auto updates breaking versions. Or k8s doesn't do versioning correctly. Both very bad. Which was it?
- poincaredisk 2y ago
  I assume the first one, but it's more complicated. K8s used to have a lot of features (included very important ones) in the "beta" namespace. There are no stability guarantees there, but everyone used them anyway. Over time they graduated to the "stable" namespace, and after some transitory period they were removed from the beta namespace. This broke old deployments, when admins ignored warnings for two or three major releases.
  - dmurray 2y ago
    It's an odd choice to break backwards compatibility by removing them from the beta namespace. Why not keep them available in both indefinitely?
    - pcthrowaway 2y ago
      Probably because the devs understandably can't account for every possible way people might be using it when shipping new features. But in my experience this means k8s is a bag of fiddly bits that requires some serious ops investments to be reliable for anything serious.
    - p_l 2y ago
      With one exception that was rather big change to some low-level stuff, the "remove beta tags" was done with about a year or more of runway for people to upgrade. And ultimately, it wasn't hard to upgrade, even if you deal with auto-upgrading cluster and forgot about it, because "live" deployments got auto-upgraded - you do need to update your deployment script/whatever though.
  - psini 2y ago
    Just want to mention that two or three major releases sounds very bad, but Kubernetes had the insane release cadence of 4(!) major versions every year.
- rglover 2y ago
  Technically both, but more so the former. I had a heck of a time finding accurate docs on the correct apiVersion to use for things like my ingress and service files (they had a nasty habit of doing beta versions and changing config patterns w/ little backwards compatibility). This was a few years back when your options were a lot of Googling, SO, etc, so the info I found was mixed/spotty. As a solo founder, I found what worked at the time and assumed (foolishly, in retrospect) that it would just continue to work as my needs were modest.
poincaredisk 2y ago
I use k8s for the last uhh 5 years and this never happened to me. In my case, because I self-host my cluster, do no unexpected upgrades. But I agree that maintaining k8s cluster takes some work.
- theptip 2y ago
  In the 2015-2019 period there were quite a few API improvements involving deprecating old APIs, it’s much more stable/boring now. (Eg TPR -> CRD was the big one for many cluster plugins)
llama052 2y ago
So you had auto update enabled on your cluster and didn’t keep your apiversions up to date? Sounds like user error.
- rvense 2y ago
  One of my main criteria for evaluating a platform would be how easy it is to make user errors.
- psini 2y ago
  To be honest the API versions have been a lot more stable recently but back in ~2019 when I first used Kube in production, basic APIs were getting deprecated left and right, 4 times a year; in the end yes the problems are "on you" but it so easy to miss and the results so disastrous for a platform whose selling points are toughness resilience and self-healing
w0m 2y ago
yeouch. sorry man. I've been running in AKS for 3-4 years now and never had an auto-upgrade come in I wasn't expecting. I have been ontop of alerts and security bulletins though, may have kept me ahead of the curve.
- willvarfar 2y ago
  I was once on a nice family holiday and broke my resolve and did a 'quick' check of my email and found a nastygram billing reminder from a provider. On the one hand I was super-lucky I checked my mail when I did, and on the other I didn't get he holiday I needed and was lucky to not spill over and impact my family's happiness around me.
nine_k 2y ago
How does it compare to a simpler but not hand-crafted solution, such as dokku?
- rglover 2y ago
  No Docker for starters. I played with Dokku a long time ago and remember it being decent at that time, but still too confusing for my skillset. Now, I just build my app to an encrypted tarball, upload it to a secure bucket, and then create a short-lived signed URL for instances to curl the code from. From there, I just install deps on the machine and start up the app with systemd. IMO, Docker is overkill for 99% of projects, perhaps all. One of those great ideas, poorly executed (and considering the complexity, I understand why).
akvadrako 2y ago
EKS has a tab in the dashboard that warns about all the deprecated configs in your cluster, making it pretty foolproof to avoid this by checking every couple years.
- hhh 2y ago
  Yes, and there are many open source tools that you can point at clusters to do the same. We use Kubent (Kube No Troubles) to do the same.
minkles 2y ago
The first live k8s cluster upgrade anyone has to do is usually when they think "what the fuck did I get myself in to?" It's only good for very large scale stuff. And then a lot of the time that is usually well over provisioned and could be done considerably cheaper using almost any other methodology. The only good part of Kubernetes I have found in the last 4 years of running it in production is that you can deploy any old limping crap to it and it does its best to keep it alive which means you can spend more time writing YAML and upgrading it every 2 minutes.
_xiaz 2y ago
> simple bash scripts for bootstrapping/monitoring/scaling Damn, that's the dream right there
cedws 2y ago
Weird how defensive people get about K8S when you say stuff like this. It’s like they’re desperately trying to convince you that you really do need all that complexity.
- 0perator 2y ago
  Most do not, but they still want all the toys that developers are building for “the cloud”.
- rollcat 2y ago
  I believe there's still a lot of potential for building niche / "human-scale" services/businesses, that don't inherently require the scalability of the cloud or complexity of k8s. Scaling vertically is always easier, modern server hardware has insane perf ceiling. The overall reduction in complexity is a breath of fresh air. My occasional moral dilemma is idle power usage of overprovisioned resources, but we've found some interesting things to throw at idle hardware to ease our conscience about it.
  - sswezey 2y ago
    I particularly like this moniker for such human-scale, "digital gardening"-type software: https://hobbit.software/ https://hobbit.software/
- alex_lav 2y ago
  I think it's two types of defensiveness. 1. Shovel salesman insisting all "real" gold miners use their shovels 2. Those that have already acquired shovels not wanting their purchase to be mocked/have been made in vain. Neither are grounded in reality. Why people believe their tiny applications require the same tech that Google invented to help manage their (massive) scale is beyond me.
tucnak 2y ago
So what is the alternative? Nomad?
mythz 2y ago
We're also ignoring Kubernetes and are just using GitHub Actions, Docker Compose and SSH for our CI Deployments [1]. After a one-time setup on the Deployment Server, we can deploy new Apps with just a few GitHub Action Secrets, which then gets redeployed on every commit, including running any DB Migrations. We're currently using this to deploy and run over 50 .NET Apps across 3 Hetzner VMs. [1] https://servicestack.net/posts/kubernetes_not_required https://servicestack.net/posts/kubernetes_not_required
- oldprogrammer2 2y ago
  The amount of complexity people are introducing into their infrastructure is insane. At the end of the day, we're still just building the same CRUD web apps we were building 20 years ago. We have 50x the computation power, much faster disk, much more RAM, and much faster internet. A pair of load-balanced web servers and a managed database, with Cloudflare out front, will get you really, really far.

eddd-ddde 2y ago

So either digital ocean auto updates breaking versions. Or k8s doesn't do versioning correctly. Both very bad. Which was it?

poincaredisk 2y ago
I assume the first one, but it's more complicated. K8s used to have a lot of features (included very important ones) in the "beta" namespace. There are no stability guarantees there, but everyone used them anyway. Over time they graduated to the "stable" namespace, and after some transitory period they were removed from the beta namespace. This broke old deployments, when admins ignored warnings for two or three major releases.
- dmurray 2y ago
  It's an odd choice to break backwards compatibility by removing them from the beta namespace. Why not keep them available in both indefinitely?
  - pcthrowaway 2y ago
    Probably because the devs understandably can't account for every possible way people might be using it when shipping new features. But in my experience this means k8s is a bag of fiddly bits that requires some serious ops investments to be reliable for anything serious.
  - p_l 2y ago
    With one exception that was rather big change to some low-level stuff, the "remove beta tags" was done with about a year or more of runway for people to upgrade. And ultimately, it wasn't hard to upgrade, even if you deal with auto-upgrading cluster and forgot about it, because "live" deployments got auto-upgraded - you do need to update your deployment script/whatever though.
- psini 2y ago
  Just want to mention that two or three major releases sounds very bad, but Kubernetes had the insane release cadence of 4(!) major versions every year.
rglover 2y ago
Technically both, but more so the former. I had a heck of a time finding accurate docs on the correct apiVersion to use for things like my ingress and service files (they had a nasty habit of doing beta versions and changing config patterns w/ little backwards compatibility). This was a few years back when your options were a lot of Googling, SO, etc, so the info I found was mixed/spotty. As a solo founder, I found what worked at the time and assumed (foolishly, in retrospect) that it would just continue to work as my needs were modest.

poincaredisk 2y ago

I use k8s for the last uhh 5 years and this never happened to me. In my case, because I self-host my cluster, do no unexpected upgrades. But I agree that maintaining k8s cluster takes some work.

theptip 2y ago
In the 2015-2019 period there were quite a few API improvements involving deprecating old APIs, it’s much more stable/boring now. (Eg TPR -> CRD was the big one for many cluster plugins)

llama052 2y ago

So you had auto update enabled on your cluster and didn’t keep your apiversions up to date? Sounds like user error.

rvense 2y ago
One of my main criteria for evaluating a platform would be how easy it is to make user errors.
psini 2y ago
To be honest the API versions have been a lot more stable recently but back in ~2019 when I first used Kube in production, basic APIs were getting deprecated left and right, 4 times a year; in the end yes the problems are "on you" but it so easy to miss and the results so disastrous for a platform whose selling points are toughness resilience and self-healing

w0m 2y ago

yeouch. sorry man. I've been running in AKS for 3-4 years now and never had an auto-upgrade come in I wasn't expecting. I have been ontop of alerts and security bulletins though, may have kept me ahead of the curve.

willvarfar 2y ago
I was once on a nice family holiday and broke my resolve and did a 'quick' check of my email and found a nastygram billing reminder from a provider. On the one hand I was super-lucky I checked my mail when I did, and on the other I didn't get he holiday I needed and was lucky to not spill over and impact my family's happiness around me.

nine_k 2y ago

How does it compare to a simpler but not hand-crafted solution, such as dokku?

rglover 2y ago
No Docker for starters. I played with Dokku a long time ago and remember it being decent at that time, but still too confusing for my skillset. Now, I just build my app to an encrypted tarball, upload it to a secure bucket, and then create a short-lived signed URL for instances to curl the code from. From there, I just install deps on the machine and start up the app with systemd. IMO, Docker is overkill for 99% of projects, perhaps all. One of those great ideas, poorly executed (and considering the complexity, I understand why).

akvadrako 2y ago

EKS has a tab in the dashboard that warns about all the deprecated configs in your cluster, making it pretty foolproof to avoid this by checking every couple years.

hhh 2y ago
Yes, and there are many open source tools that you can point at clusters to do the same. We use Kubent (Kube No Troubles) to do the same.

minkles 2y ago

The first live k8s cluster upgrade anyone has to do is usually when they think "what the fuck did I get myself in to?" It's only good for very large scale stuff. And then a lot of the time that is usually well over provisioned and could be done considerably cheaper using almost any other methodology. The only good part of Kubernetes I have found in the last 4 years of running it in production is that you can deploy any old limping crap to it and it does its best to keep it alive which means you can spend more time writing YAML and upgrading it every 2 minutes.

_xiaz 2y ago

> simple bash scripts for bootstrapping/monitoring/scaling Damn, that's the dream right there

cedws 2y ago

Weird how defensive people get about K8S when you say stuff like this. It’s like they’re desperately trying to convince you that you really do need all that complexity.

0perator 2y ago
Most do not, but they still want all the toys that developers are building for “the cloud”.
rollcat 2y ago
I believe there's still a lot of potential for building niche / "human-scale" services/businesses, that don't inherently require the scalability of the cloud or complexity of k8s. Scaling vertically is always easier, modern server hardware has insane perf ceiling. The overall reduction in complexity is a breath of fresh air. My occasional moral dilemma is idle power usage of overprovisioned resources, but we've found some interesting things to throw at idle hardware to ease our conscience about it.
- sswezey 2y ago
  I particularly like this moniker for such human-scale, "digital gardening"-type software: https://hobbit.software/ https://hobbit.software/
alex_lav 2y ago
I think it's two types of defensiveness. 1. Shovel salesman insisting all "real" gold miners use their shovels 2. Those that have already acquired shovels not wanting their purchase to be mocked/have been made in vain. Neither are grounded in reality. Why people believe their tiny applications require the same tech that Google invented to help manage their (massive) scale is beyond me.

tucnak 2y ago

So what is the alternative? Nomad?

mythz 2y ago

We're also ignoring Kubernetes and are just using GitHub Actions, Docker Compose and SSH for our CI Deployments [1]. After a one-time setup on the Deployment Server, we can deploy new Apps with just a few GitHub Action Secrets, which then gets redeployed on every commit, including running any DB Migrations. We're currently using this to deploy and run over 50 .NET Apps across 3 Hetzner VMs. [1] https://servicestack.net/posts/kubernetes_not_required https://servicestack.net/posts/kubernetes_not_required

oldprogrammer2 2y ago
The amount of complexity people are introducing into their infrastructure is insane. At the end of the day, we're still just building the same CRUD web apps we were building 20 years ago. We have 50x the computation power, much faster disk, much more RAM, and much faster internet. A pair of load-balanced web servers and a managed database, with Cloudflare out front, will get you really, really far.