8 ms·

Solving Common Problems with Kubernetes

174 points by sharjeelsayed 4y ago

cloudyuser1480 4y ago
- amrishk1480 4y ago
  - jonas21 4y ago
    Uh... so both of you created accounts ending in "1480" just now, just to post these comments? What gives?
    - douglas1480 4y ago
      - whatsawoman1480 4y ago
    - sean_flanigan 4y ago
      Perhaps it’s a case of numerative determinism!
iostream25 4y ago
kmoser 4y ago
The biggest problem I see is that the headline is ambiguous: it can be interpreted as "how to use Kubernetes to solve common problems," or alternately "how to solve problems commonly encountered in Kubernetes".
- Pxtl 4y ago
  Or even "this is my article on how the Kubernetes team should fix common problems for users"
- adamch 4y ago
  It's both!
- tannhaeuser 4y ago
  It's firmly in the realm of incidental complexity, viz. solving problems you wouldn't have in the first place if dynamic shared object loading, ld.so, and glibc would do their job or pointless Linux distro diversity wouldn't exist and/or static linking be used instead. And Python's inexcusable package management story, and even more inexcusable decision to base distro package management systems (yum, dnf) on Python of all languages.
deleted 4y ago
[deleted]
user3939382 4y ago
I know AWS ECS/ECR/EC2/ALBs etc well but jack about k8s. Is it kind of the same thing?
- bsuvc 4y ago
  Sort of. Each of those have their own k8s implementation. It's a standard way to define and run container workloads (jobs, services, etc) and related resources like storage, configs, secrets, etc.
  - nitwit005 4y ago
    While, yes, they do support k8s, they have independent solutions to many of the same problems.
- cheriot 4y ago
  Yes, although a slightly higher abstraction layer that can be implemented by those things. So one can theoretically describe a complex environment in a way that can be run on prem or by a variety of cloud vendors. (don't @ me just to say it's more complicated than that)
- adamch 4y ago
  Funny, I know a decent amount about k8s these days, but I've never really used AWS. I think there's a lot of systems that try to solve the same problems: "define these services, keep them running, here's how to check if they're down, here's how to restart them, keep resource usage to this level." Systemd, kubernetes, and individual cloud providers all have their own way to describe these. One of the upsides of k8s is that it gives you a general, platform-agnostic way to describe this, on AWS or GCP or self-hosted or whatever. The downside is that it's more flexible and therefore more abstract than any particular provider. The upside is that you only have to learn one set of concepts (in theory).
- dijksterhuis 4y ago
  Closet one from your list is ECS. AWS ECS is basically a docker swarm cluster customised by AWS to include auto-scaling of service containers/nodes and generally integrating better with other AWS services. k8s is that orchestration layer, but on anabolic steroids. Plus a standardised interface on any cloud platform you might need to run on. With bigger muscles comes more complexity, more maintenance work etc. (was designed for orchestrating millions of containers, so way more complexity) which is why everyone always moans about it. (including me) It’s a very cool and very useful piece of tech when yielded correctly. Just don’t expect it to work like a “kitchen sink”.
cloudking 4y ago
For a startup, I'm still trying to figure out why you would use k8s instead of Google App Engine or other managed PaaS. You never worry about the problems listed in this article. You focus on writing code, deploying it with minimal config and let Google handle all these problems. Can anyone explain?
- yrgulation 4y ago
  It’s fashionable.
- chrisbolt 4y ago
  For a startup that can do everything from scratch on one cloud provider, there aren’t many benefits. Those may come later once lock-in and complexity become a problem.
- hadlock 4y ago
  You don't need to re-write the whole CI/CD stack, you don't need to re-engineer the app as a 12 factor app if/when the thing takes off and you need a real solution. If your lead engineer gets hit by a bus tomorrow the whole deployment is in a helm chart somewhere and you can hire generic kubernetes tech #158326 to take over that job at any time. Being able to engineer the app correctly the first time around, plus guaranteed business continuity are pretty big selling points. I would not invest in any company that did not have kubernetes, or a similar orchestration platform underpinning their product. Eventually that engineer will leave and you do not want to be scrambling for their replacement in the middle of the big growth push.
- FridgeSeal 4y ago
  Off the top of my head: * you’re not using GCP * you have workloads that don’t mesh well with common cloud PaaS patterns * you want to pick-and-choose your cloud dependencies * you want to use a commodity tech-stack so that hiring, tooling and support are far more readily available. All of this comes with the obvious caveat of: sure, if you aren’t having issues, or reaching the limits of whatever PaaS solution you’re currently using, then don’t swap unnecessarily, but that still means there’s a lot of useful features that K8s offers.
- 4y ago
c54 4y ago
> What problem is k8s even trying to solve? > Say you want to deploy two Python servers. One of them needs Python 3.4 and the other needs Python 3.5. Honestly hilarious. The core value prop example is if you want to run two minor version different programming languages on one machine. In order to do that, you get to deploy and configure hundreds to thousands of lines of yaml, learn about at least 20 different abstraction jargon terms, and continue to spend all your time supporting this mess of infrastructure forever. How many engineering teams adopt kubernetes because it's what everyone's doing, versus out of genuine well-specified need? I have no idea. I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast. We don't deploy to millions of users around the globe, and we're all on one python version (for library compatibility, amongst other things). Docker is an annoying curse more than it's a boon. How much of this complexity is because python virtualenvs are confusing? How much would be solved if instead of "containers" we deployed static binaries (that were a couple hundred mb's larger apiece because they contained all their dependencies statically linked in... but who's counting). Idk. Autoscheduling can be nice, but can also be a footgun. To boot, k8s does not have sane defaults, everything's a potential footgun. In 10 years we're going to have either paved over all of the complexity, and k8s will be as invisible to us then as linux (mostly) is today. Or, we'll have realized this is insanity, and have ditched k8s for some other thing that solves a similar niche. edit: I realize this is a salty post. I don't mean to make anyone feel bad if they love to think about and use k8s. I appreciate and benefit from tutorial articles like this just as much as the next dev. It's just the nature of the beast at this point, I think.
- heywoodlh 4y ago
  > I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast. This is such a great description (I also for my job work exclusively on a platform based on Kubernetes). I ran K8s at home successfully for a while using K3s (such a great project/tool) to become more proficient. After a few months I found that there were so many features I didn't need for a homelab that the complexity wasn't worth it. I feel like, at the moment, Kubernetes is awesome when you have two ingredients: 1. You go with a managed K8s offering (such as AWS EKS) 2. You have a team of engineers dedicated to the health of the K8s platform It's pretty cool that a somewhat small platform team can wield the scaleability of a company like Google, BUT they need to be very good with K8s. And many workloads probably don't need that much scaleability.
misto 4y ago
You just can't have an article about k8s here without the debate being about its usefulness. Nicely written article nonetheless.
- alex3305 4y ago
  Besides opinions; remember that every tool has its uses. You can always cut down a tree with a shotgun, but a saw would be more efficient. Kubernetes is a complex beast, obviously not suitable for every workload and environment. But IMHO it can add value to product(s) when applied right.
emilburzo 4y ago
Although most of the articles focus on the benefits of Kubernetes for work, where you usually have more options (e.g. AWS Fargate), I've found it extremely helpful in the case where you want to self-host stuff personally/"at home", but you also want above average resilience so you use multiple nodes (e.g. 3 raspberry pi). You can probably DIY something or always run all the things on all 3 nodes to achieve the same effect, but the first time I saw a node going down and Kubernetes automatically deployed the affected pods to a healthy node, it felt like having AWS on-prem. Also, doing things in a standard way each time means it's really easy to bootstrap new things, not always reinventing the wheel. Especially cron jobs, I always lost track of those before. Now it's really easy to get an overview. Secrets as well, having it built-in just saves so much "how should I do this again". Plus base OS upgrades are now really uneventful, since besides k3s (single-binary Kubernetes distribution), there's nothing else installed on the host. The complexity is of course higher, you need a load balancer to point to all your cluster IPs since we don't get one "for free" from the cloud, shared storage is still something I'm looking into what's the best way to do (NAS with NFS, minio, ...), you most likely need a private registry, and probably others I'm missing. Nonetheless, since migrating from manually run docker images, I find it a lot more comfortable and even with less downtime (taking down a single node is a non-event now versus "everything is down").
tannhaeuser 4y ago
Is there a variant of "How many [k8s engineers] does it take to change a light bulb" yet?
- tbrownaw 4y ago
  Don't. It's better to just rebuild the entire house instead.
pjmlp 4y ago
For most problems, making use of kubernetes follows the same rule of using regular expressions.
mkl95 4y ago
I still believe Kubernetes is one of the best tools out there to build scalable systems. But I wish it was more opinionated and less leaky. Every project I build feels like a Frankenetes made of many subsystems provided by different vendors that struggle to talk to each other. The day Kubernetes provides curated bundles for common architectures that I can spin up with a few commands, I will be in love with it. For now I have a love/hate relationship with it.
- zoomzoom 4y ago
  Totally agree that k8s is great, and that it needs deeper coupling to specific common app patterns to be more useful to devs (as opposed to platform teams). IMO a new layer of abstraction above k8s is the right approach, for a variety of reasons. We’re trying to build it at withcoherence.com, please check out our early MVP!
- mathverse 4y ago
  My rule of thumb is to go with the project that has the biggest community/support out there UNLESS the other project provides a business critical feature I need. And even then it feels like we can just wait a bit until that feature gets implemented in the one with the biggest community/support.
webmonkeyuk 4y ago
What's the solution? k8s! What's the problem? Anything!
fock 4y ago
> Setting up an IngressController is probably the job of a specialized Platform team who maintain your k8s cluster, or by the cloud platform you're using. What? I thought this was a solution solving problems, not making ones which you should not solve...
- ckdarby 4y ago
  This is making the distinction of ownership. The team whose main objective is driving customer value only truly cares about the Ingress resource such as defining their routes to expose their work to the customer. Unless you want every team to take the burden of logging, tracing, auth, etc you centralize around a global edge network which is where one would define a ingress controller resource.

174 points by sharjeelsayed 4y ago

cloudyuser1480 4y ago

amrishk1480 4y ago
- jonas21 4y ago
  Uh... so both of you created accounts ending in "1480" just now, just to post these comments? What gives?
  - douglas1480 4y ago
    - whatsawoman1480 4y ago
  - sean_flanigan 4y ago
    Perhaps it’s a case of numerative determinism!

iostream25 4y ago

kmoser 4y ago

The biggest problem I see is that the headline is ambiguous: it can be interpreted as "how to use Kubernetes to solve common problems," or alternately "how to solve problems commonly encountered in Kubernetes".

Pxtl 4y ago
Or even "this is my article on how the Kubernetes team should fix common problems for users"
adamch 4y ago
It's both!
tannhaeuser 4y ago
It's firmly in the realm of incidental complexity, viz. solving problems you wouldn't have in the first place if dynamic shared object loading, ld.so, and glibc would do their job or pointless Linux distro diversity wouldn't exist and/or static linking be used instead. And Python's inexcusable package management story, and even more inexcusable decision to base distro package management systems (yum, dnf) on Python of all languages.

deleted 4y ago

[deleted]

user3939382 4y ago

I know AWS ECS/ECR/EC2/ALBs etc well but jack about k8s. Is it kind of the same thing?

bsuvc 4y ago
Sort of. Each of those have their own k8s implementation. It's a standard way to define and run container workloads (jobs, services, etc) and related resources like storage, configs, secrets, etc.
- nitwit005 4y ago
  While, yes, they do support k8s, they have independent solutions to many of the same problems.
cheriot 4y ago
Yes, although a slightly higher abstraction layer that can be implemented by those things. So one can theoretically describe a complex environment in a way that can be run on prem or by a variety of cloud vendors. (don't @ me just to say it's more complicated than that)
adamch 4y ago
Funny, I know a decent amount about k8s these days, but I've never really used AWS. I think there's a lot of systems that try to solve the same problems: "define these services, keep them running, here's how to check if they're down, here's how to restart them, keep resource usage to this level." Systemd, kubernetes, and individual cloud providers all have their own way to describe these. One of the upsides of k8s is that it gives you a general, platform-agnostic way to describe this, on AWS or GCP or self-hosted or whatever. The downside is that it's more flexible and therefore more abstract than any particular provider. The upside is that you only have to learn one set of concepts (in theory).
dijksterhuis 4y ago
Closet one from your list is ECS. AWS ECS is basically a docker swarm cluster customised by AWS to include auto-scaling of service containers/nodes and generally integrating better with other AWS services. k8s is that orchestration layer, but on anabolic steroids. Plus a standardised interface on any cloud platform you might need to run on. With bigger muscles comes more complexity, more maintenance work etc. (was designed for orchestrating millions of containers, so way more complexity) which is why everyone always moans about it. (including me) It’s a very cool and very useful piece of tech when yielded correctly. Just don’t expect it to work like a “kitchen sink”.

cloudking 4y ago

For a startup, I'm still trying to figure out why you would use k8s instead of Google App Engine or other managed PaaS. You never worry about the problems listed in this article. You focus on writing code, deploying it with minimal config and let Google handle all these problems. Can anyone explain?

yrgulation 4y ago
It’s fashionable.
chrisbolt 4y ago
For a startup that can do everything from scratch on one cloud provider, there aren’t many benefits. Those may come later once lock-in and complexity become a problem.
hadlock 4y ago
You don't need to re-write the whole CI/CD stack, you don't need to re-engineer the app as a 12 factor app if/when the thing takes off and you need a real solution. If your lead engineer gets hit by a bus tomorrow the whole deployment is in a helm chart somewhere and you can hire generic kubernetes tech #158326 to take over that job at any time. Being able to engineer the app correctly the first time around, plus guaranteed business continuity are pretty big selling points. I would not invest in any company that did not have kubernetes, or a similar orchestration platform underpinning their product. Eventually that engineer will leave and you do not want to be scrambling for their replacement in the middle of the big growth push.
FridgeSeal 4y ago
Off the top of my head: * you’re not using GCP * you have workloads that don’t mesh well with common cloud PaaS patterns * you want to pick-and-choose your cloud dependencies * you want to use a commodity tech-stack so that hiring, tooling and support are far more readily available. All of this comes with the obvious caveat of: sure, if you aren’t having issues, or reaching the limits of whatever PaaS solution you’re currently using, then don’t swap unnecessarily, but that still means there’s a lot of useful features that K8s offers.
4y ago

c54 4y ago

> What problem is k8s even trying to solve? > Say you want to deploy two Python servers. One of them needs Python 3.4 and the other needs Python 3.5. Honestly hilarious. The core value prop example is if you want to run two minor version different programming languages on one machine. In order to do that, you get to deploy and configure hundreds to thousands of lines of yaml, learn about at least 20 different abstraction jargon terms, and continue to spend all your time supporting this mess of infrastructure forever. How many engineering teams adopt kubernetes because it's what everyone's doing, versus out of genuine well-specified need? I have no idea. I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast. We don't deploy to millions of users around the globe, and we're all on one python version (for library compatibility, amongst other things). Docker is an annoying curse more than it's a boon. How much of this complexity is because python virtualenvs are confusing? How much would be solved if instead of "containers" we deployed static binaries (that were a couple hundred mb's larger apiece because they contained all their dependencies statically linked in... but who's counting). Idk. Autoscheduling can be nice, but can also be a footgun. To boot, k8s does not have sane defaults, everything's a potential footgun. In 10 years we're going to have either paved over all of the complexity, and k8s will be as invisible to us then as linux (mostly) is today. Or, we'll have realized this is insanity, and have ditched k8s for some other thing that solves a similar niche. edit: I realize this is a salty post. I don't mean to make anyone feel bad if they love to think about and use k8s. I appreciate and benefit from tutorial articles like this just as much as the next dev. It's just the nature of the beast at this point, I think.

heywoodlh 4y ago
> I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast. This is such a great description (I also for my job work exclusively on a platform based on Kubernetes). I ran K8s at home successfully for a while using K3s (such a great project/tool) to become more proficient. After a few months I found that there were so many features I didn't need for a homelab that the complexity wasn't worth it. I feel like, at the moment, Kubernetes is awesome when you have two ingredients: 1. You go with a managed K8s offering (such as AWS EKS) 2. You have a team of engineers dedicated to the health of the K8s platform It's pretty cool that a somewhat small platform team can wield the scaleability of a company like Google, BUT they need to be very good with K8s. And many workloads probably don't need that much scaleability.

misto 4y ago

You just can't have an article about k8s here without the debate being about its usefulness. Nicely written article nonetheless.

alex3305 4y ago
Besides opinions; remember that every tool has its uses. You can always cut down a tree with a shotgun, but a saw would be more efficient. Kubernetes is a complex beast, obviously not suitable for every workload and environment. But IMHO it can add value to product(s) when applied right.

emilburzo 4y ago

Although most of the articles focus on the benefits of Kubernetes for work, where you usually have more options (e.g. AWS Fargate), I've found it extremely helpful in the case where you want to self-host stuff personally/"at home", but you also want above average resilience so you use multiple nodes (e.g. 3 raspberry pi). You can probably DIY something or always run all the things on all 3 nodes to achieve the same effect, but the first time I saw a node going down and Kubernetes automatically deployed the affected pods to a healthy node, it felt like having AWS on-prem. Also, doing things in a standard way each time means it's really easy to bootstrap new things, not always reinventing the wheel. Especially cron jobs, I always lost track of those before. Now it's really easy to get an overview. Secrets as well, having it built-in just saves so much "how should I do this again". Plus base OS upgrades are now really uneventful, since besides k3s (single-binary Kubernetes distribution), there's nothing else installed on the host. The complexity is of course higher, you need a load balancer to point to all your cluster IPs since we don't get one "for free" from the cloud, shared storage is still something I'm looking into what's the best way to do (NAS with NFS, minio, ...), you most likely need a private registry, and probably others I'm missing. Nonetheless, since migrating from manually run docker images, I find it a lot more comfortable and even with less downtime (taking down a single node is a non-event now versus "everything is down").

tannhaeuser 4y ago

Is there a variant of "How many [k8s engineers] does it take to change a light bulb" yet?

tbrownaw 4y ago
Don't. It's better to just rebuild the entire house instead.

pjmlp 4y ago

For most problems, making use of kubernetes follows the same rule of using regular expressions.

mkl95 4y ago

I still believe Kubernetes is one of the best tools out there to build scalable systems. But I wish it was more opinionated and less leaky. Every project I build feels like a Frankenetes made of many subsystems provided by different vendors that struggle to talk to each other. The day Kubernetes provides curated bundles for common architectures that I can spin up with a few commands, I will be in love with it. For now I have a love/hate relationship with it.

zoomzoom 4y ago
Totally agree that k8s is great, and that it needs deeper coupling to specific common app patterns to be more useful to devs (as opposed to platform teams). IMO a new layer of abstraction above k8s is the right approach, for a variety of reasons. We’re trying to build it at withcoherence.com, please check out our early MVP!
mathverse 4y ago
My rule of thumb is to go with the project that has the biggest community/support out there UNLESS the other project provides a business critical feature I need. And even then it feels like we can just wait a bit until that feature gets implemented in the one with the biggest community/support.

webmonkeyuk 4y ago

What's the solution? k8s! What's the problem? Anything!

fock 4y ago

> Setting up an IngressController is probably the job of a specialized Platform team who maintain your k8s cluster, or by the cloud platform you're using. What? I thought this was a solution solving problems, not making ones which you should not solve...

ckdarby 4y ago
This is making the distinction of ownership. The team whose main objective is driving customer value only truly cares about the Ingress resource such as defining their routes to expose their work to the customer. Unless you want every team to take the burden of logging, tracing, auth, etc you centralize around a global edge network which is where one would define a ingress controller resource.