7 ms·

Kubernetes Failure Stories

72 points by wseqyrku 4y ago

6LLvveMx2koXfwn 4y ago
Someone in our institute thought shifting the FTP service to single pod deployments with no auto-scaling was a good idea. They were wrong.
- wmichelin 4y ago
  Can they just... setup autoscaling? Hopefully the mistake wasn't too hard to correct.
  - 6LLvveMx2koXfwn 4y ago
    Maybe, but introducing several layers of abstraction was never going to be the solution if performance was the problem.
  - jasonjayr 4y ago
    With the port mapping that FTP entails? You'd need a k8s aware FTP service that creates load balanced ports on demand. Which sounds like it could get thorny and complicated compared to a pair of AWS EC2 instances over an S3 filesystem (Or use AWS's hosted FTP service?)
ok123456 4y ago
I look forward to the day that we never have to hear about Kubernetes again.
- arinlen 4y ago
  > look forward to the day that we never have to hear about Kubernetes again. How do you expect to deploy services in a cluster of multiple nodes communicating over a private software-defined network, then?
  - coding123 4y ago
    There ARE other ways, but mesos was a freaking mess. I actually liked swarm, but it lost. k8s also brings in pretty novel ideas that destroy proprietary IaaS, such as the nginx (or other) front end ingress routers that I haven't seen implemented meaningfully in a IaaS neutral way. Sure, you may have a few properties in your k8s yaml that change from AWS vs Google vs Azure vs other k8s hosts. That's not something that takes more than a debugging session to get right. That's HUUUGE progress.
    - arinlen 4y ago
      > I actually liked swarm, but it lost. "lost" is too strong of a word. To me Docker swarm mode still holds the upper hand in terms of deployment ease and simplicity. If anyone just wants to bolt together a couple of nodes without much concern regarding performance then Docker swarm mode is perhaps the best option out there at the moment. However, this is changing very fast. Ubuntu now ships with microk8s which just works and outperforms Docker swarm mode. Perhaps Docker swarm's worst enemy is how the project was managed and promoted. They had a tool to trivially setup a cluster of nodes with close to zero challenge, but somehow Kubernetes managed to be pushed as the only option.
  - ok123456 4y ago
    docker compose. Start it as a systemd service if you want.
    - majewsky 4y ago
      docker-compose only deploys on individual nodes though.
      - ok123456 4y ago
        That's shifting the goal posts. And, that covers 99.9% of cases anyway. If you want to deploy across multiple nodes run docker compose on multiple nodes and expose the nodes to each other over a vlan that is managed as part of your actual infrastructure where it has a chance of being actually documented and put into a configuration management system.
dvtrn 4y ago
I want someone to check me on a strongly held belief, I promise I have an open mind: There’s nothing wrong with experimenting with k8s in your org. Nothing wrong with k8s, it’s complex yes but that’s not inherently problematic. But it’s been my experience over the last several jobs that many orgs are throwing mission critical workloads and hinging a lot of people’s productivity onto k8s and asking operators and engineers to ostensibly figure it out as they go. And from rants and vents I see on other tech communities, I’m not the only one making this observation. Which is sheer madness to me. Change my mind: unless you have probably, razor sharp engineers who know what they’re doing with k8s enough to give you more than a few days of availability before livened probes start falling off the face of the planet, AND can make this their sole focus, maybe you’re not ready for k8s?
- slillibri 4y ago
  I don't think it's just orgs, as there are a lot of engineers that seem to have bought into the `k8s == awesome` mentality. I like Kubernetes and I think it has a lot to recommend it, but it's not a single silver bullet for all problems.
- rco8786 4y ago
  I would take this a step further to say that an extremely high % (like 95%) of organizations using k8s do not need it and would be much better off not using it.
- deleted 4y ago
  [deleted]
- sixo 4y ago
  I think you have a lot of people coming into the general backend/business-logic tech space who are not nearly enough of "computer people" or engineering-minded to actually be responsible for the uptime of a mission critical workload. They might think they are, but they don't know what it actually takes. (Your "operators and engineers", ostensibly, CAN handle this, but they need to be up to the task too.) K8s among other things is a way of trying to abstract away the most computer/OS/network-literate parts of a software organization, so that you can be productive, as an org, hiring some computer people along with plenty of the readily-available "smart people who can write code." The former keeps your software running, the latter represent your business as code. It's a way of solving for the hiring market you HAVE, rather than the hiring market of 20 yrs ago. But it hinges on those deep-tech people being up to the task.
DawnQFunk4 4y ago
tomerbd 4y ago
If two kubernetes clusters collide will the cosmos survive?
markdog12 4y ago
Previously: https://news.ycombinator.com/item?id=20163500 https://news.ycombinator.com/item?id=20163500 https://news.ycombinator.com/item?id=18953647 https://news.ycombinator.com/item?id=18953647 https://news.ycombinator.com/item?id=26106080 https://news.ycombinator.com/item?id=26106080
PainfullyNormal 4y ago
Kubernetes appears to be continuing to increase in popularity despite a large number of these failure stories. Does anybody know why? What does kubernetes do that docker, AWS, Nomad, et al don't? And, if I'm stuck with kubernetes, is there an accepted "right way" to do kubernetes that will keep me from becoming a cautionary tale?
- dilyevsky 4y ago
  Bc these “failure stories” are nothing but run of the mill outages that anyone who runs distributed systems at scale has seen plenty of
- IThoughtYouGNU 4y ago
  Principal kubernetes consultant for a large firm here. Kubernetes uses Docker runtime so that eliminates your first question. AWS also uses Kubernetes, EKS is a modified version of Kubernetes, and most major companies do not use things like Nomad because they lack the ecosystem tooling that something like Kubernetes has. I think probably the best thing you could do to prevent issues like this is blue-green testing or proper integration testing with your infrastructure repository so you can roll back quickly when you realize that some change you made didn't work correctly. This is all in addition to significant amounts of it monitoring.
- arinlen 4y ago
  > What does kubernetes do that docker, AWS, Nomad, et al don't? Most of your comparisons are at best apples-to-oranges. For example, AWS also offers a managed Kubernetes service, which makes your comparison meaningless and just a keyword dump. Worse, AWS also offers its Elastic Container Service. Docker supports Docker swarm mode, which is easier to setup but inter-node communication is extremely slow and barely supports anything beyond launching containers. But to answer the question of why bother with Kubernetes, the answer is obvious: Kubernetes is the first coherent abstraction over heterogeneous clusters of COTS hardware, including networking, which just works and offers plenty of very useful features such as reversible deployment history. Deployments are entirely defined in code in a way that you can treat both deployments and infrastructure as cattle.
fruit2020 4y ago
What’s the alternative? Every system has its flaws
moolcool 4y ago
It seems to me like a vast majority of startups and use cases don't serve a _ton_ of traffic, nor do they do too much heavy data processing. If we're being realistic, most webapps could probably run on a Raspberry Pi and be fine. With this being the case, why is such a heavy-weight and complicated system so popular?
- abraxas 4y ago
  Literally, design by resume. I see it at my current workplace. Top talent guys are fucking around wit K8s even though we really have no use for it and could host the whole shebang on a single EC2 instance and be no worse off for it. But that "won't scale" I'm told. Never mind our startup has a TAM that involves no more than 1,000 customers with maybe a dozen accounts each.
  - jskrablin 4y ago
    It's not always about traffic scale. Sometimes it's about having a common deployment platform at different locations, on premise. Where relatively simple and small setups still bring a lot of value via pure convenience - suddenly everything is packed into Helm charts, running off basically identical CICD pipelines. But I guess that's not what "top talent guys" are interested in doing?
    - dvtrn 4y ago
      Sometimes it's about having a common deployment platform at different locations, on premise. That’s fair, that’s valid, that doesn’t however strike me as a compelling case to go off and hoist all of your stuff into k8s if there’s other equally performant and fault tolerant methods of running your application that require less operational complexity and cognitive load to triage and resolve failures. It seems to me the problem you bring up would serve an engineering team well to solve and solidify even if the application isn’t running on k8s
- bri3d 4y ago
  Transferrable experience in both directions, in and out, is a huge reason. Cynics see this as "resume driven development," but the reality is that offering training and an opportunity to use an industry standard tool isn't a bad move for hiring and retention. Additionally, as you grow, you can hire experts in k8s who have "been there before" with scale - at a huge cost, yes, but this is still a value versus something homegrown. Also, cloud-hosted k8s systems (especially GKE) aren't half bad these days. Starting with one (carefully, to avoid lock-in) and knowing you can expand later is appealing.

72 points by wseqyrku 4y ago

Someone in our institute thought shifting the FTP service to single pod deployments with no auto-scaling was a good idea. They were wrong.

wmichelin 4y ago
Can they just... setup autoscaling? Hopefully the mistake wasn't too hard to correct.
- 6LLvveMx2koXfwn 4y ago
  Maybe, but introducing several layers of abstraction was never going to be the solution if performance was the problem.
- jasonjayr 4y ago
  With the port mapping that FTP entails? You'd need a k8s aware FTP service that creates load balanced ports on demand. Which sounds like it could get thorny and complicated compared to a pair of AWS EC2 instances over an S3 filesystem (Or use AWS's hosted FTP service?)

ok123456 4y ago

I look forward to the day that we never have to hear about Kubernetes again.

arinlen 4y ago
> look forward to the day that we never have to hear about Kubernetes again. How do you expect to deploy services in a cluster of multiple nodes communicating over a private software-defined network, then?
- coding123 4y ago
  There ARE other ways, but mesos was a freaking mess. I actually liked swarm, but it lost. k8s also brings in pretty novel ideas that destroy proprietary IaaS, such as the nginx (or other) front end ingress routers that I haven't seen implemented meaningfully in a IaaS neutral way. Sure, you may have a few properties in your k8s yaml that change from AWS vs Google vs Azure vs other k8s hosts. That's not something that takes more than a debugging session to get right. That's HUUUGE progress.
  - arinlen 4y ago
    > I actually liked swarm, but it lost. "lost" is too strong of a word. To me Docker swarm mode still holds the upper hand in terms of deployment ease and simplicity. If anyone just wants to bolt together a couple of nodes without much concern regarding performance then Docker swarm mode is perhaps the best option out there at the moment. However, this is changing very fast. Ubuntu now ships with microk8s which just works and outperforms Docker swarm mode. Perhaps Docker swarm's worst enemy is how the project was managed and promoted. They had a tool to trivially setup a cluster of nodes with close to zero challenge, but somehow Kubernetes managed to be pushed as the only option.
- ok123456 4y ago
  docker compose. Start it as a systemd service if you want.
  - majewsky 4y ago
    docker-compose only deploys on individual nodes though.
    - ok123456 4y ago
      That's shifting the goal posts. And, that covers 99.9% of cases anyway. If you want to deploy across multiple nodes run docker compose on multiple nodes and expose the nodes to each other over a vlan that is managed as part of your actual infrastructure where it has a chance of being actually documented and put into a configuration management system.

dvtrn 4y ago

I want someone to check me on a strongly held belief, I promise I have an open mind: There’s nothing wrong with experimenting with k8s in your org. Nothing wrong with k8s, it’s complex yes but that’s not inherently problematic. But it’s been my experience over the last several jobs that many orgs are throwing mission critical workloads and hinging a lot of people’s productivity onto k8s and asking operators and engineers to ostensibly figure it out as they go. And from rants and vents I see on other tech communities, I’m not the only one making this observation. Which is sheer madness to me. Change my mind: unless you have probably, razor sharp engineers who know what they’re doing with k8s enough to give you more than a few days of availability before livened probes start falling off the face of the planet, AND can make this their sole focus, maybe you’re not ready for k8s?

slillibri 4y ago
I don't think it's just orgs, as there are a lot of engineers that seem to have bought into the `k8s == awesome` mentality. I like Kubernetes and I think it has a lot to recommend it, but it's not a single silver bullet for all problems.
rco8786 4y ago
I would take this a step further to say that an extremely high % (like 95%) of organizations using k8s do not need it and would be much better off not using it.
deleted 4y ago
[deleted]
sixo 4y ago
I think you have a lot of people coming into the general backend/business-logic tech space who are not nearly enough of "computer people" or engineering-minded to actually be responsible for the uptime of a mission critical workload. They might think they are, but they don't know what it actually takes. (Your "operators and engineers", ostensibly, CAN handle this, but they need to be up to the task too.) K8s among other things is a way of trying to abstract away the most computer/OS/network-literate parts of a software organization, so that you can be productive, as an org, hiring some computer people along with plenty of the readily-available "smart people who can write code." The former keeps your software running, the latter represent your business as code. It's a way of solving for the hiring market you HAVE, rather than the hiring market of 20 yrs ago. But it hinges on those deep-tech people being up to the task.

DawnQFunk4 4y ago

tomerbd 4y ago

If two kubernetes clusters collide will the cosmos survive?

markdog12 4y ago

Previously: https://news.ycombinator.com/item?id=20163500 https://news.ycombinator.com/item?id=20163500 https://news.ycombinator.com/item?id=18953647 https://news.ycombinator.com/item?id=18953647 https://news.ycombinator.com/item?id=26106080 https://news.ycombinator.com/item?id=26106080

PainfullyNormal 4y ago

Kubernetes appears to be continuing to increase in popularity despite a large number of these failure stories. Does anybody know why? What does kubernetes do that docker, AWS, Nomad, et al don't? And, if I'm stuck with kubernetes, is there an accepted "right way" to do kubernetes that will keep me from becoming a cautionary tale?

dilyevsky 4y ago
Bc these “failure stories” are nothing but run of the mill outages that anyone who runs distributed systems at scale has seen plenty of
IThoughtYouGNU 4y ago
Principal kubernetes consultant for a large firm here. Kubernetes uses Docker runtime so that eliminates your first question. AWS also uses Kubernetes, EKS is a modified version of Kubernetes, and most major companies do not use things like Nomad because they lack the ecosystem tooling that something like Kubernetes has. I think probably the best thing you could do to prevent issues like this is blue-green testing or proper integration testing with your infrastructure repository so you can roll back quickly when you realize that some change you made didn't work correctly. This is all in addition to significant amounts of it monitoring.
arinlen 4y ago
> What does kubernetes do that docker, AWS, Nomad, et al don't? Most of your comparisons are at best apples-to-oranges. For example, AWS also offers a managed Kubernetes service, which makes your comparison meaningless and just a keyword dump. Worse, AWS also offers its Elastic Container Service. Docker supports Docker swarm mode, which is easier to setup but inter-node communication is extremely slow and barely supports anything beyond launching containers. But to answer the question of why bother with Kubernetes, the answer is obvious: Kubernetes is the first coherent abstraction over heterogeneous clusters of COTS hardware, including networking, which just works and offers plenty of very useful features such as reversible deployment history. Deployments are entirely defined in code in a way that you can treat both deployments and infrastructure as cattle.

fruit2020 4y ago

What’s the alternative? Every system has its flaws

moolcool 4y ago

It seems to me like a vast majority of startups and use cases don't serve a _ton_ of traffic, nor do they do too much heavy data processing. If we're being realistic, most webapps could probably run on a Raspberry Pi and be fine. With this being the case, why is such a heavy-weight and complicated system so popular?

abraxas 4y ago
Literally, design by resume. I see it at my current workplace. Top talent guys are fucking around wit K8s even though we really have no use for it and could host the whole shebang on a single EC2 instance and be no worse off for it. But that "won't scale" I'm told. Never mind our startup has a TAM that involves no more than 1,000 customers with maybe a dozen accounts each.
- jskrablin 4y ago
  It's not always about traffic scale. Sometimes it's about having a common deployment platform at different locations, on premise. Where relatively simple and small setups still bring a lot of value via pure convenience - suddenly everything is packed into Helm charts, running off basically identical CICD pipelines. But I guess that's not what "top talent guys" are interested in doing?
  - dvtrn 4y ago
    Sometimes it's about having a common deployment platform at different locations, on premise. That’s fair, that’s valid, that doesn’t however strike me as a compelling case to go off and hoist all of your stuff into k8s if there’s other equally performant and fault tolerant methods of running your application that require less operational complexity and cognitive load to triage and resolve failures. It seems to me the problem you bring up would serve an engineering team well to solve and solidify even if the application isn’t running on k8s
bri3d 4y ago
Transferrable experience in both directions, in and out, is a huge reason. Cynics see this as "resume driven development," but the reality is that offering training and an opportunity to use an industry standard tool isn't a bad move for hiring and retention. Additionally, as you grow, you can hire experts in k8s who have "been there before" with scale - at a huge cost, yes, but this is still a value versus something homegrown. Also, cloud-hosted k8s systems (especially GKE) aren't half bad these days. Starting with one (carefully, to avoid lock-in) and knowing you can expand later is appealing.