5 ms·

The two things thing that gets me are: 1. Latch up states. It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods f

by iasay 4y ago

The two things thing that gets me are: 1. Latch up states. It's very very easy for something to go wrong and blow a whole deployment up and lose all the pods for example a health check failure. Most application frameworks have some sort of request queuing and the health checks sit in the same queue so any upstream issues and you get health check failures and flapping. Of course the autoscaler goes fucking bonkers in the middle of that. The only thing you can do is drop traffic at the network edge and wait for it to get itself together. 2. No one knows how to fix it if anything major goes wrong. Even cloud providers. It's so large and complicated that no one has enough knowledge independently to actually fix it. For example I suffered from months of weird network issues where pods would come up without network. No one to this day know why that happened and could explain it. No amount of debugging and reverse engineering even resulted in a single step forward, resulting in the only outcome being "replace the whole cluster". Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without production experience, which is difficult because there are very few people out there who are battle hardened past trivial home deployments and tiny little stacks.

tpetry 4y ago
> Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without production experience, which is difficult because there are very few people out there who are battle hardened past trivial home deployments and tiny little stacks. That‘s the problem. Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. But when anything does not work correctly the only approach is to kill the complete machine and start a new cluster on a server. Have fun with statefull data which needs to be copied…
- iasay 4y ago
  We sort of waltzed around that one with PV's on EBS on Amazon and also shifted the control plane to them. But there are still some serious problems in that space.
  - sleepybrett 4y ago
    A limitation of EBS on amazon that I've run into a few times is that EBS volumes must connect to EC2 in their zone only. So if your k8s cluster has nodes across multiple AZs (which obv. important) your pod that mounts that pv will always be zone locked. This can also be problematic if you write a pod that mounts claims that are in different zones, that will never work. There are also amazon limitations to how many volumes per node and I used to see problems with ebs volumes 'unmounting' from nodes getting stuck. The later was always problematic and required an admin with a hammer. However i've never seens a 'kube' problem per se, they've all been aws problems.
    - nijave 4y ago
      Resource zone affinity is annoying but usually works fine if you have 1 ASG per zone (Azure and afaik GCP are the same way). You mainly need to be careful during initial creation to make sure the volumes are spread out WaitForFirstConsumer is very helpful, too (you basically guarantee the pod can be scheduled at least once before creating the resources which greatly improves the likelihood it can get rescheduled in the future without getting stuck) Don't remember seeing any issues on AWS but I've seen Azure CSI take about 7.5 minutes to unmount a volume from 1 and mount to another (so each pod in a statefulset can take around 8-10 minutes)
- throwaway894345 4y ago
  It's not like you can let someone run bare VMs without any production experience either. You need a PaaS for something like that, even then... > Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. This hasn't been my experience, and certainly the Kubernetes project doesn't advertise the software as "so simple anyone can do it" or any such thing. Kubernetes definitely requires experience--it's not a PaaS, but a framework on which something like a PaaS could be built.
sleepybrett 4y ago
1. You don't 'lose' the pods. If your pod fails on liveness or readiness checks it restarts, over and over until it passes. 2. Depends on your team for sure. If you are on a team that's like "We'll just spin this up and it will be fine." you ignore the fact that 'things happen'. I've seen similar situations with companies that deploy on bare linux servers as well, some update breaks something or something isn't optimally configured. Things go wrong with systems, people who know the systems are needed to fix them. It sounds like if you intend on using kubernetes you should learn to troubleshoot it. It's not difficult to find people who know how to actually run kubernetes, it's just hard to convince them to switch jobs for you. I get a decent numbers of cold, non recruiter, linkedin contacts a month, and frankly many recruiters... but my current job pays well and matches my worklife balance. Zero people who are reaching out from startups offer the whole package, they can usually come in fine on the money, but I'm not working a hundred hours a week with little to no support. On the 'established corporate enterprise' side of the house, they can be inflexible when it comes to vacation time, salary ranges etc. but I've found a good place.
- kule 4y ago
  RE 1. Just to be precise (as I thought this until recently): * liveness check will cause pod restarts. * Readiness check causes the pod to be removed from the round-robin of new traffic requests so it has time to recover/finish processing what it's working on.
  - sleepybrett 4y ago
    You are correct, readiness removes it from any service objects it would be an endpoint to. My point was more 'the pod doesn't go away'. I've seen some people do stuff with the HPA that could cause it to scale down to minimum replicas if its in a broken state, depending on what stats you are using to scale, but that's more of a 'kubernetes doing what you told it to do' problem.
otabdeveloper4 4y ago
> Don't get me wrong, I still like it Stockholm syndrome?

by iasay 4y ago

tpetry 4y ago

> Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without production experience, which is difficult because there are very few people out there who are battle hardened past trivial home deployments and tiny little stacks. That‘s the problem. Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. But when anything does not work correctly the only approach is to kill the complete machine and start a new cluster on a server. Have fun with statefull data which needs to be copied…

iasay 4y ago
We sort of waltzed around that one with PV's on EBS on Amazon and also shifted the control plane to them. But there are still some serious problems in that space.
- sleepybrett 4y ago
  A limitation of EBS on amazon that I've run into a few times is that EBS volumes must connect to EC2 in their zone only. So if your k8s cluster has nodes across multiple AZs (which obv. important) your pod that mounts that pv will always be zone locked. This can also be problematic if you write a pod that mounts claims that are in different zones, that will never work. There are also amazon limitations to how many volumes per node and I used to see problems with ebs volumes 'unmounting' from nodes getting stuck. The later was always problematic and required an admin with a hammer. However i've never seens a 'kube' problem per se, they've all been aws problems.
  - nijave 4y ago
    Resource zone affinity is annoying but usually works fine if you have 1 ASG per zone (Azure and afaik GCP are the same way). You mainly need to be careful during initial creation to make sure the volumes are spread out WaitForFirstConsumer is very helpful, too (you basically guarantee the pod can be scheduled at least once before creating the resources which greatly improves the likelihood it can get rescheduled in the future without getting stuck) Don't remember seeing any issues on AWS but I've seen Azure CSI take about 7.5 minutes to unmount a volume from 1 and mount to another (so each pod in a statefulset can take around 8-10 minutes)
throwaway894345 4y ago
It's not like you can let someone run bare VMs without any production experience either. You need a PaaS for something like that, even then... > Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. This hasn't been my experience, and certainly the Kubernetes project doesn't advertise the software as "so simple anyone can do it" or any such thing. Kubernetes definitely requires experience--it's not a PaaS, but a framework on which something like a PaaS could be built.

sleepybrett 4y ago

1. You don't 'lose' the pods. If your pod fails on liveness or readiness checks it restarts, over and over until it passes. 2. Depends on your team for sure. If you are on a team that's like "We'll just spin this up and it will be fine." you ignore the fact that 'things happen'. I've seen similar situations with companies that deploy on bare linux servers as well, some update breaks something or something isn't optimally configured. Things go wrong with systems, people who know the systems are needed to fix them. It sounds like if you intend on using kubernetes you should learn to troubleshoot it. It's not difficult to find people who know how to actually run kubernetes, it's just hard to convince them to switch jobs for you. I get a decent numbers of cold, non recruiter, linkedin contacts a month, and frankly many recruiters... but my current job pays well and matches my worklife balance. Zero people who are reaching out from startups offer the whole package, they can usually come in fine on the money, but I'm not working a hundred hours a week with little to no support. On the 'established corporate enterprise' side of the house, they can be inflexible when it comes to vacation time, salary ranges etc. but I've found a good place.

kule 4y ago
RE 1. Just to be precise (as I thought this until recently): * liveness check will cause pod restarts. * Readiness check causes the pod to be removed from the round-robin of new traffic requests so it has time to recover/finish processing what it's working on.
- sleepybrett 4y ago
  You are correct, readiness removes it from any service objects it would be an endpoint to. My point was more 'the pod doesn't go away'. I've seen some people do stuff with the HPA that could cause it to scale down to minimum replicas if its in a broken state, depending on what stats you are using to scale, but that's more of a 'kubernetes doing what you told it to do' problem.

otabdeveloper4 4y ago

> Don't get me wrong, I still like it Stockholm syndrome?