8 ms·

A visual guide on troubleshooting Kubernetes deployments

240 points by Cantaro86 7y ago

StreamBright 7y ago
I guess I can just print this out and show to people any time somebody wants to move to k8s.
- domlebo70 7y ago
  My thoughts exactly. It's too complex for anything I'v ever needed.
  - nailer 7y ago
    Well yes, people who already pay for external cloud providers running k8s for little business value is a meme at this point. If you are creating a cloud provider though then you're very much welcome to use k8s.
  - q3k 7y ago
    I'm somewhat sure you could make a similarly complicated looking chart that would entail: - using ping - sshing into a box - using ssh port forwarding - using ps aux - understanding system load - understanding oom killer messages - is this systemd? use systemctl, otherwise figure it out - is this systemd? use journalctl, otherwise figure out where an app is logging - is nginx running? what box is it running on? - is the nginx config okay? - what about haproxy? are health checks going through? Plus dealing with whatever configuration management system you're using, credential store you're using, etc. Kubernetes, while complex, does take over a huge chunk of responsibilities from existing deployments. As such, a large part of any debugging will be done via kubectl and on a much different level of abstraction than if you were dealing with individual machines and services deployed on them via Ansible. Yes, there's new things to learn. But for many cases (not even just scale!) it's quite worth it.
    - base698 7y ago
      TBF a lot of people don't know what any of that is and use Heroku or equivalent. I've worked with people who don't even know how to use find or chmod, much less ps, htop, and other system tools.
    - StreamBright 7y ago
      Yep, and many of these projects are 10+ years old rock solid code that very rarely breaks and the failure scenarios are well understood (the only real exception is systemd). Can you tell the same about k8s?
  - vitalus 7y ago
    I understand that Kubernetes is a complex project, but I struggle to see how this comment adds to the discourse. For those that do need a solution like Kubernetes, charts like this are helpful, and the knowledge requirements certainly aren't too steep relative to comparable platforms. What about Kubernetes makes it especially worth going in and commenting that you don't have a use case for it, in a thread dedicated to Kubernetes? Would you make similar comments about other platforms/libraries/technologies that you, for whatever reason, don't have a use case for?
    - fheyfhth14353 7y ago
      Because echo chambers need their bubble's pierced every so often. Kubernetes is such garbage that no one runs it on their servers and if they do they have an army dedicated to managing it 24/7. Go to any company that's running it and ask them how they feel about it.
    - StreamBright 7y ago
      Because many people (including managers) take HN seriously and then go and try to implement or push for this being implemented because they read it on HN. Unfortunately they cannot go to HN when a outage explodes in their face, caused by an unknown problem in k8s. There is no clear value proposition for using k8s. It is trading time to figure out what would make the most sense for a project for black box complexity in most cases and that is a very bad deal. You pay the price once for the first one and all the time for the second one. If I do not comment here then you have a nice echo chamber that all fine and dandy with k8s and there is no downside at all using it. Btw. it is not only k8s and HN supposed to be a discussion site not a fanboy club of broken tech. I understand that the latter is much more appealing to many people.
- Kamshak 7y ago
  There are for sure a few concepts you need to learn with k8s, however I wonder if it would be much shorter if you did a guide like this for manually done / bash script powered / ansible deployments
  - organsnyder 7y ago
    Every other large-ish product I've maintained would have a chart at least this large. Complex systems are complicated to maintain, regardless of platform.
    - base698 7y ago
      You've worked on complex systems with charts? Most places I've worked lack charts and documentation. The flow chart only exists in pieces of the people's heads still working there, the rest in employees who left. That's one of the strengths with K8s, at least there is documentation, unlike a 1000 line perl script with no tests which does service discovery.
      - organsnyder 7y ago
        I said "would have a chart" not "had a chart". :-D Of course it wasn't properly documented. :-( I fully agree on that being a major strength of k8s. Looking at the resource manifests gives a ton of insight into how a system is supposed to function.
  - StreamBright 7y ago
    > manually done / bash script powered / ansible deployments Sure but I am not comparing it against manually done / bash script powered / Ansible deployments. Even if I was the CI/CD pipeline would capture many of the problems early and than I have an immutable AMI with autoscaling groups that is very easy to debug, networking setup is super simple and load-balancing is also straightforward to troubleshoot. Cramming all of these into a simple project that is not as easy to debug is not worth the effort for us.
  - linsomniac 7y ago
- kube-system 7y ago
  It’s 11 commands to debug everything about an application deployment... from downloading the docker image all the way to debugging the load balancer. Most git cheat sheets have 3 times the number of commands. Most of the diamonds listed here aren’t things you have to remember, they’re statuses that are explicitly called out in the responses of those commands. This is an over-complicated chart that in reality is summed up as: look at the pod, look at the service, look at ingress.
- markbnj 7y ago
  > I guess I can just print this out and show to people any time somebody wants to move to k8s. A complete flow chart of all the possible states for any non-trivial thing appears complex. In practice the actual problems you run into are part of a much smaller set of possibilities.
  - StreamBright 7y ago
    Yeah, but I also happened to run into production problems non of the k8s exports could fix or even pinpoint why it was happening.
- fheyfhth14353 7y ago
  Only sane response here and being downvoted to Oblivion.
  - StreamBright 7y ago
    Well HN turned into this place where you cannot have a decent discussion anymore, there is the downvoter crew and if you dare to disagree with them they nuke your comments. Unfortunate.
    - gouggoug 7y ago
      Well, it's not like the GP was trying to create a decent conversation in the first place; it wasn't and that's actually _why_ it's being downvoted.
      - StreamBright 7y ago
        I guess it was too bad that I wanted to point out how bad is the complexity of k8s.
        asdfggh123 7y ago
        I've stopped caring about anything related to a permanent handle. Say what you need to say with anonymous accounts. Turns out it doesn't really matter in the grand scheme of things.
        madeuppoints 7y ago
        Because the points are made up and comments are a lousy signal of prestige. It's much better to make something and let the work speak for itself.
organsnyder 7y ago
Note that this flowchart includes monitoring, logging, and networking, in addition to typical app stuff. Of course it's complicated. The article does contain a lot of useful information. I don't think it was intended to ride the "k8s is too complicated compared to vanilla X" bandwagon, despite the huge flowchart at the top.
- Kaiyou 7y ago
  Flowchar looks pretty straightforward to me. Nothing I'd categorize as complicated.
  - emj 7y ago
    It's a good chart if you want to teach people the commands needed for trouble shooting.
gravypod 7y ago
Looking at this I think it would be really cool for someone to build a Wheel of Misfortune to help dry run some of these scenarios for debugging.
billfruit 7y ago
Haven't had to work with k8s yet, but that flow chart looks really detailed. It must have taken much effort to make it, that makes me wonder, for how long shall it remain current? Will 6 months down the line it become invalid in subtle manner that it can only add to the confusion.
- etxm 7y ago
  All the CLI commands used have been in for quite a few “minor” versions. I’d say it’s probably valid for the foreseeable future for debugging k8s _primitive_ resources. With the customization you can do to k8s the debugging can get a bit more weird when dealing with operators and web hooks.
nodesocket 7y ago
Is there a way to prevent Kubernetes from killing and restarting a pod (from a deployment) when you are debugging it with kubectl exec -it? I.E. inform Kubernetes I am using this pod, don’t restart it automatically.
- markbnj 7y ago
  Depends on why it is getting restarted. If it's exceeding mem limits and being oomkilled that's the kernel, not k8s. If PID 1 inside the namespace is terminating then k8s will restart the pod. No way to prevent that I am aware of, but presumably you can't do much debugging once that happens anyway. If the process is failing liveness probes and getting terminated for being unhealthy probably the simplest approach is just to patch away those probes until you have the workload stable.
  - nodesocket 7y ago
    Yes failing liveness probe.
    - anirudhrx 7y ago
      Liveness probes are used by the kubelet to restart the underlying container and are independent of the deployment object. This has come up before in https://github.com/kubernetes/kubernetes/issues/57187 https://github.com/kubernetes/kubernetes/issues/57187 but sadly, isn't possible yet. Your best bet is to create a new pod and hope for repro, or one way might be to have a configmap that is mounted into the pod that contains a debug flag that your liveness probe also looks at - i.e. "debug == true || curl localhost:6789". Not a clean solution but may work for the interim.
- plughs 7y ago
  Not as far as I can tell, and as far as I'm concerned it is the biggest pain point when debugging k8s problems. If your pods is cycling though CrashLoopBackOff and 'describe' has no useful information and the 'logs' don't give you a clue, you are flat out of luck. That pod is gone before you can extract any useful information from it. I'd like to see some means of keeping a pod running even if the application is crashing. Sometimes logs are being sent to a file, sometimes running the application manually gives a clue. Lacking that - well I've found various tricks such as changing the image to 'ubuntu' and replacing the command with 'tail -f /dev/null' so a crashing pod will stay alive and I can exec into it and run the real command by hand. But it's a hack and doesn't always get the info you need.
de_watcher 7y ago
What about kubectl randomly hanging and some PLEG errors in the kubelet log?
- erulabs 7y ago
  Kubectl randomly hanging sounds like a problem with the Metrics api. You can use -v9 on kubectl and see what it’s pausing on. Note that metrics aren’t refreshed every single run - instead on a timer - hence the “random” feel
orthoxerox 7y ago
I like how it has quite a few boxes that say "no idea what the reason might be". Each one of them hides another chart as big as this one, if not bigger.
jtdev 7y ago
Yikes
nathanwh 7y ago
I enjoyed petulantly answering every question with “no” and the funding the final state to be “Consult Stackoverflow”
- dsnuh 7y ago
  Try https://magick8sball.github.io/ https://magick8sball.github.io/ for those scenarios. :)
fheyfhth14353 7y ago
Tickles my funny bone every time I see the industry jump on a new bandwagon that starts simple and then inevitably adapts to the real world that the old technology was addressing. When are we gonna learn folks? Props to the author for the chart. He's actually providing real value to all the suckers, err, SREs stuck dealing with this stuff.
tinco 7y ago
Fun fact, on GCP Kubernetes you can have green lightbulbs on every single dashboard, and your entire site can be down anyway. Our CI/CD was leaking "review" deployments, I forgot about them until one day I upgraded a node and the entire site went down, even though everything was green. Turned out there is some sort of naximum amount of nginx entries in ingress and we were hitting it. That was some frantic debugging, solution was just to delete the spurious review deployments.
- nhumrich 7y ago
  This is true of anything, not just k8s. There is always a possibility of all system monitors to be green but the apps dead/bad for some reason. This is why you need more than system monitors. You need app monitoring as well.
  - StreamBright 7y ago
    That is called broken monitoring.

240 points by Cantaro86 7y ago

StreamBright 7y ago

I guess I can just print this out and show to people any time somebody wants to move to k8s.

domlebo70 7y ago
My thoughts exactly. It's too complex for anything I'v ever needed.
- nailer 7y ago
  Well yes, people who already pay for external cloud providers running k8s for little business value is a meme at this point. If you are creating a cloud provider though then you're very much welcome to use k8s.
- q3k 7y ago
  I'm somewhat sure you could make a similarly complicated looking chart that would entail: - using ping - sshing into a box - using ssh port forwarding - using ps aux - understanding system load - understanding oom killer messages - is this systemd? use systemctl, otherwise figure it out - is this systemd? use journalctl, otherwise figure out where an app is logging - is nginx running? what box is it running on? - is the nginx config okay? - what about haproxy? are health checks going through? Plus dealing with whatever configuration management system you're using, credential store you're using, etc. Kubernetes, while complex, does take over a huge chunk of responsibilities from existing deployments. As such, a large part of any debugging will be done via kubectl and on a much different level of abstraction than if you were dealing with individual machines and services deployed on them via Ansible. Yes, there's new things to learn. But for many cases (not even just scale!) it's quite worth it.
  - base698 7y ago
    TBF a lot of people don't know what any of that is and use Heroku or equivalent. I've worked with people who don't even know how to use find or chmod, much less ps, htop, and other system tools.
  - StreamBright 7y ago
    Yep, and many of these projects are 10+ years old rock solid code that very rarely breaks and the failure scenarios are well understood (the only real exception is systemd). Can you tell the same about k8s?
- vitalus 7y ago
  I understand that Kubernetes is a complex project, but I struggle to see how this comment adds to the discourse. For those that do need a solution like Kubernetes, charts like this are helpful, and the knowledge requirements certainly aren't too steep relative to comparable platforms. What about Kubernetes makes it especially worth going in and commenting that you don't have a use case for it, in a thread dedicated to Kubernetes? Would you make similar comments about other platforms/libraries/technologies that you, for whatever reason, don't have a use case for?
  - fheyfhth14353 7y ago
    Because echo chambers need their bubble's pierced every so often. Kubernetes is such garbage that no one runs it on their servers and if they do they have an army dedicated to managing it 24/7. Go to any company that's running it and ask them how they feel about it.
  - StreamBright 7y ago
    Because many people (including managers) take HN seriously and then go and try to implement or push for this being implemented because they read it on HN. Unfortunately they cannot go to HN when a outage explodes in their face, caused by an unknown problem in k8s. There is no clear value proposition for using k8s. It is trading time to figure out what would make the most sense for a project for black box complexity in most cases and that is a very bad deal. You pay the price once for the first one and all the time for the second one. If I do not comment here then you have a nice echo chamber that all fine and dandy with k8s and there is no downside at all using it. Btw. it is not only k8s and HN supposed to be a discussion site not a fanboy club of broken tech. I understand that the latter is much more appealing to many people.
Kamshak 7y ago
There are for sure a few concepts you need to learn with k8s, however I wonder if it would be much shorter if you did a guide like this for manually done / bash script powered / ansible deployments
- organsnyder 7y ago
  Every other large-ish product I've maintained would have a chart at least this large. Complex systems are complicated to maintain, regardless of platform.
  - base698 7y ago
    You've worked on complex systems with charts? Most places I've worked lack charts and documentation. The flow chart only exists in pieces of the people's heads still working there, the rest in employees who left. That's one of the strengths with K8s, at least there is documentation, unlike a 1000 line perl script with no tests which does service discovery.
    - organsnyder 7y ago
      I said "would have a chart" not "had a chart". :-D Of course it wasn't properly documented. :-( I fully agree on that being a major strength of k8s. Looking at the resource manifests gives a ton of insight into how a system is supposed to function.
- StreamBright 7y ago
  > manually done / bash script powered / ansible deployments Sure but I am not comparing it against manually done / bash script powered / Ansible deployments. Even if I was the CI/CD pipeline would capture many of the problems early and than I have an immutable AMI with autoscaling groups that is very easy to debug, networking setup is super simple and load-balancing is also straightforward to troubleshoot. Cramming all of these into a simple project that is not as easy to debug is not worth the effort for us.
- linsomniac 7y ago
kube-system 7y ago
It’s 11 commands to debug everything about an application deployment... from downloading the docker image all the way to debugging the load balancer. Most git cheat sheets have 3 times the number of commands. Most of the diamonds listed here aren’t things you have to remember, they’re statuses that are explicitly called out in the responses of those commands. This is an over-complicated chart that in reality is summed up as: look at the pod, look at the service, look at ingress.
markbnj 7y ago
> I guess I can just print this out and show to people any time somebody wants to move to k8s. A complete flow chart of all the possible states for any non-trivial thing appears complex. In practice the actual problems you run into are part of a much smaller set of possibilities.
- StreamBright 7y ago
  Yeah, but I also happened to run into production problems non of the k8s exports could fix or even pinpoint why it was happening.
fheyfhth14353 7y ago
Only sane response here and being downvoted to Oblivion.
- StreamBright 7y ago
  Well HN turned into this place where you cannot have a decent discussion anymore, there is the downvoter crew and if you dare to disagree with them they nuke your comments. Unfortunate.
  - gouggoug 7y ago
    Well, it's not like the GP was trying to create a decent conversation in the first place; it wasn't and that's actually _why_ it's being downvoted.
    - StreamBright 7y ago
      I guess it was too bad that I wanted to point out how bad is the complexity of k8s.
      - asdfggh123 7y ago
        I've stopped caring about anything related to a permanent handle. Say what you need to say with anonymous accounts. Turns out it doesn't really matter in the grand scheme of things.
      - madeuppoints 7y ago
        Because the points are made up and comments are a lousy signal of prestige. It's much better to make something and let the work speak for itself.

organsnyder 7y ago

Note that this flowchart includes monitoring, logging, and networking, in addition to typical app stuff. Of course it's complicated. The article does contain a lot of useful information. I don't think it was intended to ride the "k8s is too complicated compared to vanilla X" bandwagon, despite the huge flowchart at the top.

Kaiyou 7y ago
Flowchar looks pretty straightforward to me. Nothing I'd categorize as complicated.
- emj 7y ago
  It's a good chart if you want to teach people the commands needed for trouble shooting.

gravypod 7y ago

Looking at this I think it would be really cool for someone to build a Wheel of Misfortune to help dry run some of these scenarios for debugging.

billfruit 7y ago

Haven't had to work with k8s yet, but that flow chart looks really detailed. It must have taken much effort to make it, that makes me wonder, for how long shall it remain current? Will 6 months down the line it become invalid in subtle manner that it can only add to the confusion.

etxm 7y ago
All the CLI commands used have been in for quite a few “minor” versions. I’d say it’s probably valid for the foreseeable future for debugging k8s _primitive_ resources. With the customization you can do to k8s the debugging can get a bit more weird when dealing with operators and web hooks.

nodesocket 7y ago

Is there a way to prevent Kubernetes from killing and restarting a pod (from a deployment) when you are debugging it with kubectl exec -it? I.E. inform Kubernetes I am using this pod, don’t restart it automatically.

markbnj 7y ago
Depends on why it is getting restarted. If it's exceeding mem limits and being oomkilled that's the kernel, not k8s. If PID 1 inside the namespace is terminating then k8s will restart the pod. No way to prevent that I am aware of, but presumably you can't do much debugging once that happens anyway. If the process is failing liveness probes and getting terminated for being unhealthy probably the simplest approach is just to patch away those probes until you have the workload stable.
- nodesocket 7y ago
  Yes failing liveness probe.
  - anirudhrx 7y ago
    Liveness probes are used by the kubelet to restart the underlying container and are independent of the deployment object. This has come up before in https://github.com/kubernetes/kubernetes/issues/57187 https://github.com/kubernetes/kubernetes/issues/57187 but sadly, isn't possible yet. Your best bet is to create a new pod and hope for repro, or one way might be to have a configmap that is mounted into the pod that contains a debug flag that your liveness probe also looks at - i.e. "debug == true || curl localhost:6789". Not a clean solution but may work for the interim.
plughs 7y ago
Not as far as I can tell, and as far as I'm concerned it is the biggest pain point when debugging k8s problems. If your pods is cycling though CrashLoopBackOff and 'describe' has no useful information and the 'logs' don't give you a clue, you are flat out of luck. That pod is gone before you can extract any useful information from it. I'd like to see some means of keeping a pod running even if the application is crashing. Sometimes logs are being sent to a file, sometimes running the application manually gives a clue. Lacking that - well I've found various tricks such as changing the image to 'ubuntu' and replacing the command with 'tail -f /dev/null' so a crashing pod will stay alive and I can exec into it and run the real command by hand. But it's a hack and doesn't always get the info you need.

de_watcher 7y ago

What about kubectl randomly hanging and some PLEG errors in the kubelet log?

erulabs 7y ago
Kubectl randomly hanging sounds like a problem with the Metrics api. You can use -v9 on kubectl and see what it’s pausing on. Note that metrics aren’t refreshed every single run - instead on a timer - hence the “random” feel

orthoxerox 7y ago

I like how it has quite a few boxes that say "no idea what the reason might be". Each one of them hides another chart as big as this one, if not bigger.

jtdev 7y ago

Yikes

nathanwh 7y ago

I enjoyed petulantly answering every question with “no” and the funding the final state to be “Consult Stackoverflow”

dsnuh 7y ago
Try https://magick8sball.github.io/ https://magick8sball.github.io/ for those scenarios. :)

fheyfhth14353 7y ago

Tickles my funny bone every time I see the industry jump on a new bandwagon that starts simple and then inevitably adapts to the real world that the old technology was addressing. When are we gonna learn folks? Props to the author for the chart. He's actually providing real value to all the suckers, err, SREs stuck dealing with this stuff.

tinco 7y ago

Fun fact, on GCP Kubernetes you can have green lightbulbs on every single dashboard, and your entire site can be down anyway. Our CI/CD was leaking "review" deployments, I forgot about them until one day I upgraded a node and the entire site went down, even though everything was green. Turned out there is some sort of naximum amount of nginx entries in ingress and we were hitting it. That was some frantic debugging, solution was just to delete the spurious review deployments.

nhumrich 7y ago
This is true of anything, not just k8s. There is always a possibility of all system monitors to be green but the apps dead/bad for some reason. This is why you need more than system monitors. You need app monitoring as well.
- StreamBright 7y ago
  That is called broken monitoring.