7 ms·

Debugging Network Stalls on Kubernetes

180 points by chmaynard 7y ago

destitude 7y ago
Would these zombie cgroups be able to be detected in any of the kubernetes statsd metrics emitted?
- theojulienne 7y ago
  We didn't find any metric that surfaced zombie cgroups, presumably because the kernel mostly tries to hide them from user space since they have been deleted, but haven't been cleaned up. The only way we found at the time to track them was via a BCC script and observing the latency on reading the /sys/fs/cgroup/memory/memory.stat file.
seminatl 7y ago
This does not really have anything to do with either kubernetes or networks. If your computer is busy, it won't be able to process packets. Accessing certain kernel stats via proc, sys, or other special files may be really expensive. For example /proc/pid/smaps of a running mysqld takes 2 seconds on a computer I happen to have on hand. Sometimes when you have many cores it is expensive to produce some of the fields of /proc/pid/stat because the kernel has to visit numerous per-cpu data structures. /proc/pid/statm is better for this reason, if it contains what you are looking for. TL;DR reading kernel stats can take a long time and cost a lot of CPU cycles. It costs more for more containers, and more on bigger machines.
- CapacitorSet 7y ago
  I feel like the title is somewhat appropriate: part of the post was about selectively removing parts of Kubernetes networking, and digging through the kernel networking stack, to troubleshoot the issue
- gouggoug 7y ago
  True, it does not have anything to do with k8s or networks, but, that's the context in which this issue arose: when they noticed a higher network latency on a kubernetes cluster. The value of this blog post isn't only in the "why" ("reading kernel stats can take a long time and cost a lot of CPU cycles"), but also in the "how" they went about finding the cause of the symptoms is also of interest.
- simmers 7y ago
  Simple answer really - anything with Kubernetes in the title gets more clicks.
- rumanator 7y ago
  > This does not really have anything to do with either kubernetes or networks. The article is literally about an issue that was experienced while operating kkbernetes clusters. FTA: > Essentially, applications running on our Kubernetes clusters would observe seemingly random latency of up to and over 100ms on connections, which would cause downstream timeouts or retries. Sounds like a problem affecting Kubernetes to me, and an important one. More importantly, it sounds like a non-trivial problem that others operating Kubernetes clusters would be interested in learning how to identify and how to search for the root cause.
  - seminatl 7y ago
    It has literally nothing to do with K8s. An equally suitable title would have been "Debugging network stalls on the Intel Xeon processor" or "Debugging network stalls on planet Earth".
sciurus 7y ago
Fantastic write-up. I'm curious how many people were involved in the investigation and how long it took.
- theojulienne 7y ago
  This is a great question, thank you for asking! Initially a few teams around the org had folks investigating poor performance from different perspectives of the applications that were observing issues. Once it was clear that it wasn’t the applications themselves or their configuration at fault, the team that runs our Kubernetes infrastructure started collating information together (in github issues) and getting to the point of having a clear repro (the Vegeta test) and what to look out for. This was the slowest part of the process because we needed to understand that something non-application-level was going on (and because “random network latency” is a very difficult thing to narrow down) - it probably took on the order of months from the first sign of an issue to fixing all the other issues that were contributing to small amounts of latency and being sure we still had an underlying problem to find. At that point it became clear that something more low level was going on, we put together a focus team from a selection of teams to investigate the underlying cause - that was a group of about 5 engineers actively working on it, with another 5-10 interested engineers following along and helping out. Folks were typically working in pairs or solo to dive in to different potential leads, looping in everyone else in Slack as they go. Most of the work here was finding signal in the noise, we found a lot of other smaller system-level issues along the way that got ruled out and/or low priority to fix. There were other DNS related issues at play, fixing those also improved things, but not the specific underlying issue in the post here. Going down the specific path in the post took just a few days once the first few steps showed something was wrong at the packet level. The remediation from there was also just a few days, because we already had infrastructure in place to detect a known issue and mitigate in a safe/graceful way. The focus team was working on this as a primary task for around a few weeks overall.
bigbluedots 7y ago
I learned a lot reading this writeup, thanks for posting!
Polyisoprene 7y ago
Inefficient Linux cadvisor implementation causes unexpected latency. Also this affects buzzword compatible technologies.
- nielsole 7y ago
  What makes you say it was a cadvisor issue? Isn't it clearly a kernel issue?
  - alecco 7y ago
    It looks more like Docker is pushing the Linux kernel in unprecedented ways. You can't blame Linux for having a cache there for (so-far) normal workloads. And cadvisor was pushing the problem further by reading constantly stats.
hogetesco 7y ago
I read the article with great fun. The question is, how do I run the bcc script in the article? Is it on a sidecar container or on a worker node machine?
- theojulienne 7y ago
  The bcc script was run on the Kubernetes node itself directly over SSH, but it should be possible to run it in a privileged container as well.
suresk 7y ago
Super interesting, thanks for sharing! These things are always my favorite to deal with - the feeling at the end when you figure it out is amazing, and you usually learn a ton on your way there, too.
maxpert 7y ago
I keep seeing articles showing up on debugging network issues like DNS or something else. Makes me wonder how much engineering effort we are putting in to fight the tool. Have these people considered/investigated other tools like Nomad (hashi corp)? How much value Kube actually adds vs these issue?
- sytelus 7y ago
  Kubernetes is CORBA of this generation. It will float around because of few heavyweights preaching it and suckers falling for it. Eventually it will die like all overly-complex, hideously-designed and poorly-implemented things do. You can save your time by just ignoring this nonsense.
  - loopz 7y ago
    Remembering CORBA and glad it never took off: What if Kubernetes is the java of this generation? Java is not yet dying, and for better or worse, do solve organizational problems around code. Kubernetes is likewise becoming a business standard. Look into VMWare, IBM/RedHat and remote service providers offering it as standard. One can write huge books on java, both on the strengths and solutions, but also the weaknesses. For smaller companies and projects, it would probably be overkill to start with the java/JVM monolith, even today. Though java is the reason I jumped off the dev bandwagon, I can defend the language in hindsight, because it do solve several huge "unnecessary" problems other languages/runtimes simply don't address. The operational costs having been hidden away with expensive hardware upgrades. In hindsight, I don't see huge obstacles that the same may be said about Kubernetes in the future. These low-level kinks will be worked out, and provide for the masses what they can't or won't achieve alone. The masses don't need Google-scale, but if they can get the 15-year old mature distributed systems tech almost for free, there are several reasons to standardize.
  - nickthemagicman 7y ago
    I think React.js is the CORBA of this generation. k8's might be too but I think the fact that containers are so portable it will be easy to migrate your application away from k8s if you want and to a new infrastructure. That's kind of the beauty of it.
KaiserPro 7y ago
What I am struggling to understand is why is it a good idea to to ip inside ip. Giving containers their own nic, and IP isn't hard, and in most cases is >> faster than spending time using some immature tunnelling protocol. I can see that doing multi-cloud deploys with transparent failover might benefit from a VPN, but short of operating on a hostile network I can't see its worth the heartache to deploy an overlay network.
- navaati 7y ago
  This a thousand ! You don't even need to give them actual nics, just veth pairs with routing (some CNI plugons do just that btw). I think a big part of why overlay nets are so common is that a lot of the engineers deploying these systems have a shaky or non existant understanding of networking and routing, they are stuck in the "local network and a default route" model and if you only have that, overlay it is...
  - silasb 7y ago
    Do you know what CNI plugins support veth pairs?
    - boulos 7y ago
      The built-in bridge one does. Maybe you meant something else by your question though?
    - CameronNemo 7y ago
      https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md https://github.com/containernetworking/plugins/blob/master/p...
    - elFarto 7y ago
      I've tried using Bridget[1] before, but didn't actually manage to get it working properly (although the problem was likely firewall related). [1] https://github.com/kvaps/bridget https://github.com/kvaps/bridget
    - 7y ago
acd 7y ago
There has been issues with Kubernetes and NAT reported from Xing engineering. 12 min into the video https://www.youtube.com/watch?v=MoIdU0J0f0E https://www.youtube.com/watch?v=MoIdU0J0f0E
- theojulienne 7y ago
  This insert_failed issue described in the video was one of the ones we discovered during investigations as well, but it was already well understood because of this excellent Xing blog post which was extremely useful and referenced internally a lot: https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02 https://tech.xing.com/a-reason-for-unexplained-connection-ti...
je42 7y ago
Awesome deep dive ! I wonder how one can debug this without deep insights into the networking stack. I hope i ll never have a problem like this myself ;)
joemag 7y ago
Is the tunneling protocol really IPIP for these overlays? Oh boy - that’s going to really suck on wide multi path networks like the ones used by cloud providers.

180 points by chmaynard 7y ago

destitude 7y ago

Would these zombie cgroups be able to be detected in any of the kubernetes statsd metrics emitted?

theojulienne 7y ago
We didn't find any metric that surfaced zombie cgroups, presumably because the kernel mostly tries to hide them from user space since they have been deleted, but haven't been cleaned up. The only way we found at the time to track them was via a BCC script and observing the latency on reading the /sys/fs/cgroup/memory/memory.stat file.

seminatl 7y ago

This does not really have anything to do with either kubernetes or networks. If your computer is busy, it won't be able to process packets. Accessing certain kernel stats via proc, sys, or other special files may be really expensive. For example /proc/pid/smaps of a running mysqld takes 2 seconds on a computer I happen to have on hand. Sometimes when you have many cores it is expensive to produce some of the fields of /proc/pid/stat because the kernel has to visit numerous per-cpu data structures. /proc/pid/statm is better for this reason, if it contains what you are looking for. TL;DR reading kernel stats can take a long time and cost a lot of CPU cycles. It costs more for more containers, and more on bigger machines.

CapacitorSet 7y ago
I feel like the title is somewhat appropriate: part of the post was about selectively removing parts of Kubernetes networking, and digging through the kernel networking stack, to troubleshoot the issue
gouggoug 7y ago
True, it does not have anything to do with k8s or networks, but, that's the context in which this issue arose: when they noticed a higher network latency on a kubernetes cluster. The value of this blog post isn't only in the "why" ("reading kernel stats can take a long time and cost a lot of CPU cycles"), but also in the "how" they went about finding the cause of the symptoms is also of interest.
simmers 7y ago
Simple answer really - anything with Kubernetes in the title gets more clicks.
rumanator 7y ago
> This does not really have anything to do with either kubernetes or networks. The article is literally about an issue that was experienced while operating kkbernetes clusters. FTA: > Essentially, applications running on our Kubernetes clusters would observe seemingly random latency of up to and over 100ms on connections, which would cause downstream timeouts or retries. Sounds like a problem affecting Kubernetes to me, and an important one. More importantly, it sounds like a non-trivial problem that others operating Kubernetes clusters would be interested in learning how to identify and how to search for the root cause.
- seminatl 7y ago
  It has literally nothing to do with K8s. An equally suitable title would have been "Debugging network stalls on the Intel Xeon processor" or "Debugging network stalls on planet Earth".

sciurus 7y ago

Fantastic write-up. I'm curious how many people were involved in the investigation and how long it took.

theojulienne 7y ago
This is a great question, thank you for asking! Initially a few teams around the org had folks investigating poor performance from different perspectives of the applications that were observing issues. Once it was clear that it wasn’t the applications themselves or their configuration at fault, the team that runs our Kubernetes infrastructure started collating information together (in github issues) and getting to the point of having a clear repro (the Vegeta test) and what to look out for. This was the slowest part of the process because we needed to understand that something non-application-level was going on (and because “random network latency” is a very difficult thing to narrow down) - it probably took on the order of months from the first sign of an issue to fixing all the other issues that were contributing to small amounts of latency and being sure we still had an underlying problem to find. At that point it became clear that something more low level was going on, we put together a focus team from a selection of teams to investigate the underlying cause - that was a group of about 5 engineers actively working on it, with another 5-10 interested engineers following along and helping out. Folks were typically working in pairs or solo to dive in to different potential leads, looping in everyone else in Slack as they go. Most of the work here was finding signal in the noise, we found a lot of other smaller system-level issues along the way that got ruled out and/or low priority to fix. There were other DNS related issues at play, fixing those also improved things, but not the specific underlying issue in the post here. Going down the specific path in the post took just a few days once the first few steps showed something was wrong at the packet level. The remediation from there was also just a few days, because we already had infrastructure in place to detect a known issue and mitigate in a safe/graceful way. The focus team was working on this as a primary task for around a few weeks overall.

bigbluedots 7y ago

I learned a lot reading this writeup, thanks for posting!

Polyisoprene 7y ago

Inefficient Linux cadvisor implementation causes unexpected latency. Also this affects buzzword compatible technologies.

nielsole 7y ago
What makes you say it was a cadvisor issue? Isn't it clearly a kernel issue?
- alecco 7y ago
  It looks more like Docker is pushing the Linux kernel in unprecedented ways. You can't blame Linux for having a cache there for (so-far) normal workloads. And cadvisor was pushing the problem further by reading constantly stats.

hogetesco 7y ago

I read the article with great fun. The question is, how do I run the bcc script in the article? Is it on a sidecar container or on a worker node machine?

theojulienne 7y ago
The bcc script was run on the Kubernetes node itself directly over SSH, but it should be possible to run it in a privileged container as well.

suresk 7y ago

Super interesting, thanks for sharing! These things are always my favorite to deal with - the feeling at the end when you figure it out is amazing, and you usually learn a ton on your way there, too.

maxpert 7y ago

I keep seeing articles showing up on debugging network issues like DNS or something else. Makes me wonder how much engineering effort we are putting in to fight the tool. Have these people considered/investigated other tools like Nomad (hashi corp)? How much value Kube actually adds vs these issue?

sytelus 7y ago
Kubernetes is CORBA of this generation. It will float around because of few heavyweights preaching it and suckers falling for it. Eventually it will die like all overly-complex, hideously-designed and poorly-implemented things do. You can save your time by just ignoring this nonsense.
- loopz 7y ago
  Remembering CORBA and glad it never took off: What if Kubernetes is the java of this generation? Java is not yet dying, and for better or worse, do solve organizational problems around code. Kubernetes is likewise becoming a business standard. Look into VMWare, IBM/RedHat and remote service providers offering it as standard. One can write huge books on java, both on the strengths and solutions, but also the weaknesses. For smaller companies and projects, it would probably be overkill to start with the java/JVM monolith, even today. Though java is the reason I jumped off the dev bandwagon, I can defend the language in hindsight, because it do solve several huge "unnecessary" problems other languages/runtimes simply don't address. The operational costs having been hidden away with expensive hardware upgrades. In hindsight, I don't see huge obstacles that the same may be said about Kubernetes in the future. These low-level kinks will be worked out, and provide for the masses what they can't or won't achieve alone. The masses don't need Google-scale, but if they can get the 15-year old mature distributed systems tech almost for free, there are several reasons to standardize.
- nickthemagicman 7y ago
  I think React.js is the CORBA of this generation. k8's might be too but I think the fact that containers are so portable it will be easy to migrate your application away from k8s if you want and to a new infrastructure. That's kind of the beauty of it.

KaiserPro 7y ago

What I am struggling to understand is why is it a good idea to to ip inside ip. Giving containers their own nic, and IP isn't hard, and in most cases is >> faster than spending time using some immature tunnelling protocol. I can see that doing multi-cloud deploys with transparent failover might benefit from a VPN, but short of operating on a hostile network I can't see its worth the heartache to deploy an overlay network.

navaati 7y ago
This a thousand ! You don't even need to give them actual nics, just veth pairs with routing (some CNI plugons do just that btw). I think a big part of why overlay nets are so common is that a lot of the engineers deploying these systems have a shaky or non existant understanding of networking and routing, they are stuck in the "local network and a default route" model and if you only have that, overlay it is...
- silasb 7y ago
  Do you know what CNI plugins support veth pairs?
  - boulos 7y ago
    The built-in bridge one does. Maybe you meant something else by your question though?
  - CameronNemo 7y ago
    https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md https://github.com/containernetworking/plugins/blob/master/p...
  - elFarto 7y ago
    I've tried using Bridget[1] before, but didn't actually manage to get it working properly (although the problem was likely firewall related). [1] https://github.com/kvaps/bridget https://github.com/kvaps/bridget
  - 7y ago

acd 7y ago

There has been issues with Kubernetes and NAT reported from Xing engineering. 12 min into the video https://www.youtube.com/watch?v=MoIdU0J0f0E https://www.youtube.com/watch?v=MoIdU0J0f0E

theojulienne 7y ago
This insert_failed issue described in the video was one of the ones we discovered during investigations as well, but it was already well understood because of this excellent Xing blog post which was extremely useful and referenced internally a lot: https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02 https://tech.xing.com/a-reason-for-unexplained-connection-ti...

je42 7y ago

Awesome deep dive ! I wonder how one can debug this without deep insights into the networking stack. I hope i ll never have a problem like this myself ;)

joemag 7y ago

Is the tunneling protocol really IPIP for these overlays? Oh boy - that’s going to really suck on wide multi path networks like the ones used by cloud providers.