4 ms·

Kubernetes and Kernel Panics

113 points by zdw 3y ago

birdiesanders 3y ago
This is a really creative solution to orphaned pods.
speedgoose 3y ago
Does the AWS EC2 network filters UDP packets with a spoofed source? Is the firewall well configured? Because it sounds like something spamming UDP packets with IP spoofing could take down all kubernetes nodes. Maybe signing the kernel panic message could help too.
- vlovich123 3y ago
  > Luckily the netconsole-setup command makes the setup pretty easy. All the configuration options can be set dynamically as well, so that when the endpoint changes one can point to the new IP. There’s no spoofing happening here…
  - wrs 3y ago
    Yes, but what if there was? A safer alternative would be to simply do the annotation and leave k8s to do the deletes on timeout as usual.
  - dafelst 3y ago
    I think OP is highlighting that spamming spoofed UDP packets to this service is an attack vector for a denial of service attack.
    - dastbe 3y ago
      in that case, yes aws ec2 (and other cloud providers) disable ip spoofing by default for VMs in your VPC.
      - speedgoose 3y ago
        Thanks. That’s a good feature from AWS.
- alexeldeib 3y ago
  From the description, I’d think this is all in cluster, not public.
- nonameiguess 3y ago
  If you mean traffic from outside of the cluster, there is no reason for their cluster nodes to have public IP addresses, so they wouldn't even be routable from outside the VPC. I would expect this controller to run on the control plane and probably not be Internet-accessible by any means, but even if it were, I'm sure the security group only allows UDP port 6666 from inside the cluster, presumably based on source security group, not IP, so even if you could somehow spoof a private IP address and get it into the VPC, it would still not be accepted. If you mean malicious traffic coming from their own pods, they could simply add Kubernetes network policies not allowing UDP port 6666 egress from any pod since the traffic they're actually hoping for would always come from a kernel, not a pod, and thus egress from a different interface. So now you're left with the remaining attack vectors being spoofed IPs, or potentially even the correct IP but a spoofed packet, coming directly from EC2 instances in the same VPC, either by launching separate instances or compromising clusters nodes at the OS level. An insider with that level of access is always a threat pretty much no matter what you do because they could just delete your nodes through the EC2 service. You mitigate that with thorough audit logging, no shared or long-lived credentials, deep vetting of a very small number of personnel you give that level of access to, and the threat of near-certain jail time if someone does it on purpose anyway.
throwaway892238 3y ago
looks like doesn't support 802.1Q?
- geraldhh 3y ago
  how come?
MarkSweep 3y ago
Is Netflix still using their Titus container platform? It’s mentioned at the start of the post but the rest of the post seems to be able Kubernetes. Also the Titus repos on GitHub were archived a year ago. https://github.com/Netflix/titus https://github.com/Netflix/titus
- dilyevsky 3y ago
  Titus used to be a framework on top of mesos. Guess it's now a framework on top of kubernetes
  - btown 3y ago
    Yep: https://netflixtechblog.com/consistent-caching-mechanism-in-titus-gateway-6cb89b9ce296 https://netflixtechblog.com/consistent-caching-mechanism-in-...
- dirtyhand 3y ago
  Titus is a federation layer that sits on top of many Kubernetes clusters
Elucalidavah 3y ago
Shouldn't ec2 kernel panic be handled on the ec2-level instead? It's not like they're running hardware nodes, after all. And surely kernel panics on k8 nodes shouldn't really be a thing? Are they loading custom kernel modules at random times in there?
- dboreham 3y ago
  EC2 it appears does not do this (capture a kernel panic reason and put it somewhere that an orchestration system can find it later). Kernel panics are a thing. Whether a kernel is hosting k8s processes seems not terribly relevant to the probability it will panic? k8s seems neither likely to induce panics more often, nor to reduce their probability, imho. This is probably a case of "at scale, everything happens", including kernel panics in VMs that most people assume never panic.
pstuart 3y ago
This is great info, and looks to be usable even outside of k8s.
theptip 3y ago
Curious why this isn’t published to Pod events, seems like that would be the more natural place. They mentioned not wanting to patch Status but that too seems non-standard? For the problem of where to run this if this was upstreamed, it’s an interesting problem - seems like you could make the case for the kubelet node agent baking this behavior in and gossiping to neighbors when the node terminates.
- dilyevsky 3y ago
  Probably be way less work and more reliable to emit node event when it comes back and scan for that. Seems like a case of google cargo culting. Remember kids, Google runs real machines and you probably dont
jeffrallen 3y ago
Tl;Dr: use netconsole to send console messages via UDP to the k8s controller, where they are scanned for panics.
Yasuraka 3y ago
I wish the article had some examples for things that caused a panic, I didn't hear of let alone have one in half a decade
- jeffbee 3y ago
  Kernel panics are more common in Kubernetes because the container memory limitations mean the kernel is more often trying to reclaim memory, or failing to allocate, and that exercises little-tested error paths that are full of bugs.
- dilyevsky 3y ago
  Happens all the time especially on bleeding edge kernels. 4.1x was especially bad. Usually driver problems but sometimes other bugs too
__turbobrew__ 3y ago
Interesting to learn about last gasp packets, seems like a useful mechanism for any system. I generally find that observability into kubernetes is not that great especially for analytical queries. Answering the question of “where where my pods running for a replicaset at time X” is surprisingly not easy to answer with just events. Also, events are designed to have a short ttl so you need to extract them out of etcd and into some sort of long term storage and indexed in a way which you can run useful queries. At larger scales kube-state-metrics will blow up about any prometheus server outside of a full blown billion metric stream scale mimir deployment.
- smoothwater8912 3y ago
  Sloop is pretty good for helping you understand events over time. It has an embedded database so you need to be careful with resources (apparently it's possible to split separately if required). Filtering out extraneous events can help minimise resource utilisation - it's amazing how many events a real-world cluster can generate running things like Kyverno, Flux etc. https://github.com/salesforce/sloop https://github.com/salesforce/sloop
10000truths 3y ago
I'm not sure I understand the purpose of using netconsole in virtual machines. The serial console is easier to use, has fewer moving parts, and doesn't rely on a functional network stack.

113 points by zdw 3y ago

This is a really creative solution to orphaned pods.

Does the AWS EC2 network filters UDP packets with a spoofed source? Is the firewall well configured? Because it sounds like something spamming UDP packets with IP spoofing could take down all kubernetes nodes. Maybe signing the kernel panic message could help too.

vlovich123 3y ago
> Luckily the netconsole-setup command makes the setup pretty easy. All the configuration options can be set dynamically as well, so that when the endpoint changes one can point to the new IP. There’s no spoofing happening here…
- wrs 3y ago
  Yes, but what if there was? A safer alternative would be to simply do the annotation and leave k8s to do the deletes on timeout as usual.
- dafelst 3y ago
  I think OP is highlighting that spamming spoofed UDP packets to this service is an attack vector for a denial of service attack.
  - dastbe 3y ago
    in that case, yes aws ec2 (and other cloud providers) disable ip spoofing by default for VMs in your VPC.
    - speedgoose 3y ago
      Thanks. That’s a good feature from AWS.
alexeldeib 3y ago
From the description, I’d think this is all in cluster, not public.
nonameiguess 3y ago
If you mean traffic from outside of the cluster, there is no reason for their cluster nodes to have public IP addresses, so they wouldn't even be routable from outside the VPC. I would expect this controller to run on the control plane and probably not be Internet-accessible by any means, but even if it were, I'm sure the security group only allows UDP port 6666 from inside the cluster, presumably based on source security group, not IP, so even if you could somehow spoof a private IP address and get it into the VPC, it would still not be accepted. If you mean malicious traffic coming from their own pods, they could simply add Kubernetes network policies not allowing UDP port 6666 egress from any pod since the traffic they're actually hoping for would always come from a kernel, not a pod, and thus egress from a different interface. So now you're left with the remaining attack vectors being spoofed IPs, or potentially even the correct IP but a spoofed packet, coming directly from EC2 instances in the same VPC, either by launching separate instances or compromising clusters nodes at the OS level. An insider with that level of access is always a threat pretty much no matter what you do because they could just delete your nodes through the EC2 service. You mitigate that with thorough audit logging, no shared or long-lived credentials, deep vetting of a very small number of personnel you give that level of access to, and the threat of near-certain jail time if someone does it on purpose anyway.

throwaway892238 3y ago

looks like doesn't support 802.1Q?

geraldhh 3y ago
how come?

MarkSweep 3y ago

Is Netflix still using their Titus container platform? It’s mentioned at the start of the post but the rest of the post seems to be able Kubernetes. Also the Titus repos on GitHub were archived a year ago. https://github.com/Netflix/titus https://github.com/Netflix/titus

dilyevsky 3y ago
Titus used to be a framework on top of mesos. Guess it's now a framework on top of kubernetes
- btown 3y ago
  Yep: https://netflixtechblog.com/consistent-caching-mechanism-in-titus-gateway-6cb89b9ce296 https://netflixtechblog.com/consistent-caching-mechanism-in-...
dirtyhand 3y ago
Titus is a federation layer that sits on top of many Kubernetes clusters

Elucalidavah 3y ago

Shouldn't ec2 kernel panic be handled on the ec2-level instead? It's not like they're running hardware nodes, after all. And surely kernel panics on k8 nodes shouldn't really be a thing? Are they loading custom kernel modules at random times in there?

dboreham 3y ago
EC2 it appears does not do this (capture a kernel panic reason and put it somewhere that an orchestration system can find it later). Kernel panics are a thing. Whether a kernel is hosting k8s processes seems not terribly relevant to the probability it will panic? k8s seems neither likely to induce panics more often, nor to reduce their probability, imho. This is probably a case of "at scale, everything happens", including kernel panics in VMs that most people assume never panic.

pstuart 3y ago

This is great info, and looks to be usable even outside of k8s.

theptip 3y ago

Curious why this isn’t published to Pod events, seems like that would be the more natural place. They mentioned not wanting to patch Status but that too seems non-standard? For the problem of where to run this if this was upstreamed, it’s an interesting problem - seems like you could make the case for the kubelet node agent baking this behavior in and gossiping to neighbors when the node terminates.

dilyevsky 3y ago
Probably be way less work and more reliable to emit node event when it comes back and scan for that. Seems like a case of google cargo culting. Remember kids, Google runs real machines and you probably dont

jeffrallen 3y ago

Tl;Dr: use netconsole to send console messages via UDP to the k8s controller, where they are scanned for panics.

Yasuraka 3y ago

I wish the article had some examples for things that caused a panic, I didn't hear of let alone have one in half a decade

jeffbee 3y ago
Kernel panics are more common in Kubernetes because the container memory limitations mean the kernel is more often trying to reclaim memory, or failing to allocate, and that exercises little-tested error paths that are full of bugs.
dilyevsky 3y ago
Happens all the time especially on bleeding edge kernels. 4.1x was especially bad. Usually driver problems but sometimes other bugs too

__turbobrew__ 3y ago

Interesting to learn about last gasp packets, seems like a useful mechanism for any system. I generally find that observability into kubernetes is not that great especially for analytical queries. Answering the question of “where where my pods running for a replicaset at time X” is surprisingly not easy to answer with just events. Also, events are designed to have a short ttl so you need to extract them out of etcd and into some sort of long term storage and indexed in a way which you can run useful queries. At larger scales kube-state-metrics will blow up about any prometheus server outside of a full blown billion metric stream scale mimir deployment.

smoothwater8912 3y ago
Sloop is pretty good for helping you understand events over time. It has an embedded database so you need to be careful with resources (apparently it's possible to split separately if required). Filtering out extraneous events can help minimise resource utilisation - it's amazing how many events a real-world cluster can generate running things like Kyverno, Flux etc. https://github.com/salesforce/sloop https://github.com/salesforce/sloop

10000truths 3y ago

I'm not sure I understand the purpose of using netconsole in virtual machines. The serial console is easier to use, has fewer moving parts, and doesn't rely on a functional network stack.