5 ms·

Debugging a Linux network stack crash via a single register value

299 points by jgrahamc 5y ago

uvdn7 5y ago
Brilliant!! What's really cool is that Jakub approached the crash systemically. There are hard bugs in life e.g. cache inconsistencies, kernel bugs. What' more important than fixing them with one-off solution is to come up with a systemic approach to them. Great job! Cloudflare is indeed a really cool place.
stuff4ben 5y ago
Wow, that was a journey and very well written technical article! I didn't understand a good half of it since it's been 20 years since my single class in assembly language. But it makes me feel good that people like this exist!
- kingcharles 5y ago
  This was my thought too. I can follow the x86 and C code, but the networking and kernel debugging was way above my pay grade. Thank FSM for cleverer people than myself. Another case of feeling like an imposter...
Agingcoder 5y ago
Brilliant article, thanks for posting. This is typically the kind of post which makes me want to actually apply, mostly because I can relate to it : it's not magical, doesn't require gazillions of tpus, or petabytes of storage : it's plain old, excellent engineering. I wonder how often these issues creep up in practice, and how long it took the author to sort it out though! I've had my share of compiler bugs/kernel bugs, and they're usually quite expensive to understand, and it takes a long time to convince yourself that the bug is not in your code (granted, there's an oops here!)
- babelfish 5y ago
  https://cloudflare.com/careers https://cloudflare.com/careers :)
abainbridge 5y ago
Impressive bug hunting skills. When was this bug introduced? Does anyone maintain a list of when all the known Linux kernel bugs were introduced? I'd love to know how many bugs are added to the kernel each year, and if the rate is changing. I'm not trying to troll. I think the Linux kernel is an amazing piece of software engineering. I just think this would be an interesting metric.
- LukeShu 5y ago
  The kernel folks do a pretty good job of keeping track of which past commits a new commit "fixes", which they put in the commit message. For example, the patch linked in the article says: Fixes: bf296b125b21 ("tcp: Add GRO support") Fixes: f993bc25e519 ("net: core: handle encapsulation offloads when computing segment lengths") Fixes: e20cf8d3f1f7 ("udp: implement GRO for plain UDP sockets.") That said, using this to track how many bugs are introduced each year is problematic. It's often the case that commit A introduces a bug, commit B aims to fix it and says "Fixes: A" but turns out to only be a partial fix, and then commit C completes the fix and says "Fixes: B". Naively, based on the se annotations it would make sense to say "B introduced a bug", but as in my example, this isn't always the case. Greg KH discusses this in his talk "CVEs are dead" (video: https://www.youtube.com/watch?v=HeeoTE9jLjM https://www.youtube.com/watch?v=HeeoTE9jLjM slides: https://github.com/gregkh/presentation-cve-is-dead/blob/master/cve-linux-kernel.pdf https://github.com/gregkh/presentation-cve-is-dead/blob/mast... ).
  - sbierwagen 5y ago
    For the curious, bf296b125b21 was from 2008, f993bc25e519 was from 2014 and e20cf8d3f1f7 was in 2018.
Hikikomori 5y ago
Reminds me of https://quickview.cloudapps.cisco.com/quickview/bug/CSCso05336 https://quickview.cloudapps.cisco.com/quickview/bug/CSCso053...
megous 5y ago
"One register value", and the article starts with a full stack dump even with source code references available. :)
- jerjerjer 5y ago
  Yes, excellent article but clickbaity title. Honestly, all memory access errors eventually boil down to a "one register value" and many other error types probably too. I mean, what else is there if we go down far enough, really?
- AlphENsign_Tech 5y ago
  exactly kind of articles we need FTW
deleted 5y ago
[deleted]
diskzero 5y ago
What a fun read! Was the bug fixed as as a result of this investigation or was the fix already in a patch that these machines didn't have. I wasn't able to figure that out by reading the article.
- azinman2 5y ago
  If you click the link to the patch you see it was submitted by cloudflare.
- xbar 5y ago
  It was fixed as a result of the article. Jakub opens with "About a year ago...." and closes with a link to the fix thread where he gets it accepted into the kernel back in July 2021. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d51c5907e9809a803b276883d203f45849abd4d6 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
abridgett 5y ago
Superb article: - shows how to debug a kernel oops - show use of extra tools (bpf, scapy, kasan) - dives deep into more esoteric bits of networking explaining from basic to how the kernel implementation works - demonstrates proof of the theory in multiple ways Yes, it is a big "come work here on interesting stuff with fantastic people" (and show off your own skills and learn something neat) - and it's done without bragging and I'm sure will help others to debug their next OOPS :) Kudos to Jakub (and presumably reviewers)
holonomically 5y ago
Microsoft is working on a project to formalize various parts of the web stack and it would be interesting if their work also carried over to the lower parts of the networking stack like in this article. [1] I suspect this bug would have been caught if the segment handling logic was implemented in a language with a formal specification for the segment headers. Is cloudflare working on any formalization efforts like Microsoft's Project Everest? 1: https://www.microsoft.com/en-us/research/project/project-everest-verified-secure-implementations-https-ecosystem/ https://www.microsoft.com/en-us/research/project/project-eve...
ndesaulniers 5y ago
I had a similar experience recently when we were trying to get AMDGPU Linux kernel drivers to run without panic'ing. ./scripts/decodecode is what produces the disassembly of the code trace from the panic. (Seeing its output converted in Intel syntax in this post is...heresy) For AMDGPU, the issue was that the x86 Linux kernel doesn't use 16B stack alignment (it uses 8B stack alignment), yet just the AMDGPU driver was forcing the stack alignment for itself back to 16B. The AMDGPU driver uses SSE2 instructions that require 16B stack alignment. From the trace, seeing RSP as a multiple of 8 and not 16 was the smoking gun (ie. a single register). The fix was to use the same stack alignment (8B) consistently in the driver when using sse2 (except for old GCC versions), see this series: https://lore.kernel.org/lkml/20191016230209.39663-2-ndesaulniers@google.com/ https://lore.kernel.org/lkml/20191016230209.39663-2-ndesauln.... Stack alignment has obvious ABI implications.
quotemstr 5y ago
Nobody should have to decode register values, look up source files versions, or reconstruct a stack trace by hand. Crashes, both in user- and kernel-space, should produce neat, tidy, self-contained dumps that include not only the entire machine state, but globally unique build IDs for all binaries involved in the crash. And the debugger ought to be able to load one of these crash dumps and find the debug symbols and source files automatically. Windows has been able to do this for decades. Why, in Unix-land, are we still reading text reports about crashes and puzzling over specific register values?
- bitcharmer 5y ago
  How on earth are you going to map a particular spot in the code without access to the sources? I'd love you to explain how hunting kernel bugs is easier on windows.
  - quotemstr 5y ago
    Microsoft publishes symbols that you can use to decode kernel crashes. The availability of source code isn't my point though: the point is that parsing text debug dumps is stone age crap and we should move to a model where crashes produce self-describing crash dumps that users can load into debuggers that find all relevant metadata automatically. You shouldn't have to puzzle over the meaning of each register in the x86_64 to figure out what value a function parameter had at crash time
- drewg123 5y ago
  I don't know why this is downvoted. A crashdump would have made this 10x easier to debug. The answer is because "its linux" and crashdumps (except on a few distros) are not part of the culture. These days linux actually kexec's another kernel with reserved physical memory to handle doing the dumps. This was absurdly hard to setup the last time I needed to do linux kernel debugging (years ago). The linux kernel is generally compiled without frame pointers, and with other optimizations that are hostile to debugging. We have kernel and userspace crashdumps enabled on our fleet of FreeBSD CDN servers, and having a real crashdump and not just a stack has been super helpful.
p2p_astroturf 5y ago
Oh you think Google is cool? Bro CloudFlare hackers can debug a kernel panic through just a single register. I'm going to install CloudFlare on my website because everyone's doing it!

299 points by jgrahamc 5y ago

Brilliant!! What's really cool is that Jakub approached the crash systemically. There are hard bugs in life e.g. cache inconsistencies, kernel bugs. What' more important than fixing them with one-off solution is to come up with a systemic approach to them. Great job! Cloudflare is indeed a really cool place.

stuff4ben 5y ago

Wow, that was a journey and very well written technical article! I didn't understand a good half of it since it's been 20 years since my single class in assembly language. But it makes me feel good that people like this exist!

kingcharles 5y ago
This was my thought too. I can follow the x86 and C code, but the networking and kernel debugging was way above my pay grade. Thank FSM for cleverer people than myself. Another case of feeling like an imposter...

Agingcoder 5y ago

Brilliant article, thanks for posting. This is typically the kind of post which makes me want to actually apply, mostly because I can relate to it : it's not magical, doesn't require gazillions of tpus, or petabytes of storage : it's plain old, excellent engineering. I wonder how often these issues creep up in practice, and how long it took the author to sort it out though! I've had my share of compiler bugs/kernel bugs, and they're usually quite expensive to understand, and it takes a long time to convince yourself that the bug is not in your code (granted, there's an oops here!)

babelfish 5y ago
https://cloudflare.com/careers https://cloudflare.com/careers :)

abainbridge 5y ago

Impressive bug hunting skills. When was this bug introduced? Does anyone maintain a list of when all the known Linux kernel bugs were introduced? I'd love to know how many bugs are added to the kernel each year, and if the rate is changing. I'm not trying to troll. I think the Linux kernel is an amazing piece of software engineering. I just think this would be an interesting metric.

LukeShu 5y ago
The kernel folks do a pretty good job of keeping track of which past commits a new commit "fixes", which they put in the commit message. For example, the patch linked in the article says: Fixes: bf296b125b21 ("tcp: Add GRO support") Fixes: f993bc25e519 ("net: core: handle encapsulation offloads when computing segment lengths") Fixes: e20cf8d3f1f7 ("udp: implement GRO for plain UDP sockets.") That said, using this to track how many bugs are introduced each year is problematic. It's often the case that commit A introduces a bug, commit B aims to fix it and says "Fixes: A" but turns out to only be a partial fix, and then commit C completes the fix and says "Fixes: B". Naively, based on the se annotations it would make sense to say "B introduced a bug", but as in my example, this isn't always the case. Greg KH discusses this in his talk "CVEs are dead" (video: https://www.youtube.com/watch?v=HeeoTE9jLjM https://www.youtube.com/watch?v=HeeoTE9jLjM slides: https://github.com/gregkh/presentation-cve-is-dead/blob/master/cve-linux-kernel.pdf https://github.com/gregkh/presentation-cve-is-dead/blob/mast... ).
- sbierwagen 5y ago
  For the curious, bf296b125b21 was from 2008, f993bc25e519 was from 2014 and e20cf8d3f1f7 was in 2018.

Hikikomori 5y ago

Reminds me of https://quickview.cloudapps.cisco.com/quickview/bug/CSCso05336 https://quickview.cloudapps.cisco.com/quickview/bug/CSCso053...

megous 5y ago

"One register value", and the article starts with a full stack dump even with source code references available. :)

jerjerjer 5y ago
Yes, excellent article but clickbaity title. Honestly, all memory access errors eventually boil down to a "one register value" and many other error types probably too. I mean, what else is there if we go down far enough, really?
AlphENsign_Tech 5y ago
exactly kind of articles we need FTW

deleted 5y ago

[deleted]

diskzero 5y ago

What a fun read! Was the bug fixed as as a result of this investigation or was the fix already in a patch that these machines didn't have. I wasn't able to figure that out by reading the article.

azinman2 5y ago
If you click the link to the patch you see it was submitted by cloudflare.
xbar 5y ago
It was fixed as a result of the article. Jakub opens with "About a year ago...." and closes with a link to the fix thread where he gets it accepted into the kernel back in July 2021. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d51c5907e9809a803b276883d203f45849abd4d6 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

abridgett 5y ago

Superb article: - shows how to debug a kernel oops - show use of extra tools (bpf, scapy, kasan) - dives deep into more esoteric bits of networking explaining from basic to how the kernel implementation works - demonstrates proof of the theory in multiple ways Yes, it is a big "come work here on interesting stuff with fantastic people" (and show off your own skills and learn something neat) - and it's done without bragging and I'm sure will help others to debug their next OOPS :) Kudos to Jakub (and presumably reviewers)

holonomically 5y ago

Microsoft is working on a project to formalize various parts of the web stack and it would be interesting if their work also carried over to the lower parts of the networking stack like in this article. [1] I suspect this bug would have been caught if the segment handling logic was implemented in a language with a formal specification for the segment headers. Is cloudflare working on any formalization efforts like Microsoft's Project Everest? 1: https://www.microsoft.com/en-us/research/project/project-everest-verified-secure-implementations-https-ecosystem/ https://www.microsoft.com/en-us/research/project/project-eve...

ndesaulniers 5y ago

I had a similar experience recently when we were trying to get AMDGPU Linux kernel drivers to run without panic'ing. ./scripts/decodecode is what produces the disassembly of the code trace from the panic. (Seeing its output converted in Intel syntax in this post is...heresy) For AMDGPU, the issue was that the x86 Linux kernel doesn't use 16B stack alignment (it uses 8B stack alignment), yet just the AMDGPU driver was forcing the stack alignment for itself back to 16B. The AMDGPU driver uses SSE2 instructions that require 16B stack alignment. From the trace, seeing RSP as a multiple of 8 and not 16 was the smoking gun (ie. a single register). The fix was to use the same stack alignment (8B) consistently in the driver when using sse2 (except for old GCC versions), see this series: https://lore.kernel.org/lkml/20191016230209.39663-2-ndesaulniers@google.com/ https://lore.kernel.org/lkml/20191016230209.39663-2-ndesauln.... Stack alignment has obvious ABI implications.

quotemstr 5y ago

Nobody should have to decode register values, look up source files versions, or reconstruct a stack trace by hand. Crashes, both in user- and kernel-space, should produce neat, tidy, self-contained dumps that include not only the entire machine state, but globally unique build IDs for all binaries involved in the crash. And the debugger ought to be able to load one of these crash dumps and find the debug symbols and source files automatically. Windows has been able to do this for decades. Why, in Unix-land, are we still reading text reports about crashes and puzzling over specific register values?

bitcharmer 5y ago
How on earth are you going to map a particular spot in the code without access to the sources? I'd love you to explain how hunting kernel bugs is easier on windows.
- quotemstr 5y ago
  Microsoft publishes symbols that you can use to decode kernel crashes. The availability of source code isn't my point though: the point is that parsing text debug dumps is stone age crap and we should move to a model where crashes produce self-describing crash dumps that users can load into debuggers that find all relevant metadata automatically. You shouldn't have to puzzle over the meaning of each register in the x86_64 to figure out what value a function parameter had at crash time
drewg123 5y ago
I don't know why this is downvoted. A crashdump would have made this 10x easier to debug. The answer is because "its linux" and crashdumps (except on a few distros) are not part of the culture. These days linux actually kexec's another kernel with reserved physical memory to handle doing the dumps. This was absurdly hard to setup the last time I needed to do linux kernel debugging (years ago). The linux kernel is generally compiled without frame pointers, and with other optimizations that are hostile to debugging. We have kernel and userspace crashdumps enabled on our fleet of FreeBSD CDN servers, and having a real crashdump and not just a stack has been super helpful.

p2p_astroturf 5y ago

Oh you think Google is cool? Bro CloudFlare hackers can debug a kernel panic through just a single register. I'm going to install CloudFlare on my website because everyone's doing it!