6 ms·

Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)

206 points by goranmoomin 2y ago

mseepgood 2y ago
(2017) Previous discussion: https://news.ycombinator.com/item?id=15845118 https://news.ycombinator.com/item?id=15845118
- rollulus 2y ago
  Ah, I remember 2017’s HN where the mere word “Go” in the title made posts skyrocket to 498 points. In 2024 the title should’ve contained “Rust” for the same outcome. Quite curious what will be required in 2031.
  - aapoalas 2y ago
    Maybe Zig or Carbon? Just spitballing, personally it's of course Rust forever.
  - keybored 2y ago
    Rust has always had a fair number of supporters on HN. Certainly since 2015.
  - freep1zza 2y ago
    Try them you'll love it :)
BobbyJo 2y ago
This is honestly wild. 99% of devs would have found a work around and moved on. Going so far as to create a multi-kernel test bench to narrow down the source of the instability is a level of dedication I have not personally seen, and I respect it.
- randomdata 2y ago
  By the same token, you might be the first person I have ever seen give respect to the 1x developer. I respect that. We could no doubt all learn a thing or two from the 1x developer that doesn't rush through everything with quick solutions.
  - sanbor 2y ago
    If you check the issue[1] he reported the crash on November 7th and reported the issue is related to gcc and the kernel on November 8th. At least he was very quick going through the rabbit hole. [1]: https://github.com/prometheus/node_exporter/issues/730 https://github.com/prometheus/node_exporter/issues/730
    - randomdata 2y ago
      [flagged]
      - prerok 2y ago
        Emm, 10x developer is a myth. The real 10x developer is the one that creates such an infrastructure/libraries/culture that enables 10 other engineers to move fast. The 10x developer is not the person that is 10x faster than other developers on the code base simply because they can hold the spaghetti code they wrote in their head.
      - deleted 2y ago
        [deleted]
      - gouggoug 2y ago
        > means that a 1x developer will take around 10 hours to get to the same place A "workaround" isn't an adequate substitute for actual the understanding and fixing of the root cause of a bug. What you think is a 10x developer is, in fact, a short-term 10x developer, medium-term 1x developer, long-term -10x developer. Their work while seemingly great at first is just accrued debt with an incredibly high interest rate. But they're rarely the ones fixing it. Now, like everything, a balance needs to be struck between spending hours fixing a bug _the right way_, or, finding a temporary workaround. The real 10x developer is incredibly good at finding this balance.
- abbbi 2y ago
  Marcan is an beast, this guy really loves to go down the rabbit holes.
- toast0 2y ago
  Problems like this tend to come back and haunt you though. Sure, you can set max threads to 1 and move on with what you're doing for a while... but a lot of people run Go so they can have a lot more than 1 thread. I've run into some of these where it's a lot more rare to hit, and so then it's reasonable to not do the thing that hurts, but watch out for it in the future. Sometimes you get lucky and it magically fixes itself forever; sometimes the weird case that you only hit with internal traffic ends up getting hit by public tratfic a lot. Crashes like this where a wild write breaks something at a distance are always a PITA to debug (especially here, where the wild write is harmless if there's no data race)
im3w1l 2y ago
I feel like I still didnt fully understand what's going on here. Is the following correct? "Threads hava a 'canonical' stack that the OS auto-grows for you as you use more of it. But you can also create your own stack by putting any value you want in RSP. This is what the Go program did, and the vDSO, assuming it ran on an auto-growing stack, tried to probe it, which lead to corruption."
- derefr 2y ago
  I believe that Golang, as a green-threaded runtime, is allocating a separate carrier thread for running syscalls on, so that a blocking syscall won’t block the green-threads. These syscall carrier threads are allocated with a distinct initial stack size + stack size limit than green-thread-scheduler carrier threads are, because it’s expected that syscalls will always just enter kernel code (which has its own stack.) But vDSOs don’t enter the kernel. They just run as userland code; and so they depend on the userland allocated stack to be arbitrarily deep in a way that kernel calls don’t. As shown in the article, Golang seems to have code specifically for dealing with vDSO-type pseudo-syscalls — but this is likely a specialization of the pre-existing syscall-carrier-thread allocation code, and so started off with a bad assumption about how much stack should be allocated for the created threads. (I should also point out that the OS stack size specified in the ELF executable headers, only guarantees the stack size of the initial thread of a process created by exec(2). All further threads get their stacks allocated explicitly in userland code by libpthreads or the like calling malloc(2). Normally these abstractions just reuse the same config params from the executable (unless you override them, using e.g. pthreads_attr_setstacksize). But, as the article says, Golang implements its own support for things like this, and so can implement special thread-allocation strategies per carrier thread type.)
- jeffbee 2y ago
  Thread stacks in Linux are demand-paged: if you touch the next page then it magically exists, up to a limit. But the machine is not concerned with the convenient properties of this virtual memory area. To the CPU the register RSP is just an operand, expressed or implied, to some instructions.
- saagarjha 2y ago
wolf550e 2y ago
You can follow Hector Martin @marcan at https://social.treehouse.systems/@marcan/ https://social.treehouse.systems/@marcan/ He works on Asahi Linux, a Linux port to arm64 Apple hardware.
Thaxll 2y ago
One thing I learned from that post back then is that you can instruct Grub to ignore some part of your physical memory. Really nice trick, not sure this is doable on Windows / Mac?
- dwattttt 2y ago
  I was just looking at the bcdedit options earlier today, this is doable on Windows
Agingcoder 2y ago
This is very elegant. I’ve had my share of nasty system bugs ( compilers and kernels ) , but the dedication and the speed with which he went through it is quite remarkable. The explanations are also very clear. Thanks for posting.
ncruces 2y ago
Related (for the hash based bisecting): https://research.swtch.com/bisect https://research.swtch.com/bisect
deleted 2y ago
[deleted]
fulafel 2y ago
I guess Gentoo Hardened meant the difficulty level in this case.
umvi 2y ago
I'm pretty confident in my computer abilities, but when I read stuff like this I feel like I have no skills at all compared to this guy. Like I'm still a high school athlete and he's an Olympian (also he was only 26 when he wrote the article).
- switch007 2y ago
  > but when I read stuff like this I feel like I have no skills at all compared to this guy Comparison is the thief of joy!
Terr_ 2y ago
> Over the course of 22 kernel builds, I managed to simplify the config so much that the kernel had no networking support, no filesystems, no block device core, and didn’t even support PCI (still works fine on a VM though!). Flashbacks to a job where I was asked to figure out why a newer kernel was crashing. This was a very frustrating time, because I had (have) basically zero real C/C++ experience but I'd helped out with Bitbake recipes and everyone else was busy or moved to other projects. To cut a multiweek tale of dozens of recompilations short: The kernel was fine. The headless custom hardware was fine. The problem was a hypervisor misconfiguration, overwriting part of the kernel address space. All of our kernels have been corrupt, but this was the first one where the layout meant it mattered. A month of frustration, two characters to fix, the highest ratio I've encountered so far. My reward for struggling through a complex problem I was unqualified for? "Great, now we need to backport security patches from the main Linux kernel to the SoC vendor's custom fork..."
Hajcnga 2y ago
Could someone explain why Google is obsessed with small stack sizes? The musl library also has an extremely small thread stack size, which makes many applications crash that are used to 8192K on Linux. On Linux the 8192K aren't reserved unless the are actually used, so what is the point? Ok, Golang will allocate green threads from its own allocator, but 104 bytes?!
- neonsunset 2y ago
  I wonder if they might be favoring spawning numerous instances of an application with few threads and little memory use to scale at the replica level over just scaling within an application? At least Go defaults do suggest that (and are also the reason for very poor performance on large many-core nodes).
- samatman 2y ago
  I don't believe that Google has anything at all to do with musl libc. Applications should not be assuming a hard-coded stack size, that's a bug. They should be checking the value of PTHREAD_STACK_MIN. That is what it's there for.

206 points by goranmoomin 2y ago

mseepgood 2y ago

(2017) Previous discussion: https://news.ycombinator.com/item?id=15845118 https://news.ycombinator.com/item?id=15845118

rollulus 2y ago
Ah, I remember 2017’s HN where the mere word “Go” in the title made posts skyrocket to 498 points. In 2024 the title should’ve contained “Rust” for the same outcome. Quite curious what will be required in 2031.
- aapoalas 2y ago
  Maybe Zig or Carbon? Just spitballing, personally it's of course Rust forever.
- keybored 2y ago
  Rust has always had a fair number of supporters on HN. Certainly since 2015.
- freep1zza 2y ago
  Try them you'll love it :)

BobbyJo 2y ago

This is honestly wild. 99% of devs would have found a work around and moved on. Going so far as to create a multi-kernel test bench to narrow down the source of the instability is a level of dedication I have not personally seen, and I respect it.

randomdata 2y ago
By the same token, you might be the first person I have ever seen give respect to the 1x developer. I respect that. We could no doubt all learn a thing or two from the 1x developer that doesn't rush through everything with quick solutions.
- sanbor 2y ago
  If you check the issue[1] he reported the crash on November 7th and reported the issue is related to gcc and the kernel on November 8th. At least he was very quick going through the rabbit hole. [1]: https://github.com/prometheus/node_exporter/issues/730 https://github.com/prometheus/node_exporter/issues/730
  - randomdata 2y ago
    [flagged]
    - prerok 2y ago
      Emm, 10x developer is a myth. The real 10x developer is the one that creates such an infrastructure/libraries/culture that enables 10 other engineers to move fast. The 10x developer is not the person that is 10x faster than other developers on the code base simply because they can hold the spaghetti code they wrote in their head.
    - deleted 2y ago
      [deleted]
    - gouggoug 2y ago
      > means that a 1x developer will take around 10 hours to get to the same place A "workaround" isn't an adequate substitute for actual the understanding and fixing of the root cause of a bug. What you think is a 10x developer is, in fact, a short-term 10x developer, medium-term 1x developer, long-term -10x developer. Their work while seemingly great at first is just accrued debt with an incredibly high interest rate. But they're rarely the ones fixing it. Now, like everything, a balance needs to be struck between spending hours fixing a bug _the right way_, or, finding a temporary workaround. The real 10x developer is incredibly good at finding this balance.
abbbi 2y ago
Marcan is an beast, this guy really loves to go down the rabbit holes.
toast0 2y ago
Problems like this tend to come back and haunt you though. Sure, you can set max threads to 1 and move on with what you're doing for a while... but a lot of people run Go so they can have a lot more than 1 thread. I've run into some of these where it's a lot more rare to hit, and so then it's reasonable to not do the thing that hurts, but watch out for it in the future. Sometimes you get lucky and it magically fixes itself forever; sometimes the weird case that you only hit with internal traffic ends up getting hit by public tratfic a lot. Crashes like this where a wild write breaks something at a distance are always a PITA to debug (especially here, where the wild write is harmless if there's no data race)

im3w1l 2y ago

I feel like I still didnt fully understand what's going on here. Is the following correct? "Threads hava a 'canonical' stack that the OS auto-grows for you as you use more of it. But you can also create your own stack by putting any value you want in RSP. This is what the Go program did, and the vDSO, assuming it ran on an auto-growing stack, tried to probe it, which lead to corruption."

derefr 2y ago
I believe that Golang, as a green-threaded runtime, is allocating a separate carrier thread for running syscalls on, so that a blocking syscall won’t block the green-threads. These syscall carrier threads are allocated with a distinct initial stack size + stack size limit than green-thread-scheduler carrier threads are, because it’s expected that syscalls will always just enter kernel code (which has its own stack.) But vDSOs don’t enter the kernel. They just run as userland code; and so they depend on the userland allocated stack to be arbitrarily deep in a way that kernel calls don’t. As shown in the article, Golang seems to have code specifically for dealing with vDSO-type pseudo-syscalls — but this is likely a specialization of the pre-existing syscall-carrier-thread allocation code, and so started off with a bad assumption about how much stack should be allocated for the created threads. (I should also point out that the OS stack size specified in the ELF executable headers, only guarantees the stack size of the initial thread of a process created by exec(2). All further threads get their stacks allocated explicitly in userland code by libpthreads or the like calling malloc(2). Normally these abstractions just reuse the same config params from the executable (unless you override them, using e.g. pthreads_attr_setstacksize). But, as the article says, Golang implements its own support for things like this, and so can implement special thread-allocation strategies per carrier thread type.)
jeffbee 2y ago
Thread stacks in Linux are demand-paged: if you touch the next page then it magically exists, up to a limit. But the machine is not concerned with the convenient properties of this virtual memory area. To the CPU the register RSP is just an operand, expressed or implied, to some instructions.
saagarjha 2y ago

wolf550e 2y ago

You can follow Hector Martin @marcan at https://social.treehouse.systems/@marcan/ https://social.treehouse.systems/@marcan/ He works on Asahi Linux, a Linux port to arm64 Apple hardware.

Thaxll 2y ago

One thing I learned from that post back then is that you can instruct Grub to ignore some part of your physical memory. Really nice trick, not sure this is doable on Windows / Mac?

dwattttt 2y ago
I was just looking at the bcdedit options earlier today, this is doable on Windows

Agingcoder 2y ago

This is very elegant. I’ve had my share of nasty system bugs ( compilers and kernels ) , but the dedication and the speed with which he went through it is quite remarkable. The explanations are also very clear. Thanks for posting.

ncruces 2y ago

Related (for the hash based bisecting): https://research.swtch.com/bisect https://research.swtch.com/bisect

deleted 2y ago

[deleted]

fulafel 2y ago

I guess Gentoo Hardened meant the difficulty level in this case.

umvi 2y ago

I'm pretty confident in my computer abilities, but when I read stuff like this I feel like I have no skills at all compared to this guy. Like I'm still a high school athlete and he's an Olympian (also he was only 26 when he wrote the article).

switch007 2y ago
> but when I read stuff like this I feel like I have no skills at all compared to this guy Comparison is the thief of joy!

Terr_ 2y ago

> Over the course of 22 kernel builds, I managed to simplify the config so much that the kernel had no networking support, no filesystems, no block device core, and didn’t even support PCI (still works fine on a VM though!). Flashbacks to a job where I was asked to figure out why a newer kernel was crashing. This was a very frustrating time, because I had (have) basically zero real C/C++ experience but I'd helped out with Bitbake recipes and everyone else was busy or moved to other projects. To cut a multiweek tale of dozens of recompilations short: The kernel was fine. The headless custom hardware was fine. The problem was a hypervisor misconfiguration, overwriting part of the kernel address space. All of our kernels have been corrupt, but this was the first one where the layout meant it mattered. A month of frustration, two characters to fix, the highest ratio I've encountered so far. My reward for struggling through a complex problem I was unqualified for? "Great, now we need to backport security patches from the main Linux kernel to the SoC vendor's custom fork..."

Hajcnga 2y ago

Could someone explain why Google is obsessed with small stack sizes? The musl library also has an extremely small thread stack size, which makes many applications crash that are used to 8192K on Linux. On Linux the 8192K aren't reserved unless the are actually used, so what is the point? Ok, Golang will allocate green threads from its own allocator, but 104 bytes?!

neonsunset 2y ago
I wonder if they might be favoring spawning numerous instances of an application with few threads and little memory use to scale at the replica level over just scaling within an application? At least Go defaults do suggest that (and are also the reason for very poor performance on large many-core nodes).
samatman 2y ago
I don't believe that Google has anything at all to do with musl libc. Applications should not be assuming a hard-coded stack size, that's a bug. They should be checking the value of PTHREAD_STACK_MIN. That is what it's there for.