4 ms·
Debugging a FUSE deadlock in the Linux kernel
- nyanpasu64 4y agoI'm reminded of Bedrock Linux hanging during sleep because Linux sleeps the FUSE daemon while a process is waiting on FUSE and unable to be suspended for sleep: https://news.ycombinator.com/item?id=34583495 https://news.ycombinator.com/item?id=34583495
- ParadigmComplex 4y agoTo be clear, Bedrock Linux does not hang during sleep. Rather, it cannot enter sleep consistently; if it fails to, it simply continues on as it was operating before the request to suspend. The underlying issue is a long standing The Linux kernel bug [0] in which it cannot reliably suspend if FUSE is in use. When the kernel detects this scenario, it simply doesn't suspend, but rather logs its difficulty to dmesg. This is not specific to Bedrock, but hits all projects on Linux which leverage Linux's FUSE functionality [1]. It probably hits Netflix's ndrive as well. In the few months since my comment you've linked, I've put some work into a possible FUSE-less Bedrock implementation. It will likely have some downsides compared to a FUSE-based solution, but the trade-off may be worthwhile for some users. While it's too early to commit to this, I'm hoping to eventually support switching between a FUSE-mode and a non-FUSE-mode with a reboot to allow users to pick the desired trade-off in the desired contexts. [0] https://bugzilla.kernel.org/show_bug.cgi?id=34932 https://bugzilla.kernel.org/show_bug.cgi?id=34932 https://bugzilla.kernel.org/show_bug.cgi?id=198879 https://bugzilla.kernel.org/show_bug.cgi?id=198879 https://lists.debian.org/debian-kernel/2011/10/msg00412.html https://lists.debian.org/debian-kernel/2011/10/msg00412.html [1] https://github.com/keybase/client/issues/12458 https://github.com/keybase/client/issues/12458 https://github.com/libfuse/libfuse/issues/248 https://github.com/libfuse/libfuse/issues/248 https://bugs.launchpad.net/ubuntu/+source/sshfs-fuse/+bug/1702368 https://bugs.launchpad.net/ubuntu/+source/sshfs-fuse/+bug/17...
- loeg 4y agoThis article is mostly about rediscovering the distinction between interruptible and non-interruptible waits. At the end there is some sort of deadlock between namespace teardown, signal delivery, and FUSE, but... it isn't articulated in a way that is super comprehensible to me. The kernel flushes open files on exit and also kills things in the namespace on exit. But that means the race condition was always hitable if you killed the FUSE daemon at the wrong time relative to the FUSE client shutdown? It's not totally obvious to me why this would impact other non-FUSE filesystems. Signal delivery and multithreaded process teardown in the kernel is certainly tricky, and it's really easy to get these weird edge cases wrong.
- sargun 4y agoI don't understand why wants_signal returns false on PF_EXITING even if the signal is SIGKILL (and from the kernel). Shouldn't wake up the process still, so it can get out of the flush? I am curious, if you were just to walk over every PID in the pid namespace after sending zap_pid_ns_processes, and perform wake ups, would it break out of the `wait_event` loop? Btw, this class of weirdness with FUSE isn't that unusual.
- tych0 4y agoAuthor here (hi Sargun), it's not really about rediscovering killable vs. unkillable waits, and any confusion is probably a result of my poor writing. The crux of it is that once you've called exit_signals() from do_exit(), signals will not get delivered. So if you subsequently use the kernel's completions or other wait code, you will not get the signal from zap_pid_ns_processes(), so you don't know to wake up and exit. There's a test case here if people want to play around: https://github.com/tych0/kernel-utils/tree/master/fuse2 https://github.com/tych0/kernel-utils/tree/master/fuse2
- loeg 4y agoShould processes not be able to wait after exit_signals? That seems like a plausible invariant.
- ndesaulniers 4y agoI thought Netflix used a BSD? https://papers.freebsd.org/2019/fosdem/looney-netflix_and_freebsd/ https://papers.freebsd.org/2019/fosdem/looney-netflix_and_fr...
- prpl 4y agoI thought <bigcorp> used <xyz> is always a funny question/assertion. Bigcorps are large and diverse. In this case - this seems to be user/desktop facing as it's a fuse module for studio assets.
- eatonphil 4y agoThey use FreeBSD for the CDN but I think their application servers use Linux. https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 https://netflixtechblog.com/linux-performance-analysis-in-60...
- loeg 4y agoNetflix uses FreeBSD for their dataplane and AWS Linux for control plane, at least as of ~2020.
- kayson 4y agoTIL about `ps awwfux`. Great way to remember it
- chatmasta 4y agoSee explainshell [0], a great resource, for the full explanation of these flags. (Another one of my favorites is cat -vet) [0] https://explainshell.com/explain?cmd=ps+awwfux https://explainshell.com/explain?cmd=ps+awwfux
- sureglymop 4y agoAnother one I like is `netstat -tulpen` (tulpen is german for tulips)
- asicsp 4y agoI use `cat -A` on Linux, which is equivalent to `cat -vET`
- comex 4y agoThe explainshell output is actually largely wrong here. For historical reasons, GNU ps has two completely different sets of flags depending on whether you use a dash (ps -abcd) or not (ps abcd). The man page documents both sets, but explainshell is getting confused and showing a mix of dash versions and non-dash versions. For example, it shows the documentation for '-f' ("full-format listing"), but it should be showing the documentation for 'f' ("ASCII art process hierarchy (forest)"). (I considered filing a bug report, but there is one already: https://github.com/idank/explainshell/issues/214 https://github.com/idank/explainshell/issues/214)
- chr-s 4y agoI love `ps axufww` so much and ah fuck to having learned it that way. It's the first thing I run on a server when logging in. It tells you so much about the system. That, `w` and a `dmesg -T` will go 90% of the way to diagnosing most system issues.
- yonrg 3y ago