8 ms·

Hastening Linux process cleanup with process_mrelease

92 points by chmaynard 5y ago

teddyh 5y ago
This seems useful for systemd when stopping services.
calpaterson 5y ago
> That work did not address one other unfortunate characteristic of the OOM killer, though: its opinion of what is the least important process on the system tends to differ from that of the system's users. My experience of the linux OOM killer is not that its opinion differs from mine but that it has no opinion at all for a long, long time after the system is in deep trouble. The OOM killer simply does not act quickly enough to save systems. Sadly it's not customisable but 'earlyoom' (packaged for debian and probably everything else) is. I turned it on when I was messing about when debugging a badly behaved bit of software which went into a memory allocation loop and have just left it on. It's saved me a few times and I now plan to leave it on forever. Looks like oomd is an idea along the same lines but with slightly different goals. It's not in my distro so not an easy option for me.
- freedomben 5y ago
  This is my experience also. By the time the OOM killer kicks in, the system has been locked for 15 to 20 minutes already. If it was production you've already terminated the instance. If it's your laptop or desktop, you've already held the power button. Fedora has earlyoom enabled by default but so far it hasn't saved me. I really need to look into configuring it. How did you get started? Man pages? Blog post?
  - cmurf 5y ago
    earlyoom is enable on Fedora 32 and 33 Workstation edition; and 33 KDE spin. On Fedora 34, all editions and spins have systemd-oomd enabled. It does take some initial configuration of systemd service units since oomd works by cgroupsv2 based accounting and killing of entire cgroups, not PIDs. This work is still a work in progress, with uresourced setting up the initial resource allocations (with planned obsolescence). It should be safe to run uresourced on any edition or spin but right now it's only enabled by default on Workstation edition. If the results you're getting aren't expected, there's still some chance it's a bug somewhere, so you should report it against systemd, attach `journalctl -b -o short-monotonic --no-hostname` or at least ~10 minutes of logging prior to the unexpected behavior you're reporting.
  - calpaterson 5y ago
    The manpage combined with some (forced) experimentation with the wonky code mentioned was enough for me. I run with this config as even the earlyoom defaults were not strict enough: -r 30 -m 5 -s 80 I run with 16gb of memory and do use swap. In practice if swap is growing at all once memory is near full, I'm in trouble and action needs to be taken.
  - garaetjjte 5y ago
    >If it's your laptop or desktop, you've already held the power button. Note that if enabled you can use SysRq keys to invoke OOM killer manually.
  - binarybanana 5y ago
    While that is true, you can invoke the OOM killer at any time you want by pressing Ctrl+Alt+Print("SysRq")+F(uck?). Given the letter they picked it seems like they're fully aware of the deficiency. As to why it was never really addressed, that's a good question. The recent le9[1] patch addresses the most annoying symptom of running out of memory by keeping some amount of RAM reserved for clean pages (like static program code) so that those won't get swapped in and out so much. That greatly improves responsiveness under low memory conditions. [1]: https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch
- GranPC 5y ago
  I used to experience this a long time ago, and I know many people who still do - but on my system (running an Ubuntu 18.04 derived distribution) the OOM killer takes at most 3-4 seconds to step in and kill whichever process is consuming the most memory. Does anyone know if Ubuntu/Debian tunes the OOM killer differently to try and stop this from happening?
  - disabled 5y ago
    This post sums it up best: https://news.ycombinator.com/item?id=27652599 https://news.ycombinator.com/item?id=27652599
    - GranPC 5y ago
      I don't see how any of that relates to the OOM killer, am I missing something?
- jeffbee 5y ago
  The reason Linux systems lock up like that is because the kernel will let a process fill all memory up with dirty pages that need writeback, then as soon as it needs some memory the first thing it does it drops all of the in-memory copies of file-backed pages, which includes all of your programs. Then whenever one of your programs wants to run, or continue running by branching to a far address, it has to swap in that code from disk again. Even though you thought your system does not "have swap", it does have swap in effect. The workaround for this is to copy important programs into memory and pin them there with mlock. It is particularly important that if you rely on a userspace OOM killer it gets locked into memory.
  - phsau 5y ago
    I've also found that under these conditions kswapd will effectively consume all your CPU time. The time it spends running is probably proportional to your maximum memory too - in our case it parses through 500+GB of LRU. The blocking writeback behaviour can be managed effectively with dirty page writeback ratio tuning. You don't want to block trying to write 50GB to disk at once when you hit the high dirty page watermark.
  - scottlamb 5y ago
    > The workaround for this is to copy important programs into memory and pin them there with mlock. Nit: I don't think it's necessary to copy the program into anonymous memory to use mlock. You might be thinking of huge pages (transparent or otherwise), which are supported on anonymous pages and unfortunately aren't yet supported on ext4- or btrfs-backed file pages.
  - pengaru 5y ago
    The resource control aspects of cgroups actually improve this situation, if you take advantage of them. You'll still see that page fault thrashing but it becomes isolated to the cgroup experiencing the memory pressure. It doesn't bring down the entire system in my experience.
  - chrsig 5y ago
    I'd never considered the program itself getting dropped and re-read from disk -- but it makes perfect sense. I'm curious if executing the programs from a ramdisk would achieve the same effect.
- saurik 5y ago
  FWIW, I have a very distinct memory of the time OOM decided to kill init (pid 1), which was the most incredibly incompetent choice it possibly could have made ;P.
- hjek 5y ago
  You should try `nohang`. It comes with nice notifications about memory status and when it kills a process. I find that it reacts timely.
rbanffy 5y ago
A possible solution not to this, but for the OOM killer would be an “importance” attribute orthogonal to the task priority - it’s ok to kill the NTP server or to bounce the DNS proxy. It’s much less OK to kill Emacs or my desktop.
- jeffbee 5y ago
  That's what /proc/<pid>/oom_adj & /proc/<pid>/oom_score_adj are for.
  - rbanffy 5y ago
    Cool. Thank you! How long it’s been there under my nose without me realising it? :-D
    - akx 5y ago
      According to https://lwn.net/Articles/391226/ https://lwn.net/Articles/391226/ I'd say about twelve years :-)
- im3w1l 5y ago
  I don't know... Everytime I have an OOM it's been "my fault". Either because I wrote some code that allocated memory in a busy loop, or because I tried to compile something in a memory starved VM. I could totally see myself running out of memory trying to open /dev/sda in emacs or doing something stupid in elisp. So killing emacs makes a lot more sense than some innocent daemon.
smcameron 5y ago
So, if a process is in an uninterruptible sleep, because say, it's doing i/o, say reading from disk, and furthermore, say it's using O_DIRECT, so the storage stack is going to set things up to bypass the buffer cache and DMA directly into the process's memory, and then you rip away the process's memory and give it to another process... and then the DMA completes, and kaboom? The DMA just clobbered an unsuspecting process's memory? That is, it was my understanding that the reason a process is in an uninterruptible sleep is generally because it's waiting for a DMA to complete, and if you were permitted to interrupt it, the DMA would eventually complete and clobber who knows what. Ripping the memory away from such a process would, to my first glance, seem to encounter the same problem -- how do you stop the pending DMA (which might already be in progress, but which might also take awhile to complete.) Whatever method, it would be device dependent, which makes it impractical (who's going to retrofit all the device drivers with DMA stopping APIs, and I'm sure there are many devices that have no way to stop pending DMAs, and anyway, maybe the DMA is already in progress, only half completed.) Maybe do it at the pci level, unmap DMA buffers. But many current drivers will generally assume that they never give their devices bad bus addresses, yet now the device is attempting DMA to a bad bus address (i.e. a suddenly unmapped bus address). Well, I've been out of the linux driver game for awhile now, so perhaps I'm missing or forgetting something. Ripping memory out from under a process with pending DMA sounds pretty sketchy to me though. But, if dumb old me can think of this, of course the kernel developers can also, and undoubtedly did. Wonder how it really works?
- phendrenad2 5y ago
  Mayne usermode processes shouldn't have uninterruptible I/O access.
  - scottlamb 5y ago
    > Mayne usermode processes shouldn't have uninterruptible I/O access. This isn't something userspace processes opt into. It's just how blocking read and write syscalls on filesystems work [1]. If you ever hit a bad block, you may notice that threads that touch it through the fs just hang and you can't recover or kill them. This is how it's always been on Linux and other Unix systems. I of course absolutely hate this and am 100% behind you if you're suggesting changing it. [1] With the exception of nfs if you set the "intr" mount option.
Ericson2314 5y ago
Reading about "pidfds" makes me wish CloudABI fully became a thing.
- ungamedplayer 5y ago
  Can you share more details on this ?
  - Ericson2314 5y ago
    Which? Pidfds were linked in the article to a great other article. CloudABI was about making an ABI/API so capsicum violations were build-time errors. Everything as a file descriptor is necessary for capsicum. WASM's WASI is sort of the spiritual successor, but I prefer a diversity of tactics so CloudABI and WASI should both exist. CloudABI is the best way to save desktop computing that's not a complete Hail Mary.
    - Ericson2314 5y ago
      https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7da0cb6@JohnEricson.me/T/#m6be1994668e6f34837496c86f37f9fe52bfae990 https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7d... I tried to kick of more discussion of spawning processes with pidfds --- I think sticking with fork + (fanicier) exec was the weakest part of CloudABI, but understandable at the time given one was trying to be implementable by all Unix-like kernels. The response from the main pidfds person is that plan should be forthcoming this fall!

92 points by chmaynard 5y ago

This seems useful for systemd when stopping services.

> That work did not address one other unfortunate characteristic of the OOM killer, though: its opinion of what is the least important process on the system tends to differ from that of the system's users. My experience of the linux OOM killer is not that its opinion differs from mine but that it has no opinion at all for a long, long time after the system is in deep trouble. The OOM killer simply does not act quickly enough to save systems. Sadly it's not customisable but 'earlyoom' (packaged for debian and probably everything else) is. I turned it on when I was messing about when debugging a badly behaved bit of software which went into a memory allocation loop and have just left it on. It's saved me a few times and I now plan to leave it on forever. Looks like oomd is an idea along the same lines but with slightly different goals. It's not in my distro so not an easy option for me.

freedomben 5y ago
This is my experience also. By the time the OOM killer kicks in, the system has been locked for 15 to 20 minutes already. If it was production you've already terminated the instance. If it's your laptop or desktop, you've already held the power button. Fedora has earlyoom enabled by default but so far it hasn't saved me. I really need to look into configuring it. How did you get started? Man pages? Blog post?
- cmurf 5y ago
  earlyoom is enable on Fedora 32 and 33 Workstation edition; and 33 KDE spin. On Fedora 34, all editions and spins have systemd-oomd enabled. It does take some initial configuration of systemd service units since oomd works by cgroupsv2 based accounting and killing of entire cgroups, not PIDs. This work is still a work in progress, with uresourced setting up the initial resource allocations (with planned obsolescence). It should be safe to run uresourced on any edition or spin but right now it's only enabled by default on Workstation edition. If the results you're getting aren't expected, there's still some chance it's a bug somewhere, so you should report it against systemd, attach `journalctl -b -o short-monotonic --no-hostname` or at least ~10 minutes of logging prior to the unexpected behavior you're reporting.
- calpaterson 5y ago
  The manpage combined with some (forced) experimentation with the wonky code mentioned was enough for me. I run with this config as even the earlyoom defaults were not strict enough: -r 30 -m 5 -s 80 I run with 16gb of memory and do use swap. In practice if swap is growing at all once memory is near full, I'm in trouble and action needs to be taken.
- garaetjjte 5y ago
  >If it's your laptop or desktop, you've already held the power button. Note that if enabled you can use SysRq keys to invoke OOM killer manually.
- binarybanana 5y ago
  While that is true, you can invoke the OOM killer at any time you want by pressing Ctrl+Alt+Print("SysRq")+F(uck?). Given the letter they picked it seems like they're fully aware of the deficiency. As to why it was never really addressed, that's a good question. The recent le9[1] patch addresses the most annoying symptom of running out of memory by keeping some amount of RAM reserved for clean pages (like static program code) so that those won't get swapped in and out so much. That greatly improves responsiveness under low memory conditions. [1]: https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch
GranPC 5y ago
I used to experience this a long time ago, and I know many people who still do - but on my system (running an Ubuntu 18.04 derived distribution) the OOM killer takes at most 3-4 seconds to step in and kill whichever process is consuming the most memory. Does anyone know if Ubuntu/Debian tunes the OOM killer differently to try and stop this from happening?
- disabled 5y ago
  This post sums it up best: https://news.ycombinator.com/item?id=27652599 https://news.ycombinator.com/item?id=27652599
  - GranPC 5y ago
    I don't see how any of that relates to the OOM killer, am I missing something?
jeffbee 5y ago
The reason Linux systems lock up like that is because the kernel will let a process fill all memory up with dirty pages that need writeback, then as soon as it needs some memory the first thing it does it drops all of the in-memory copies of file-backed pages, which includes all of your programs. Then whenever one of your programs wants to run, or continue running by branching to a far address, it has to swap in that code from disk again. Even though you thought your system does not "have swap", it does have swap in effect. The workaround for this is to copy important programs into memory and pin them there with mlock. It is particularly important that if you rely on a userspace OOM killer it gets locked into memory.
- phsau 5y ago
  I've also found that under these conditions kswapd will effectively consume all your CPU time. The time it spends running is probably proportional to your maximum memory too - in our case it parses through 500+GB of LRU. The blocking writeback behaviour can be managed effectively with dirty page writeback ratio tuning. You don't want to block trying to write 50GB to disk at once when you hit the high dirty page watermark.
- scottlamb 5y ago
  > The workaround for this is to copy important programs into memory and pin them there with mlock. Nit: I don't think it's necessary to copy the program into anonymous memory to use mlock. You might be thinking of huge pages (transparent or otherwise), which are supported on anonymous pages and unfortunately aren't yet supported on ext4- or btrfs-backed file pages.
- pengaru 5y ago
  The resource control aspects of cgroups actually improve this situation, if you take advantage of them. You'll still see that page fault thrashing but it becomes isolated to the cgroup experiencing the memory pressure. It doesn't bring down the entire system in my experience.
- chrsig 5y ago
  I'd never considered the program itself getting dropped and re-read from disk -- but it makes perfect sense. I'm curious if executing the programs from a ramdisk would achieve the same effect.
saurik 5y ago
FWIW, I have a very distinct memory of the time OOM decided to kill init (pid 1), which was the most incredibly incompetent choice it possibly could have made ;P.
hjek 5y ago
You should try `nohang`. It comes with nice notifications about memory status and when it kills a process. I find that it reacts timely.

rbanffy 5y ago

A possible solution not to this, but for the OOM killer would be an “importance” attribute orthogonal to the task priority - it’s ok to kill the NTP server or to bounce the DNS proxy. It’s much less OK to kill Emacs or my desktop.

jeffbee 5y ago
That's what /proc/<pid>/oom_adj & /proc/<pid>/oom_score_adj are for.
- rbanffy 5y ago
  Cool. Thank you! How long it’s been there under my nose without me realising it? :-D
  - akx 5y ago
    According to https://lwn.net/Articles/391226/ https://lwn.net/Articles/391226/ I'd say about twelve years :-)
im3w1l 5y ago
I don't know... Everytime I have an OOM it's been "my fault". Either because I wrote some code that allocated memory in a busy loop, or because I tried to compile something in a memory starved VM. I could totally see myself running out of memory trying to open /dev/sda in emacs or doing something stupid in elisp. So killing emacs makes a lot more sense than some innocent daemon.

smcameron 5y ago

So, if a process is in an uninterruptible sleep, because say, it's doing i/o, say reading from disk, and furthermore, say it's using O_DIRECT, so the storage stack is going to set things up to bypass the buffer cache and DMA directly into the process's memory, and then you rip away the process's memory and give it to another process... and then the DMA completes, and kaboom? The DMA just clobbered an unsuspecting process's memory? That is, it was my understanding that the reason a process is in an uninterruptible sleep is generally because it's waiting for a DMA to complete, and if you were permitted to interrupt it, the DMA would eventually complete and clobber who knows what. Ripping the memory away from such a process would, to my first glance, seem to encounter the same problem -- how do you stop the pending DMA (which might already be in progress, but which might also take awhile to complete.) Whatever method, it would be device dependent, which makes it impractical (who's going to retrofit all the device drivers with DMA stopping APIs, and I'm sure there are many devices that have no way to stop pending DMAs, and anyway, maybe the DMA is already in progress, only half completed.) Maybe do it at the pci level, unmap DMA buffers. But many current drivers will generally assume that they never give their devices bad bus addresses, yet now the device is attempting DMA to a bad bus address (i.e. a suddenly unmapped bus address). Well, I've been out of the linux driver game for awhile now, so perhaps I'm missing or forgetting something. Ripping memory out from under a process with pending DMA sounds pretty sketchy to me though. But, if dumb old me can think of this, of course the kernel developers can also, and undoubtedly did. Wonder how it really works?

phendrenad2 5y ago
Mayne usermode processes shouldn't have uninterruptible I/O access.
- scottlamb 5y ago
  > Mayne usermode processes shouldn't have uninterruptible I/O access. This isn't something userspace processes opt into. It's just how blocking read and write syscalls on filesystems work [1]. If you ever hit a bad block, you may notice that threads that touch it through the fs just hang and you can't recover or kill them. This is how it's always been on Linux and other Unix systems. I of course absolutely hate this and am 100% behind you if you're suggesting changing it. [1] With the exception of nfs if you set the "intr" mount option.

Ericson2314 5y ago

Reading about "pidfds" makes me wish CloudABI fully became a thing.

ungamedplayer 5y ago
Can you share more details on this ?
- Ericson2314 5y ago
  Which? Pidfds were linked in the article to a great other article. CloudABI was about making an ABI/API so capsicum violations were build-time errors. Everything as a file descriptor is necessary for capsicum. WASM's WASI is sort of the spiritual successor, but I prefer a diversity of tactics so CloudABI and WASI should both exist. CloudABI is the best way to save desktop computing that's not a complete Hail Mary.
  - Ericson2314 5y ago
    https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7da0cb6@JohnEricson.me/T/#m6be1994668e6f34837496c86f37f9fe52bfae990 https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7d... I tried to kick of more discussion of spawning processes with pidfds --- I think sticking with fork + (fanicier) exec was the weakest part of CloudABI, but understandable at the time given one was trying to be implementable by all Unix-like kernels. The response from the main pidfds person is that plan should be forthcoming this fall!