12 ms·

I've now disabled systemd-oomd on my Fedora desktops

60 points by jessaustin 4y ago

eimrine 4y ago
Great blog, I really enjoyed some blogposts about UNIX which may be found on this resource.
josephcsible 4y ago
When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality?
- voidr 4y ago
  Can I ship a badly configured Linux kernel in my distro? Yes. Is that the fault of the kernel developers? No. I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it. Replacing system components is hard and the early days are going to suck. Just because there is a lot of pain and suffering with Wayland does not mean that it was a mistake and we should have just stuck with X11. Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
  - josephcsible 4y ago
    > Can I ship a badly configured Linux kernel in my distro? Yes. Is that the fault of the kernel developers? No. If the default configuration were good and you changed it to be bad, that's entirely your fault. But if the default configuration were bad and you just didn't fix it, that's mostly if not entirely upstream's fault. > I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it. Yes, we can do that. The point is that we shouldn't have to. > Replacing system components is hard and the early days are going to suck. Nobody's complaining that things are getting replaced. We're complaining that the replacement is worse than the old thing. > Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn. Systemd didn't exist until 2010, so any changes from 1970 to 2010 clearly can't have made it necessary. And the BSDs are all still good, functional operating systems, despite not being able to run it, so it's clearly not necessary even for what's changed since 2010.
- matthewmacleod 4y ago
  Probably around the same time as people stop posting poorly informed comments about systemd on Hacker News (i.e. approximately never)
- asddubs 4y ago
  Seems to me like in this case we've just gone from a broken system to another broken system
  - orra 4y ago
    Quite. Can we not phase out overcommit? Windows doesn't do it. I think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in favour of posix_spawn()?
    - josephcsible 4y ago
      What would we do about all of the processes that fork but don't ever exec? And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
      - AnIdiotOnTheNet 4y ago
        What's this whole open-source thing worth if we can't fix those to do it a better way? Alternatively, just let programs opt-out of overcommit and also OOM killer. Then only badly behaving programs will be shot in the head randomly.
        tinus_hn 4y ago
        Too bad it’s basically impossible to deal with the possibility that you request memory and don’t get it, apart from crashing.
        RunningDroid 4y ago
        You could hang and hope someone else free()'s enough for you to continue, but the system will lock up if too many processes try that without giving up after a timeout
        AnIdiotOnTheNet 4y ago
        There are lots of non-fatal ways to handle an out of memory condition (including doing your own paging) depending on the use-case of the allocation. Programmers suffer from learned helplessness because they're writing for systems that lie to them about memory and languages that make handling errors a pain.
- aliqot 4y ago
  > When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality? We aren't. Our time's up. The new generation is on now, and we didn't teach them hardly anything we learned because we were so invested in making fun of them for enjoying ubuntu. Best way we can move past it is to take the next curmudgeonly step and move to OpenBSD and Plan9, as they're the only things sufficiently dense and opaque to be as yet out of reach of the current generation.
  - deleted 4y ago
    [deleted]
  - 0dayz 4y ago
    How exactly do these to OS, do things the "correct way"?
    - hanafudafan 4y ago
      How don't they?
  - ectospheno 4y ago
    I’m a certified curmudgeon then. If a workload can run on OpenBSD then it does when I’m working. If I’m ever speaking the programmers language and cursing a computer then it is rarely an OpenBSD box. They just work.
- freedomben 4y ago
  Do you seriously consider the kernel OOM killer to be "battle-tested" and (implied) sufficient?
  - josephcsible 4y ago
    I'm not saying the kernel one is perfect. I'm just saying it's better than this one.
- kaba0 4y ago
  You do realize that oomd has as much to do with systemd as your bash script that has a service file written for it?
  - josephcsible 4y ago
    What are you talking about? Systemd-oomd is literally part of systemd. Its source code is maintained in the same Git repo, bugs are tracked in the same project, etc. Are you getting it mixed up with Facebook's oomd?
- hanafudafan 4y ago
  I never started, I went right from Debian 5 to OpenBSD.
- Shorel 4y ago
  The problem in this case is deeper. The Linux kernel overcommits memory. OOMD is a workaround for this behaviour. And it can't be disabled. I disabled the kernel memory overcommit and then neither Chrome nor Edge were running. I can't use my computer without my browser of choice. Therefore, overcommit can't be disabled.
  - bcrl 4y ago
    Maybe the problem is with the bloated apps that don't free memory? You can't blame the kernel for that. I routinely see Firefox consuming 10GB of RAM on my Linux desktop. It's nuts.
  - chlorion 4y ago
    Chrome and Edge not functioning without overcommit is not the fault of the kernel though. The browsers will allocate more memory than is physically available in the system but often not "touch" it. Without over-commit this causes an OOM situation and the kernel kills the browser, but that's what it's supposed to do when over-commit is disabled! I believe some of these massive allocations are used for security purposes or somehow related to the chromium sandbox but I can't remember exactly what's going on. Some sanitizers like ASAN do something similar for detecting invalid memory accesses (and hence can't be used without over-commit either)!
- WastingMyTime89 4y ago
  If you read the article, you will see that the issue is that Fedora configures it really poorly for a single user system not that systemd-oomd doesn't work. They enroll all the user processes into it.
jraph 4y ago
Killing the user's graphical session is terrible. On the other hand, not having a userspace OOM killer means (meant?) that if your system runs out of memory, it will become so slow as to be totally unresponsive and the only thing you can do is to reboot. Which is even worse. Is that still true today or did the kernel OOM killer improved? How does earlyoom compare against systemd-oomd?
- bigpeopleareold 4y ago
  I'll just say that earlyoom is much more bearable out-of-the-box than what this sounds like and much better than waiting for Linux to trigger the internal OOM killer (or trying to remember the SysRq key sequence to just trigger it. I always forget and don't have it close by when I need it :) ). I had to play around with the configuration a little to get it to match the conditions I could potentially get an OOM, but when it did reach it on my desktop, it would kill something like Slack, definitely not the whole X session. (FWIW, this happened when building a large Java project, that would burst with memory usage at certain stages of a build, on my computer that just has 16gb. Outside of managing OOM, it's something to play around with before giving up and just upping the RAM. :) )
  - thecosmicfrog 4y ago
    Alt+SysRq+REISUB Or, as a mnemonic: Reboot Even If System Utterly Borked* *Being Irish, I prefer "Banjaxed" for the last word, personally.
    - bigpeopleareold 4y ago
      Ah yeah, that :D But that's a full terminate process/filesystem sync/shutdown + restart ... You got me to look it up: Alt-SysRq-F just for the kernel oom_killer ... On ThinkPads, that's: Press Fn-Alt-S, then release Fn-S, then press the desired letter. Result from dmesg: "[154256.637037] sysrq: This sysrq operation is disabled." Bleh. Didn't work when you needed it :) Of course these can be enabled, but earlyoom is soo much easier than remembering the OOM dance or the REISUB dance (particularly if it is just an OOM :D)
nolls 4y ago
> By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes This is so obviously poorly thought-out as to border malicious negligence.
RealStickman_ 4y ago
I've been running nohang for quite a while now on all my desktop linux devices. In the rare cases where I mismanaged my memory or had a program hang it has always managed to kill the culprit and only the culprit. It also gives me a nice desktop notification about what it killed, though I'm not sure where/if a more permanent log is written.
egberts1 4y ago
Systemd-oomd, may be great for non-desktop server's needs for some, but definitely not great for single-user desktops, much less multi-user desktops. THE Pain point remains in systemd-oomd's poor (and often, no) crash dump or traceback support. Nevertheless, systemd-oomd remains an excellent OOM mass-killer (via cgroupv2), just not the discriminating assassin ... as (mis-)configured ... by Redhat/Fedora.
- superkuh 4y ago
  This is non-init related systemd from IBM->Red Hat->Fedora in a nutshell. It's good for fleets of mega-corp servers. It's bad for human people running linux desktops. But since IBM is the primary entity paying people to write non-init systemd is everywhere and only niche distros avoid it.
  - nullc 4y ago
    Have you ever met someone running fleets of mega-corp servers who was a fan of systemd? When I've seen them complain the systemd people tell them that they're just stuck in their ways and that systemd is a revelation on desktops where users don't have the historical unix experience holding them back...
    - JoeyBananas 4y ago
      They don't love systemd because they only interact with it when they're debugging Asok the intern's broken services.
    - dontlaugh 4y ago
      It’s particularly good on servers. All preceding systems were much worse. It’s only useful on desktop because of its similarities with launchd, which is particularly good on desktop.
    - superkuh 4y ago
      I've just heard from people claiming the absurd way NIC are named in systemd is good for server fleet types and the like. Because it's obviously terrible for human people with desktop computers. I guess I just assumed the bad design decisions had some purpose.
xg15 4y ago
Dumb question: why is this a thing at all if the kernel oom killer already exists? Doesn't this effectively do the same as the existing oom killer, but worse?
- josephcsible 4y ago
  Yes it does. Basically everything under the systemd umbrella does the same thing as something that already existed (remember that we had perfectly good Linux boxes before its release in 2010), but worse than the old thing.
  - kaba0 4y ago
    Is it really mentally healthy to hate to this degree a goddamn free, libre, open-source project?
    - josephcsible 4y ago
      It's not like systemd gave us our freedom. The things it replaced were also FLOSS.
    - hansvm 4y ago
      Their post seemed pretty tame. No evocative emotions or deity-assisted emphasis. They just said that between the available free, libre, open-source projects they prefer what they had before -- akin to saying they like to install qutebrowser rather than firefox.
  - Avamander 4y ago
    I have serious doubts about people's familiarity with the alternatives if they say something like this.
- dinosaurdynasty 4y ago
  Iirc, systemd-oomd comes from oomd which comes from Facebook who had issues with the kernel oom killer. (The kernel oom killer, especially with slow swap, can also take ages to activate, which means you're just down the entire time, and it's better to kill stuff more often than to sometimes freeze.)
fbhabbed 4y ago
I confirm that the state of systemd-oomd is ridicolous right now and will easily kill important background processes that aren't even taking up that much RAM. It killed my Thunar file transfer, for example
raspyberr 4y ago
Systemd seems great for servers and sysadmins. Does it's size, scope, and complexity actually do something for the grandmas of the world who use Linux? For that basic type of user, was the millions of lines of code and probably hundreds of thousands of hours of programmer's time worth it?
- trynewideas 4y ago
  Probably, since all of those users rely on services provided by servers managed by sysadmins. But it's such a misfit on desktop distros that the criticism there is valid and much louder. It never should've been a one-daemon-fits-all solution.
  - dinosaurdynasty 4y ago
    systemd has never been a one-daemon-fits-all solution... it's dozens of daemons.
    - ykonstant 4y ago
      Collectively known as legiond-systemd.
- josephcsible 4y ago
  I'm a sysadmin for a medium-sized fleet of servers, and systemd definitely isn't great for me.
  - aliqot 4y ago
    In what way?
    - josephcsible 4y ago
      It makes changes that break things (like suddenly kill -9'ing all of user's nohup and screen processes when they exited their SSH sessions, completely defeating the purpose of the tools) and replaces old simple ways of doing things with new complicated ways, without benefits of the new ways to justify the changes.
trynewideas 4y ago
Mint opted out of systemd-oomd with v21: https://news.ycombinator.com/item?id=31960266 https://news.ycombinator.com/item?id=31960266 Ubuntu 22.04 was also affected: https://news.ycombinator.com/item?id=31530739 https://news.ycombinator.com/item?id=31530739 And flipped a config flag to make it less aggressive: https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill
opan 4y ago
I'm not using systemd, but have been pretty happy with earlyoom. It tends to take out my web browser I think. Fairly inoffensive.
nullc 4y ago
OOMD seems to have a bug that if you have >64 GB of ram (or maybe >128GB?) it will randomly kill very large processes even though you're nowhere near running out of memory (much less swap). Cost me god knows how much money or wasted power from lost computations until I figured that out and disabled it.
- worthless-trash 4y ago
  Did you lodge a bug, i'd like to weigh in on that.
- jessaustin 4y ago
  Cost me god knows how much money... This might have made some IaaS firms happy?
bravetraveler 4y ago
Same here; been disabling it since it was introduced to Fedora... a release or two ago. My desktops have 64GB+ of memory typically, they're virtualization workhorses. I don't know what it is in oomd, but any time I make actual real use of it, things started getting killed. Zero memory pressure. Talking sections of 60+GB free, so it's not fragmentation either. The grouping of the entire session the link suggests seems likely
simion314 4y ago
Maybe Off-topic, not sure if systemd is to blame or Apache or the package maintainers , but on a LAMP stack i was creating some images in the temp folder then do some work on them. So for debugging I wanted to opent he temp folder and look at the images. The files were not there, the system is now full of magic, it pretends it gives you a temp file like /tmp/file1.png but the file is not there. I fcking Have o idea why they think is a good idea to lie to me what the file path is, I had luck that some other person hit this issue and somehow google managed to put it as a result to my searches.
- 5- 4y ago
  that sounds like my good old enemy https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html https://www.freedesktop.org/software/systemd/man/tmpfiles.d....
- ciupicri 4y ago
  It's because Apache httpd or other services are probably using their own "version"/"instance" of /tmp for better isolation from the rest of the system and extra security. From the point of view of the program the files are written under /tmp, but in reality they're stored somewhere else. Try this to see the files: ls /tmp/systemd-private-$(systemd-id128 boot-id)-httpd.service-*/tmp Run systemctl show --all httpd.service and look out for the PrivateTmp directive [1]. Use systemctl edit httpd.service to override it. [1]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp= https://www.freedesktop.org/software/systemd/man/systemd.exe...
  - simion314 4y ago
    Thanks. But this is such a antiuser feature, like the incident was like 1 I use std function to get a temp file path 2 I write content in that file 3 I do all the checks to ensure the write worked, I use std functions to check file exists 4 I use std function to print the file path And the file path is a lie, It all would be fine if it would use the real path, and if a newb would hardocde "/tmp/file1" then I would be happy with a error message that I can't write there . I assume they had some reasons for the lie, I am not convinced it was worth it, I do not like magic hidden stuff.
pengaru 4y ago
> Achievement unlocked: I just had systemd-oomd kill my entire X session for unclear reasons. > Does it log details about the session it killed so I could see, say, which program was using too much memory? > Nope. > > Systemd-oomd has now been voted off my island. https://github.com/systemd/systemd/blob/v252/src/oom/oomd-manager.c#L409 https://github.com/systemd/systemd/blob/v252/src/oom/oomd-ma... https://github.com/systemd/systemd/blob/v252/src/oom/oomd-manager.c#L532 https://github.com/systemd/systemd/blob/v252/src/oom/oomd-ma... Chris's wiki has now been voted out of my sight.
- Avamander 4y ago
  To be totally fair, that also happens with the kernel's OOM killer. Sometimes you wish X had been killed, instead of a total freeze or for example LUKS/dm-crypt being unable to allocate memory (and causing way way way worse issues than X being killed).
- alyandon 4y ago
  https://github.com/systemd/systemd/blame/e8dc52766e1fdb4f8c09c3ab654d1270e1090c8d/src/oom/oomd-manager.c#L409 https://github.com/systemd/systemd/blame/e8dc52766e1fdb4f8c0... Yeah, looks like logging was added 1.5 years ago so hard to take his perspective seriously.
  - pengaru 4y ago
    It's also problematic that he doesn't seem to have reported the issue to either his distributor or upstream systemd. If he did, it's not mentioned in his rant. There aren't any links to any bug reports where the developers demonstrably failed to care or anything rant-worthy. Considering systemd-oomd is a relatively new addition to the project, contributed by FB staff, it wouldn't surprise me if there were desktop integration issues yet to be sorted out by the various distros. The post doesn't strike me as made in good faith, seems written by someone with an axe to grind with systemd.
eternityforest 4y ago
Systemd is wonderful. But could they please stop trying to invent lightweight replacements? You're not suckless. Nobody wants you to be suckless. I like you specifically because you aren't anything like suckless. Don't replace anything unless you have close to full feature parity. Nobody wants your minimum viable product components. The people who do don't want systemd. Init system? Absolutely. Timers? You've done a wonderful job. Time sync? Please no, crony does it better.
Operative0198 4y ago
So many things wrong with this article. You can absolutely see which programs triggered systemd-oomd into action in journalctl logs. Even better, I now seem to get notifications with the culprit whenever it kicks in for my desktop setup. I can also see the design decision made to kill the graphical session when the system becomes unresponsive. The alternative to that situation pre-systemd-oomd was a force reboot of the system.

60 points by jessaustin 4y ago

Great blog, I really enjoyed some blogposts about UNIX which may be found on this resource.

When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality?

voidr 4y ago
Can I ship a badly configured Linux kernel in my distro? Yes. Is that the fault of the kernel developers? No. I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it. Replacing system components is hard and the early days are going to suck. Just because there is a lot of pain and suffering with Wayland does not mean that it was a mistake and we should have just stuck with X11. Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
- josephcsible 4y ago
  > Can I ship a badly configured Linux kernel in my distro? Yes. Is that the fault of the kernel developers? No. If the default configuration were good and you changed it to be bad, that's entirely your fault. But if the default configuration were bad and you just didn't fix it, that's mostly if not entirely upstream's fault. > I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it. Yes, we can do that. The point is that we shouldn't have to. > Replacing system components is hard and the early days are going to suck. Nobody's complaining that things are getting replaced. We're complaining that the replacement is worse than the old thing. > Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn. Systemd didn't exist until 2010, so any changes from 1970 to 2010 clearly can't have made it necessary. And the BSDs are all still good, functional operating systems, despite not being able to run it, so it's clearly not necessary even for what's changed since 2010.
matthewmacleod 4y ago
Probably around the same time as people stop posting poorly informed comments about systemd on Hacker News (i.e. approximately never)
asddubs 4y ago
Seems to me like in this case we've just gone from a broken system to another broken system
- orra 4y ago
  Quite. Can we not phase out overcommit? Windows doesn't do it. I think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in favour of posix_spawn()?
  - josephcsible 4y ago
    What would we do about all of the processes that fork but don't ever exec? And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
    - AnIdiotOnTheNet 4y ago
      What's this whole open-source thing worth if we can't fix those to do it a better way? Alternatively, just let programs opt-out of overcommit and also OOM killer. Then only badly behaving programs will be shot in the head randomly.
      - tinus_hn 4y ago
        Too bad it’s basically impossible to deal with the possibility that you request memory and don’t get it, apart from crashing.
        RunningDroid 4y ago
        You could hang and hope someone else free()'s enough for you to continue, but the system will lock up if too many processes try that without giving up after a timeout
        AnIdiotOnTheNet 4y ago
        There are lots of non-fatal ways to handle an out of memory condition (including doing your own paging) depending on the use-case of the allocation. Programmers suffer from learned helplessness because they're writing for systems that lie to them about memory and languages that make handling errors a pain.
aliqot 4y ago
> When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality? We aren't. Our time's up. The new generation is on now, and we didn't teach them hardly anything we learned because we were so invested in making fun of them for enjoying ubuntu. Best way we can move past it is to take the next curmudgeonly step and move to OpenBSD and Plan9, as they're the only things sufficiently dense and opaque to be as yet out of reach of the current generation.
- deleted 4y ago
  [deleted]
- 0dayz 4y ago
  How exactly do these to OS, do things the "correct way"?
  - hanafudafan 4y ago
    How don't they?
- ectospheno 4y ago
  I’m a certified curmudgeon then. If a workload can run on OpenBSD then it does when I’m working. If I’m ever speaking the programmers language and cursing a computer then it is rarely an OpenBSD box. They just work.
freedomben 4y ago
Do you seriously consider the kernel OOM killer to be "battle-tested" and (implied) sufficient?
- josephcsible 4y ago
  I'm not saying the kernel one is perfect. I'm just saying it's better than this one.
kaba0 4y ago
You do realize that oomd has as much to do with systemd as your bash script that has a service file written for it?
- josephcsible 4y ago
  What are you talking about? Systemd-oomd is literally part of systemd. Its source code is maintained in the same Git repo, bugs are tracked in the same project, etc. Are you getting it mixed up with Facebook's oomd?
hanafudafan 4y ago
I never started, I went right from Debian 5 to OpenBSD.
Shorel 4y ago
The problem in this case is deeper. The Linux kernel overcommits memory. OOMD is a workaround for this behaviour. And it can't be disabled. I disabled the kernel memory overcommit and then neither Chrome nor Edge were running. I can't use my computer without my browser of choice. Therefore, overcommit can't be disabled.
- bcrl 4y ago
  Maybe the problem is with the bloated apps that don't free memory? You can't blame the kernel for that. I routinely see Firefox consuming 10GB of RAM on my Linux desktop. It's nuts.
- chlorion 4y ago
  Chrome and Edge not functioning without overcommit is not the fault of the kernel though. The browsers will allocate more memory than is physically available in the system but often not "touch" it. Without over-commit this causes an OOM situation and the kernel kills the browser, but that's what it's supposed to do when over-commit is disabled! I believe some of these massive allocations are used for security purposes or somehow related to the chromium sandbox but I can't remember exactly what's going on. Some sanitizers like ASAN do something similar for detecting invalid memory accesses (and hence can't be used without over-commit either)!
WastingMyTime89 4y ago
If you read the article, you will see that the issue is that Fedora configures it really poorly for a single user system not that systemd-oomd doesn't work. They enroll all the user processes into it.

jraph 4y ago

Killing the user's graphical session is terrible. On the other hand, not having a userspace OOM killer means (meant?) that if your system runs out of memory, it will become so slow as to be totally unresponsive and the only thing you can do is to reboot. Which is even worse. Is that still true today or did the kernel OOM killer improved? How does earlyoom compare against systemd-oomd?

bigpeopleareold 4y ago
I'll just say that earlyoom is much more bearable out-of-the-box than what this sounds like and much better than waiting for Linux to trigger the internal OOM killer (or trying to remember the SysRq key sequence to just trigger it. I always forget and don't have it close by when I need it :) ). I had to play around with the configuration a little to get it to match the conditions I could potentially get an OOM, but when it did reach it on my desktop, it would kill something like Slack, definitely not the whole X session. (FWIW, this happened when building a large Java project, that would burst with memory usage at certain stages of a build, on my computer that just has 16gb. Outside of managing OOM, it's something to play around with before giving up and just upping the RAM. :) )
- thecosmicfrog 4y ago
  Alt+SysRq+REISUB Or, as a mnemonic: Reboot Even If System Utterly Borked* *Being Irish, I prefer "Banjaxed" for the last word, personally.
  - bigpeopleareold 4y ago
    Ah yeah, that :D But that's a full terminate process/filesystem sync/shutdown + restart ... You got me to look it up: Alt-SysRq-F just for the kernel oom_killer ... On ThinkPads, that's: Press Fn-Alt-S, then release Fn-S, then press the desired letter. Result from dmesg: "[154256.637037] sysrq: This sysrq operation is disabled." Bleh. Didn't work when you needed it :) Of course these can be enabled, but earlyoom is soo much easier than remembering the OOM dance or the REISUB dance (particularly if it is just an OOM :D)

nolls 4y ago

> By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes This is so obviously poorly thought-out as to border malicious negligence.

RealStickman_ 4y ago

I've been running nohang for quite a while now on all my desktop linux devices. In the rare cases where I mismanaged my memory or had a program hang it has always managed to kill the culprit and only the culprit. It also gives me a nice desktop notification about what it killed, though I'm not sure where/if a more permanent log is written.

egberts1 4y ago

Systemd-oomd, may be great for non-desktop server's needs for some, but definitely not great for single-user desktops, much less multi-user desktops. THE Pain point remains in systemd-oomd's poor (and often, no) crash dump or traceback support. Nevertheless, systemd-oomd remains an excellent OOM mass-killer (via cgroupv2), just not the discriminating assassin ... as (mis-)configured ... by Redhat/Fedora.

superkuh 4y ago
This is non-init related systemd from IBM->Red Hat->Fedora in a nutshell. It's good for fleets of mega-corp servers. It's bad for human people running linux desktops. But since IBM is the primary entity paying people to write non-init systemd is everywhere and only niche distros avoid it.
- nullc 4y ago
  Have you ever met someone running fleets of mega-corp servers who was a fan of systemd? When I've seen them complain the systemd people tell them that they're just stuck in their ways and that systemd is a revelation on desktops where users don't have the historical unix experience holding them back...
  - JoeyBananas 4y ago
    They don't love systemd because they only interact with it when they're debugging Asok the intern's broken services.
  - dontlaugh 4y ago
    It’s particularly good on servers. All preceding systems were much worse. It’s only useful on desktop because of its similarities with launchd, which is particularly good on desktop.
  - superkuh 4y ago
    I've just heard from people claiming the absurd way NIC are named in systemd is good for server fleet types and the like. Because it's obviously terrible for human people with desktop computers. I guess I just assumed the bad design decisions had some purpose.

xg15 4y ago

Dumb question: why is this a thing at all if the kernel oom killer already exists? Doesn't this effectively do the same as the existing oom killer, but worse?

josephcsible 4y ago
Yes it does. Basically everything under the systemd umbrella does the same thing as something that already existed (remember that we had perfectly good Linux boxes before its release in 2010), but worse than the old thing.
- kaba0 4y ago
  Is it really mentally healthy to hate to this degree a goddamn free, libre, open-source project?
  - josephcsible 4y ago
    It's not like systemd gave us our freedom. The things it replaced were also FLOSS.
  - hansvm 4y ago
    Their post seemed pretty tame. No evocative emotions or deity-assisted emphasis. They just said that between the available free, libre, open-source projects they prefer what they had before -- akin to saying they like to install qutebrowser rather than firefox.
- Avamander 4y ago
  I have serious doubts about people's familiarity with the alternatives if they say something like this.
dinosaurdynasty 4y ago
Iirc, systemd-oomd comes from oomd which comes from Facebook who had issues with the kernel oom killer. (The kernel oom killer, especially with slow swap, can also take ages to activate, which means you're just down the entire time, and it's better to kill stuff more often than to sometimes freeze.)

fbhabbed 4y ago

I confirm that the state of systemd-oomd is ridicolous right now and will easily kill important background processes that aren't even taking up that much RAM. It killed my Thunar file transfer, for example

raspyberr 4y ago

Systemd seems great for servers and sysadmins. Does it's size, scope, and complexity actually do something for the grandmas of the world who use Linux? For that basic type of user, was the millions of lines of code and probably hundreds of thousands of hours of programmer's time worth it?

trynewideas 4y ago
Probably, since all of those users rely on services provided by servers managed by sysadmins. But it's such a misfit on desktop distros that the criticism there is valid and much louder. It never should've been a one-daemon-fits-all solution.
- dinosaurdynasty 4y ago
  systemd has never been a one-daemon-fits-all solution... it's dozens of daemons.
  - ykonstant 4y ago
    Collectively known as legiond-systemd.
josephcsible 4y ago
I'm a sysadmin for a medium-sized fleet of servers, and systemd definitely isn't great for me.
- aliqot 4y ago
  In what way?
  - josephcsible 4y ago
    It makes changes that break things (like suddenly kill -9'ing all of user's nohup and screen processes when they exited their SSH sessions, completely defeating the purpose of the tools) and replaces old simple ways of doing things with new complicated ways, without benefits of the new ways to justify the changes.

trynewideas 4y ago

Mint opted out of systemd-oomd with v21: https://news.ycombinator.com/item?id=31960266 https://news.ycombinator.com/item?id=31960266 Ubuntu 22.04 was also affected: https://news.ycombinator.com/item?id=31530739 https://news.ycombinator.com/item?id=31530739 And flipped a config flag to make it less aggressive: https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill

opan 4y ago

I'm not using systemd, but have been pretty happy with earlyoom. It tends to take out my web browser I think. Fairly inoffensive.

nullc 4y ago

OOMD seems to have a bug that if you have >64 GB of ram (or maybe >128GB?) it will randomly kill very large processes even though you're nowhere near running out of memory (much less swap). Cost me god knows how much money or wasted power from lost computations until I figured that out and disabled it.

worthless-trash 4y ago
Did you lodge a bug, i'd like to weigh in on that.
jessaustin 4y ago
Cost me god knows how much money... This might have made some IaaS firms happy?

bravetraveler 4y ago

Same here; been disabling it since it was introduced to Fedora... a release or two ago. My desktops have 64GB+ of memory typically, they're virtualization workhorses. I don't know what it is in oomd, but any time I make actual real use of it, things started getting killed. Zero memory pressure. Talking sections of 60+GB free, so it's not fragmentation either. The grouping of the entire session the link suggests seems likely

simion314 4y ago

Maybe Off-topic, not sure if systemd is to blame or Apache or the package maintainers , but on a LAMP stack i was creating some images in the temp folder then do some work on them. So for debugging I wanted to opent he temp folder and look at the images. The files were not there, the system is now full of magic, it pretends it gives you a temp file like /tmp/file1.png but the file is not there. I fcking Have o idea why they think is a good idea to lie to me what the file path is, I had luck that some other person hit this issue and somehow google managed to put it as a result to my searches.

5- 4y ago
that sounds like my good old enemy https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html https://www.freedesktop.org/software/systemd/man/tmpfiles.d....
ciupicri 4y ago
It's because Apache httpd or other services are probably using their own "version"/"instance" of /tmp for better isolation from the rest of the system and extra security. From the point of view of the program the files are written under /tmp, but in reality they're stored somewhere else. Try this to see the files: ls /tmp/systemd-private-$(systemd-id128 boot-id)-httpd.service-*/tmp Run systemctl show --all httpd.service and look out for the PrivateTmp directive [1]. Use systemctl edit httpd.service to override it. [1]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp= https://www.freedesktop.org/software/systemd/man/systemd.exe...
- simion314 4y ago
  Thanks. But this is such a antiuser feature, like the incident was like 1 I use std function to get a temp file path 2 I write content in that file 3 I do all the checks to ensure the write worked, I use std functions to check file exists 4 I use std function to print the file path And the file path is a lie, It all would be fine if it would use the real path, and if a newb would hardocde "/tmp/file1" then I would be happy with a error message that I can't write there . I assume they had some reasons for the lie, I am not convinced it was worth it, I do not like magic hidden stuff.

pengaru 4y ago

> Achievement unlocked: I just had systemd-oomd kill my entire X session for unclear reasons. > Does it log details about the session it killed so I could see, say, which program was using too much memory? > Nope. > > Systemd-oomd has now been voted off my island. https://github.com/systemd/systemd/blob/v252/src/oom/oomd-manager.c#L409 https://github.com/systemd/systemd/blob/v252/src/oom/oomd-ma... https://github.com/systemd/systemd/blob/v252/src/oom/oomd-manager.c#L532 https://github.com/systemd/systemd/blob/v252/src/oom/oomd-ma... Chris's wiki has now been voted out of my sight.

Avamander 4y ago
To be totally fair, that also happens with the kernel's OOM killer. Sometimes you wish X had been killed, instead of a total freeze or for example LUKS/dm-crypt being unable to allocate memory (and causing way way way worse issues than X being killed).
alyandon 4y ago
https://github.com/systemd/systemd/blame/e8dc52766e1fdb4f8c09c3ab654d1270e1090c8d/src/oom/oomd-manager.c#L409 https://github.com/systemd/systemd/blame/e8dc52766e1fdb4f8c0... Yeah, looks like logging was added 1.5 years ago so hard to take his perspective seriously.
- pengaru 4y ago
  It's also problematic that he doesn't seem to have reported the issue to either his distributor or upstream systemd. If he did, it's not mentioned in his rant. There aren't any links to any bug reports where the developers demonstrably failed to care or anything rant-worthy. Considering systemd-oomd is a relatively new addition to the project, contributed by FB staff, it wouldn't surprise me if there were desktop integration issues yet to be sorted out by the various distros. The post doesn't strike me as made in good faith, seems written by someone with an axe to grind with systemd.

eternityforest 4y ago

Systemd is wonderful. But could they please stop trying to invent lightweight replacements? You're not suckless. Nobody wants you to be suckless. I like you specifically because you aren't anything like suckless. Don't replace anything unless you have close to full feature parity. Nobody wants your minimum viable product components. The people who do don't want systemd. Init system? Absolutely. Timers? You've done a wonderful job. Time sync? Please no, crony does it better.

Operative0198 4y ago

So many things wrong with this article. You can absolutely see which programs triggered systemd-oomd into action in journalctl logs. Even better, I now seem to get notifications with the culprit whenever it kicks in for my desktop setup. I can also see the design decision made to kill the graphical session when the system becomes unresponsive. The alternative to that situation pre-systemd-oomd was a force reboot of the system.