10 ms·

Running systemd without systemd-journald

78 points by axy 4y ago

pengaru 4y ago
PSA: systemd-journald uses shared file-backed mappings via mmap() for its journal IO. You must subtract its shared memory use from its resident memory use before judging how much memory it's consuming. The file-backed shared mappings are reclaimable, because they are file-backed. The kernel will just evict the mapped journal pages at will, since they can always be faulted back in from the filesystem. TFA is much ado about nothing, learn to measure memory use properly before breaking out the pitch forks. Full disclosure: I've hacked a bunch on journald upstream.
- axy 4y ago
  Memory use has always been a mystery to me and I can easily miss some things. Thanks for pointing to. Anyway, the right solution for me is tldr, all the rest is shit.
  - iforgotpassword 4y ago
    Yeah, the rest just shows that op likes to do things like they've always done it. How you can prefer to poke around syslog and ps output to determine the state of a service instead if just doing systemctl status is beyond me for example.
    - djbusby 4y ago
      That (syslog,ps) method was likely formed by habit during the many years before systemctl existed.
    - axy 4y ago
      Because systemctl status is called by monitoring tool. It's a habit, yes. If monitoring shows the service is down, no point to manually use systemctl, and syslog with ps become best friends. And sometimes date command as well. PIs don't have hardware clocks and wrong date may lead to errors that look mysterious.
- bragr 4y ago
  I was wondering about this exact thing when the article didn't break down the ram usage. Thanks!
  - viraptor 4y ago
    I was waiting for the author to check the memory usage of rsyslog and becoming enlightened... but it didn't happen. Reminder: check your assumptions/result after changes. He could learn that rsyslog uses way more shared memory than journald (>900M on my system) and it doesn't matter.
- jeroenhd 4y ago
  This is true, but the author seems to be running their services on several Raspberry Pi like devices whose flash storage may be unstable or quick to wear out. Eliminating unnecessary writes and swap space (depending on the application), those megabytes of extra memory may be just enough what tricks the system into committing memory into swap. You can run quite a lot in 512MB of RAM if you use the right languages to write code in. I was surprised about how little RAM my moderately complex daemon written in Rust uses, for example; I expected to have to allocate a gigabyte of RAM to the VM running it (based on what other tools similar to what I was doing needed) but the entire system turned out to be quite comfortable with just a quarter of that. I didn't even try to optimise for memory usage, which is what made this so surprising. I stil had to give it some more RAM because unattended upgrades tended to get stuck, but I learned a lesson that day. Ever since I've been meaning to try to mess with Firecracker + bare bones daemons to run virtual machines services with absolutely minimal overhead. I like the virtualisation boundaries from a security standpoint much more than container boundaries and now I wonder how much I can shrink my overhead by.
  - pengaru 4y ago
    If you're concerned about storage wear you'd just run journald without /var/log/journal so it's volatile (tmpfs) only. At least that way you still have journals for your current boot and functionality like `systemctl status $service` can still tell you some journal information.
    - Spivak 4y ago
      Yeah this is a lot of work to avoid reading journald.conf, switching the storage to volatile, and capping the memory usage to whatever you want.
  - Redoubts 4y ago
    > This is true, but the author seems to be running their services on several Raspberry Pi like devices whose flash storage may be unstable or quick to wear out. Eliminating unnecessary writes and swap space (depending on the application), those megabytes of extra memory may be just enough what tricks the system into committing memory into swap. Well the author seems to want text logs instead, which seems much much worse for this.
- hedora 4y ago
  Did the Linux kernel ever fix the thing where it evicts code pages at the same priority as files mapped read/write? I haven't checked in the last 4 years or so, but, before that, every time I've worked with a Linux-based storage system that used mmap to write to files, I've ended up rewriting it to use pread/pwrite. Each time, there was no perceptible CPU hit, but there was a massive page cache / memory pressure win. It turns out that aggressively evicting warm code pages then faulting them back in is bad for system performance, even with a fast SSD.
  - jcalvinowens 4y ago
    There's nothing to "fix" here, in some cases what you want is not optimal. It is perfectly reasonable for the kernel to prioritize data pages you touched more recently than code pages by default. It's essentially a big LRU, always has been. If you don't like that, you can always use mlock(). You can also tune things like writeback sysctls and readahead behavior. But I disagree it's "broken" because it doesn't do what you want by default.
  - pengaru 4y ago
    In a post-spectre/meltdown world syscalls are a bit more expensive, you'd be hard-pressed to compete with the journal's mmap windows especially for a warm page cache, using pread/pwrite. Especially if you just went naively about it and tried turning every little object access into its own little island of buffered IO. The objects in the journal are quite small, so you'd likely end up having to implement your own page cache/buffer manager in userspace to coalesce the syscalls. It'd be far more interesting to explore an io_uring based implementation IMNSHO.
- quotemstr 4y ago
  It's not quite that simple: while clean file-backed pages are cheaper than, say, private dirty pages (which the kernel must preserve as long as anyone references them), they're not free: you're still paying an opportunity cost. That is, the kernel is, at least for a time, keeping each clean file-backed page resident when it could be keeping some other page, perhaps a more useful one, in RAM instead. If systemd-journald is append-mostly, it'd be useful to MADV_FREE (after msync) any pages behind the current write pointer so as to give the kernel a hint that it can get rid of those clean file-backed pages early. I'd actually suggest getting rid of the use of memory mapping entirely, but doing so would likely be a bigger ask.
  - viraptor 4y ago
    > the kernel is, at least for a time, keeping each clean file-backed page resident when it could be keeping some other page It's almost the same result for the standard page cache when you're reading a file, isn't it?
frankharv 4y ago
I have but one word for this guy. Devuan I see no reason not to consider it. He is a Debian user who hates the way systemd works. Devuan is for you.
- seclorum_wien 4y ago
  Three cheers for Devuan! :). Completely agree, this would've been an easy distro switch-a-rooni ..
  - egberts1 4y ago
    I switched all my Debian to Devuan. Havent looked back.
- TacticalCoder 4y ago
  It depends what you're doing but for my "workstation", Devuan is indeed working flawlessly. I'm running it since six months or so on my main PC (a little Ryzen 3700X / 32 GB or RAM) and I'm very happy with it. Now if you're a sysadmin and have come to rely on systemd and are now locked in, Devuan is obviously not for you. But if you're running Linux on a desktop or on a laptop and aren't a fan of systemd, Devuan is great.
- dark-star 4y ago
  He doesn't even know how to read man pages (journald.conf(5) describes the setup he is looking for), and he doesn't know how memory usage on Linux works (shared /mmap()ed memory is counted for each process). Do you really think his issues go away when he switches to Devuan? He'll probably just yell at different clouds Other than that, Devuan is a solid choice for people who want to get rid of systemd. It comes with the Debian-typical rather old versions of most programs, but I guess for a file server it doesn't matter much if you run kernel 4.19 or 5.15
  - Arch-TK 4y ago
    The systemd man pages are a book. And, just for avoidance of doubt, that's not a good thing. I am never surprised when someone can't find out how to configure systemd to do what they want. It's just too enterprise grade.
db48x 4y ago
Weird; the systemd journal is the feature I want most! It would be the last thing I would ever consider disabling.
- MrStonedOne 4y ago
- heretogetout 4y ago
  I don't know if this is still an issue but the last time I used journald the logs would occasionally become corrupted and journalctl would refuse to read them. The fix was to just delete the logs. I have no idea how logging got so screwed up that corruption in part of the file could make the rest of the log file unreadable. I mean, it's a journal, it's right in the name. Ever since then I switched to rsyslogd and the like. Rock solid.
  - viraptor 4y ago
    Keep in mind that rsyslog doesn't even attempt to verify logs. An alternative explanation is: my system is corrupting logs, I changed to a logging daemon which doesn't tell me about it. I mean, there could definitely be a bug in journald, but I haven't seen any fixes mentioned in changelog for the last 5 years and if it was happening in standard usage, people would notice. For recovering corrupted logs - you can still "less" them as usual. They have some extra markers, but the text is available as text. Journalctl has some special options for that too.
    - heretogetout 4y ago
      IIRC when I experienced this the logs were all in some binary format that couldn't easily be less'd. tbh I didn't do much investigation other than to see the "delete all your logs" resolution suggestion. There could have been a better option.
    - lokar 4y ago
      The fact that it can't seem to recover from a few bad records and gives up on the whole file demonstrates what terrible software it is.
saurik 4y ago
Would doing something like this work around the "journald drops the most important error messages" issue that has been known/outstanding for ten years (bug moved to GitHub six years ago), or is that more of a fundamental design mistake in systemd itself? https://github.com/systemd/systemd/issues/2913 https://github.com/systemd/systemd/issues/2913 https://bugs.freedesktop.org/show_bug.cgi?id=50184 https://bugs.freedesktop.org/show_bug.cgi?id=50184
- frankjr 4y ago
  It's not accurate to say it "drops" error messages. The bug causes these messages to not be attributed to a particular unit - you can still see them with `journalctl` but not with `journalctl -u foo`. Still pretty annoying and should absolutely be fixed (although I'm not sure if systemd is the right place to do it).
  - puffoflogic 4y ago
    > although I'm not sure if systemd is the right place to do it It's truly impressive how systemd turns out the be the right place to do absolutely anything and everything from bootloader to init to dhcp to ntp to network shares... except fix systemd bugs. systemd doesn't seem to be the right place to do that ever. Someone else needs to do it. For another example of this phenomenon, see the nohup bug.
  - saurik 4y ago
    This is actually extremely useful for me to know, so thank you so much for pointing this out!
rob_c 4y ago
Can we just admit jourald is a thorn in the side of people taking systems seriously. The default behaviour even on polished distros is just bad and it's mainly because of the mindsets behind it...
enriquto 4y ago
Spam omelette with spam sausage and spam does not have too much spam in it!
andrewstuart 4y ago
systemd is incredibly useful and powerful. If you are a developer on Linux then you really owe it to yourself to learn as much as you can about systemd. If you really understand systemd then you'll find yourself architecting your software around its capabilities. systemd, if you understand it, can mean you can completely avoid large chunks of development you might otherwise have assumed you need to do. socket activated services, nspawn, traffic accounting, the list of juicy goodness goes on and on.... Ignore the haters, they wouldn't hate if they dedicated their energy to understanding systemd instead of hating on it. In 2022 you are a really an incomplete full stack developer if systemd is not one of the technologies you know very well.
- coredog64 4y ago
  > systemd is incredibly useful and powerful. In that one sentence I think you've highlighted what garners hate from the haters. It's powerful, and it's great when that power is used for good. Those times when it's not is where people get agitated.
  - qbasic_forever 4y ago
    Do you have an example of when systemd is not used for good?
    - egberts1 4y ago
      it kills production daemons having long running stateful data whenever the netdev goes offline., systemd-networkd, that is.
      - viraptor 4y ago
        Only if you configure it to do that. You can use Wants= and After= for your service to start it after the network, but not restart otherwise. The behaviour you describe is not unreasonable to want: If your network device goes away, what exactly are you binding the socket to? How are you restoring the listening when it comes back up? But it's up to you to say which one you want.
midislack 4y ago
- deleted 4y ago
  [deleted]
- AnonymousPlanet 4y ago
  What are you using instead?
  - dijit 4y ago
    I use runit when I need process supervision from my init: https://wiki.archlinux.org/title/runit https://wiki.archlinux.org/title/runit Otherwise I’ve been using sinit with starlark based startup scripts. (Startup scripts can be any language in normal units, not just bash).
  - midislack 4y ago
    rc
m463 4y ago
"when I needed systemd binary logs? - and I realized I never needed them." that is my sentiment. Linus was hesitant about binary logs. I think they are just not unix. They are doing the wrong thing correctly (I would prefer doing the right thing poorly - or better yet doing it well)

78 points by axy 4y ago

PSA: systemd-journald uses shared file-backed mappings via mmap() for its journal IO. You must subtract its shared memory use from its resident memory use before judging how much memory it's consuming. The file-backed shared mappings are reclaimable, because they are file-backed. The kernel will just evict the mapped journal pages at will, since they can always be faulted back in from the filesystem. TFA is much ado about nothing, learn to measure memory use properly before breaking out the pitch forks. Full disclosure: I've hacked a bunch on journald upstream.

axy 4y ago
Memory use has always been a mystery to me and I can easily miss some things. Thanks for pointing to. Anyway, the right solution for me is tldr, all the rest is shit.
- iforgotpassword 4y ago
  Yeah, the rest just shows that op likes to do things like they've always done it. How you can prefer to poke around syslog and ps output to determine the state of a service instead if just doing systemctl status is beyond me for example.
  - djbusby 4y ago
    That (syslog,ps) method was likely formed by habit during the many years before systemctl existed.
  - axy 4y ago
    Because systemctl status is called by monitoring tool. It's a habit, yes. If monitoring shows the service is down, no point to manually use systemctl, and syslog with ps become best friends. And sometimes date command as well. PIs don't have hardware clocks and wrong date may lead to errors that look mysterious.
bragr 4y ago
I was wondering about this exact thing when the article didn't break down the ram usage. Thanks!
- viraptor 4y ago
  I was waiting for the author to check the memory usage of rsyslog and becoming enlightened... but it didn't happen. Reminder: check your assumptions/result after changes. He could learn that rsyslog uses way more shared memory than journald (>900M on my system) and it doesn't matter.
jeroenhd 4y ago
This is true, but the author seems to be running their services on several Raspberry Pi like devices whose flash storage may be unstable or quick to wear out. Eliminating unnecessary writes and swap space (depending on the application), those megabytes of extra memory may be just enough what tricks the system into committing memory into swap. You can run quite a lot in 512MB of RAM if you use the right languages to write code in. I was surprised about how little RAM my moderately complex daemon written in Rust uses, for example; I expected to have to allocate a gigabyte of RAM to the VM running it (based on what other tools similar to what I was doing needed) but the entire system turned out to be quite comfortable with just a quarter of that. I didn't even try to optimise for memory usage, which is what made this so surprising. I stil had to give it some more RAM because unattended upgrades tended to get stuck, but I learned a lesson that day. Ever since I've been meaning to try to mess with Firecracker + bare bones daemons to run virtual machines services with absolutely minimal overhead. I like the virtualisation boundaries from a security standpoint much more than container boundaries and now I wonder how much I can shrink my overhead by.
- pengaru 4y ago
  If you're concerned about storage wear you'd just run journald without /var/log/journal so it's volatile (tmpfs) only. At least that way you still have journals for your current boot and functionality like `systemctl status $service` can still tell you some journal information.
  - Spivak 4y ago
    Yeah this is a lot of work to avoid reading journald.conf, switching the storage to volatile, and capping the memory usage to whatever you want.
- Redoubts 4y ago
  > This is true, but the author seems to be running their services on several Raspberry Pi like devices whose flash storage may be unstable or quick to wear out. Eliminating unnecessary writes and swap space (depending on the application), those megabytes of extra memory may be just enough what tricks the system into committing memory into swap. Well the author seems to want text logs instead, which seems much much worse for this.
hedora 4y ago
Did the Linux kernel ever fix the thing where it evicts code pages at the same priority as files mapped read/write? I haven't checked in the last 4 years or so, but, before that, every time I've worked with a Linux-based storage system that used mmap to write to files, I've ended up rewriting it to use pread/pwrite. Each time, there was no perceptible CPU hit, but there was a massive page cache / memory pressure win. It turns out that aggressively evicting warm code pages then faulting them back in is bad for system performance, even with a fast SSD.
- jcalvinowens 4y ago
  There's nothing to "fix" here, in some cases what you want is not optimal. It is perfectly reasonable for the kernel to prioritize data pages you touched more recently than code pages by default. It's essentially a big LRU, always has been. If you don't like that, you can always use mlock(). You can also tune things like writeback sysctls and readahead behavior. But I disagree it's "broken" because it doesn't do what you want by default.
- pengaru 4y ago
  In a post-spectre/meltdown world syscalls are a bit more expensive, you'd be hard-pressed to compete with the journal's mmap windows especially for a warm page cache, using pread/pwrite. Especially if you just went naively about it and tried turning every little object access into its own little island of buffered IO. The objects in the journal are quite small, so you'd likely end up having to implement your own page cache/buffer manager in userspace to coalesce the syscalls. It'd be far more interesting to explore an io_uring based implementation IMNSHO.
quotemstr 4y ago
It's not quite that simple: while clean file-backed pages are cheaper than, say, private dirty pages (which the kernel must preserve as long as anyone references them), they're not free: you're still paying an opportunity cost. That is, the kernel is, at least for a time, keeping each clean file-backed page resident when it could be keeping some other page, perhaps a more useful one, in RAM instead. If systemd-journald is append-mostly, it'd be useful to MADV_FREE (after msync) any pages behind the current write pointer so as to give the kernel a hint that it can get rid of those clean file-backed pages early. I'd actually suggest getting rid of the use of memory mapping entirely, but doing so would likely be a bigger ask.
- viraptor 4y ago
  > the kernel is, at least for a time, keeping each clean file-backed page resident when it could be keeping some other page It's almost the same result for the standard page cache when you're reading a file, isn't it?

frankharv 4y ago

I have but one word for this guy. Devuan I see no reason not to consider it. He is a Debian user who hates the way systemd works. Devuan is for you.

seclorum_wien 4y ago
Three cheers for Devuan! :). Completely agree, this would've been an easy distro switch-a-rooni ..
- egberts1 4y ago
  I switched all my Debian to Devuan. Havent looked back.
TacticalCoder 4y ago
It depends what you're doing but for my "workstation", Devuan is indeed working flawlessly. I'm running it since six months or so on my main PC (a little Ryzen 3700X / 32 GB or RAM) and I'm very happy with it. Now if you're a sysadmin and have come to rely on systemd and are now locked in, Devuan is obviously not for you. But if you're running Linux on a desktop or on a laptop and aren't a fan of systemd, Devuan is great.
dark-star 4y ago
He doesn't even know how to read man pages (journald.conf(5) describes the setup he is looking for), and he doesn't know how memory usage on Linux works (shared /mmap()ed memory is counted for each process). Do you really think his issues go away when he switches to Devuan? He'll probably just yell at different clouds Other than that, Devuan is a solid choice for people who want to get rid of systemd. It comes with the Debian-typical rather old versions of most programs, but I guess for a file server it doesn't matter much if you run kernel 4.19 or 5.15
- Arch-TK 4y ago
  The systemd man pages are a book. And, just for avoidance of doubt, that's not a good thing. I am never surprised when someone can't find out how to configure systemd to do what they want. It's just too enterprise grade.

db48x 4y ago

Weird; the systemd journal is the feature I want most! It would be the last thing I would ever consider disabling.

MrStonedOne 4y ago
heretogetout 4y ago
I don't know if this is still an issue but the last time I used journald the logs would occasionally become corrupted and journalctl would refuse to read them. The fix was to just delete the logs. I have no idea how logging got so screwed up that corruption in part of the file could make the rest of the log file unreadable. I mean, it's a journal, it's right in the name. Ever since then I switched to rsyslogd and the like. Rock solid.
- viraptor 4y ago
  Keep in mind that rsyslog doesn't even attempt to verify logs. An alternative explanation is: my system is corrupting logs, I changed to a logging daemon which doesn't tell me about it. I mean, there could definitely be a bug in journald, but I haven't seen any fixes mentioned in changelog for the last 5 years and if it was happening in standard usage, people would notice. For recovering corrupted logs - you can still "less" them as usual. They have some extra markers, but the text is available as text. Journalctl has some special options for that too.
  - heretogetout 4y ago
    IIRC when I experienced this the logs were all in some binary format that couldn't easily be less'd. tbh I didn't do much investigation other than to see the "delete all your logs" resolution suggestion. There could have been a better option.
  - lokar 4y ago
    The fact that it can't seem to recover from a few bad records and gives up on the whole file demonstrates what terrible software it is.

saurik 4y ago

Would doing something like this work around the "journald drops the most important error messages" issue that has been known/outstanding for ten years (bug moved to GitHub six years ago), or is that more of a fundamental design mistake in systemd itself? https://github.com/systemd/systemd/issues/2913 https://github.com/systemd/systemd/issues/2913 https://bugs.freedesktop.org/show_bug.cgi?id=50184 https://bugs.freedesktop.org/show_bug.cgi?id=50184

frankjr 4y ago
It's not accurate to say it "drops" error messages. The bug causes these messages to not be attributed to a particular unit - you can still see them with `journalctl` but not with `journalctl -u foo`. Still pretty annoying and should absolutely be fixed (although I'm not sure if systemd is the right place to do it).
- puffoflogic 4y ago
  > although I'm not sure if systemd is the right place to do it It's truly impressive how systemd turns out the be the right place to do absolutely anything and everything from bootloader to init to dhcp to ntp to network shares... except fix systemd bugs. systemd doesn't seem to be the right place to do that ever. Someone else needs to do it. For another example of this phenomenon, see the nohup bug.
- saurik 4y ago
  This is actually extremely useful for me to know, so thank you so much for pointing this out!

rob_c 4y ago

Can we just admit jourald is a thorn in the side of people taking systems seriously. The default behaviour even on polished distros is just bad and it's mainly because of the mindsets behind it...

enriquto 4y ago

Spam omelette with spam sausage and spam does not have too much spam in it!

andrewstuart 4y ago

systemd is incredibly useful and powerful. If you are a developer on Linux then you really owe it to yourself to learn as much as you can about systemd. If you really understand systemd then you'll find yourself architecting your software around its capabilities. systemd, if you understand it, can mean you can completely avoid large chunks of development you might otherwise have assumed you need to do. socket activated services, nspawn, traffic accounting, the list of juicy goodness goes on and on.... Ignore the haters, they wouldn't hate if they dedicated their energy to understanding systemd instead of hating on it. In 2022 you are a really an incomplete full stack developer if systemd is not one of the technologies you know very well.

coredog64 4y ago
> systemd is incredibly useful and powerful. In that one sentence I think you've highlighted what garners hate from the haters. It's powerful, and it's great when that power is used for good. Those times when it's not is where people get agitated.
- qbasic_forever 4y ago
  Do you have an example of when systemd is not used for good?
  - egberts1 4y ago
    it kills production daemons having long running stateful data whenever the netdev goes offline., systemd-networkd, that is.
    - viraptor 4y ago
      Only if you configure it to do that. You can use Wants= and After= for your service to start it after the network, but not restart otherwise. The behaviour you describe is not unreasonable to want: If your network device goes away, what exactly are you binding the socket to? How are you restoring the listening when it comes back up? But it's up to you to say which one you want.

midislack 4y ago

deleted 4y ago
[deleted]
AnonymousPlanet 4y ago
What are you using instead?
- dijit 4y ago
  I use runit when I need process supervision from my init: https://wiki.archlinux.org/title/runit https://wiki.archlinux.org/title/runit Otherwise I’ve been using sinit with starlark based startup scripts. (Startup scripts can be any language in normal units, not just bash).
- midislack 4y ago
  rc

m463 4y ago

"when I needed systemd binary logs? - and I realized I never needed them." that is my sentiment. Linus was hesitant about binary logs. I think they are just not unix. They are doing the wrong thing correctly (I would prefer doing the right thing poorly - or better yet doing it well)