3 ms·

I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical stor

by hpcjoe 9y ago

I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical storage in a single box. Linked together via infiniband, 40/100 GbE, and others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS). Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well. The greatest pain was the networking. Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking. Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases. My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works. I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over. Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.

lima 9y ago
> The greatest pain was the networking. Last time I checked, systemd - the init system - wasn't involved with networking at all. There's systemd-networkd, but that's an optional component and no major distro is actually using it as a default. RHEL 7 - which runs systemd - supports both NetworkManager and their legacy networking scripts. You said there were "myriad of shortcomings" and "egregious settings", some examples would help.
- hpcjoe 9y ago
  Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image. This is just a small sampling of the open source build system, that I grabbed from some of the configs we used. Yes, systemd is broken, but not irretrievably. It is fixable. # fix some systemd timeout brokenness sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf # fix systemd journaling. Yeah, really sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf # fix the INSANE logind.conf per user directory size ... hard code it to 256M sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf # # fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ... sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf # mask off systemd-udev-settle ... yes it is broken chroot ${TARGET} systemctl mask systemd-udev-settle [edited for formatting]
  - lima 9y ago
    > Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen. I agree about it being less flexible. Either way, it's a separate daemon not tied to systemd. > fix some systemd timeout brokenness Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s). ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware watchdog reboot your system may not always be a good idea. > fix systemd journaling Debian runs a non-persistent runtime journal by default and leaves the long-term storage responsibility with the syslog daemon. What's wrong with this? If anything, they're being too conservative. > RuntimeDirectorySize What's wrong with the 10% of physical RAM default? Hard coding this to a small value is potentially messing with things like flatpak's portals which store large-ish data there. > KillUserProcesses Off by default in Debian. I specifically turn it on, since it kills any rogue user processes when they log out and makes sure nobody leaves anything running in a screen instead of doing it properly (I've even seen a crashed vim instance max out CPU on a server) > mask off systemd-udev-settle ... yes it is broken Masking this introduces race conditions into your boot process. The systemd-udev-settle unit calls "udevadm settle", which waits for all pending udev actions to complete before any services are started which may depend on it. If anything is broken, it's the particular subsystem which takes too long to initialize/does not support async events. Important examples are LVM and iSCSI. If you mask the settle unit, systemd won't wait for all devices to be ready. This may or may not work and the safe default is to wait.

by hpcjoe 9y ago

lima 9y ago

> The greatest pain was the networking. Last time I checked, systemd - the init system - wasn't involved with networking at all. There's systemd-networkd, but that's an optional component and no major distro is actually using it as a default. RHEL 7 - which runs systemd - supports both NetworkManager and their legacy networking scripts. You said there were "myriad of shortcomings" and "egregious settings", some examples would help.

hpcjoe 9y ago
Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image. This is just a small sampling of the open source build system, that I grabbed from some of the configs we used. Yes, systemd is broken, but not irretrievably. It is fixable. # fix some systemd timeout brokenness sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf # fix systemd journaling. Yeah, really sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf # fix the INSANE logind.conf per user directory size ... hard code it to 256M sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf # # fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ... sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf # mask off systemd-udev-settle ... yes it is broken chroot ${TARGET} systemctl mask systemd-udev-settle [edited for formatting]
- lima 9y ago
  > Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen. I agree about it being less flexible. Either way, it's a separate daemon not tied to systemd. > fix some systemd timeout brokenness Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s). ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware watchdog reboot your system may not always be a good idea. > fix systemd journaling Debian runs a non-persistent runtime journal by default and leaves the long-term storage responsibility with the syslog daemon. What's wrong with this? If anything, they're being too conservative. > RuntimeDirectorySize What's wrong with the 10% of physical RAM default? Hard coding this to a small value is potentially messing with things like flatpak's portals which store large-ish data there. > KillUserProcesses Off by default in Debian. I specifically turn it on, since it kills any rogue user processes when they log out and makes sure nobody leaves anything running in a screen instead of doing it properly (I've even seen a crashed vim instance max out CPU on a server) > mask off systemd-udev-settle ... yes it is broken Masking this introduces race conditions into your boot process. The systemd-udev-settle unit calls "udevadm settle", which waits for all pending udev actions to complete before any services are started which may depend on it. If anything is broken, it's the particular subsystem which takes too long to initialize/does not support async events. Important examples are LVM and iSCSI. If you mask the settle unit, systemd won't wait for all devices to be ready. This may or may not work and the safe default is to wait.