6 ms·

Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use

by hpcjoe 9y ago

Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image. This is just a small sampling of the open source build system, that I grabbed from some of the configs we used. Yes, systemd is broken, but not irretrievably. It is fixable. # fix some systemd timeout brokenness sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf # fix systemd journaling. Yeah, really sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf # fix the INSANE logind.conf per user directory size ... hard code it to 256M sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf # # fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ... sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf # mask off systemd-udev-settle ... yes it is broken chroot ${TARGET} systemctl mask systemd-udev-settle [edited for formatting]

lima 9y ago
> Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen. I agree about it being less flexible. Either way, it's a separate daemon not tied to systemd. > fix some systemd timeout brokenness Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s). ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware watchdog reboot your system may not always be a good idea. > fix systemd journaling Debian runs a non-persistent runtime journal by default and leaves the long-term storage responsibility with the syslog daemon. What's wrong with this? If anything, they're being too conservative. > RuntimeDirectorySize What's wrong with the 10% of physical RAM default? Hard coding this to a small value is potentially messing with things like flatpak's portals which store large-ish data there. > KillUserProcesses Off by default in Debian. I specifically turn it on, since it kills any rogue user processes when they log out and makes sure nobody leaves anything running in a screen instead of doing it properly (I've even seen a crashed vim instance max out CPU on a server) > mask off systemd-udev-settle ... yes it is broken Masking this introduces race conditions into your boot process. The systemd-udev-settle unit calls "udevadm settle", which waits for all pending udev actions to complete before any services are started which may depend on it. If anything is broken, it's the particular subsystem which takes too long to initialize/does not support async events. Important examples are LVM and iSCSI. If you mask the settle unit, systemd won't wait for all devices to be ready. This may or may not work and the safe default is to wait.
- hpcjoe 9y ago
  > Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen. The last job developed/supported Debian based high performance storage systems in production, starting with Debian 7, then Debian 8, and Debian 9. root@localdns:~# cat /etc/debian_version 8.10 root@localdns:~# locate systemd-networkd /lib/systemd/systemd-networkd /lib/systemd/systemd-networkd-wait-online /lib/systemd/system/systemd-networkd-wait-online.service /lib/systemd/system/systemd-networkd.service /usr/share/man/man8/systemd-networkd-wait-online.8.gz /usr/share/man/man8/systemd-networkd-wait-online.service.8.gz /usr/share/man/man8/systemd-networkd.8.gz /usr/share/man/man8/systemd-networkd.service.8.gz Its there, but I disabled it in this case. > Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s). This comes from long debugging/tracing sessions where we were trying to understand why we wound up with effectively repeating timeouts. [...] On systemd-udev-settle, allowing it to run was actually the cause of a race condition on start. The only way to fix this that I found was to disable this. Once we did this, no more race condition. The system booted normally. It sounds like your experience is significantly different than mine on this. We built very large high performance computing and storage systems for many users, where the things that people used simply had to work, without surprise. Every line I showed came from a set of long and often painful debugging sessions where we had to figure out what new change had broken a key bit of software. The RuntimeDirectorySize issue actually tickled another bug, and caused some incredible paging problems when used with a specific software package. Enough end users on a system (we had many) could launch an effective DoS against the system by using these tmps. Generally, my observation, and I've seen others make a similar set elsewhere, is that the default and general distribution settings for systemd parameters appear to be generally set for desktop users (single or very few users per machine). Not for large shared resources.
  - someone12345 9y ago
    systemd-networkd is not enabled by default (even when its binaries might be installed).

by hpcjoe 9y ago

lima 9y ago

> Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen. I agree about it being less flexible. Either way, it's a separate daemon not tied to systemd. > fix some systemd timeout brokenness Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s). ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware watchdog reboot your system may not always be a good idea. > fix systemd journaling Debian runs a non-persistent runtime journal by default and leaves the long-term storage responsibility with the syslog daemon. What's wrong with this? If anything, they're being too conservative. > RuntimeDirectorySize What's wrong with the 10% of physical RAM default? Hard coding this to a small value is potentially messing with things like flatpak's portals which store large-ish data there. > KillUserProcesses Off by default in Debian. I specifically turn it on, since it kills any rogue user processes when they log out and makes sure nobody leaves anything running in a screen instead of doing it properly (I've even seen a crashed vim instance max out CPU on a server) > mask off systemd-udev-settle ... yes it is broken Masking this introduces race conditions into your boot process. The systemd-udev-settle unit calls "udevadm settle", which waits for all pending udev actions to complete before any services are started which may depend on it. If anything is broken, it's the particular subsystem which takes too long to initialize/does not support async events. Important examples are LVM and iSCSI. If you mask the settle unit, systemd won't wait for all devices to be ready. This may or may not work and the safe default is to wait.

hpcjoe 9y ago
> Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen. The last job developed/supported Debian based high performance storage systems in production, starting with Debian 7, then Debian 8, and Debian 9. root@localdns:~# cat /etc/debian_version 8.10 root@localdns:~# locate systemd-networkd /lib/systemd/systemd-networkd /lib/systemd/systemd-networkd-wait-online /lib/systemd/system/systemd-networkd-wait-online.service /lib/systemd/system/systemd-networkd.service /usr/share/man/man8/systemd-networkd-wait-online.8.gz /usr/share/man/man8/systemd-networkd-wait-online.service.8.gz /usr/share/man/man8/systemd-networkd.8.gz /usr/share/man/man8/systemd-networkd.service.8.gz Its there, but I disabled it in this case. > Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s). This comes from long debugging/tracing sessions where we were trying to understand why we wound up with effectively repeating timeouts. [...] On systemd-udev-settle, allowing it to run was actually the cause of a race condition on start. The only way to fix this that I found was to disable this. Once we did this, no more race condition. The system booted normally. It sounds like your experience is significantly different than mine on this. We built very large high performance computing and storage systems for many users, where the things that people used simply had to work, without surprise. Every line I showed came from a set of long and often painful debugging sessions where we had to figure out what new change had broken a key bit of software. The RuntimeDirectorySize issue actually tickled another bug, and caused some incredible paging problems when used with a specific software package. Enough end users on a system (we had many) could launch an effective DoS against the system by using these tmps. Generally, my observation, and I've seen others make a similar set elsewhere, is that the default and general distribution settings for systemd parameters appear to be generally set for desktop users (single or very few users per machine). Not for large shared resources.
- someone12345 9y ago
  systemd-networkd is not enabled by default (even when its binaries might be installed).