6 ms·

Question: did you every try to start a system with > 60TB storage attached with systemd?

lima 9y ago
Maybe he didn't, but I did, and it works fine.
hpcjoe 9y ago
I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical storage in a single box. Linked together via infiniband, 40/100 GbE, and others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS). Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well. The greatest pain was the networking. Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking. Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases. My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works. I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over. Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.
- lima 9y ago
  > The greatest pain was the networking. Last time I checked, systemd - the init system - wasn't involved with networking at all. There's systemd-networkd, but that's an optional component and no major distro is actually using it as a default. RHEL 7 - which runs systemd - supports both NetworkManager and their legacy networking scripts. You said there were "myriad of shortcomings" and "egregious settings", some examples would help.
  - hpcjoe 9y ago
    Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image. This is just a small sampling of the open source build system, that I grabbed from some of the configs we used. Yes, systemd is broken, but not irretrievably. It is fixable. # fix some systemd timeout brokenness sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf # fix systemd journaling. Yeah, really sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf # fix the INSANE logind.conf per user directory size ... hard code it to 256M sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf # # fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ... sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf # mask off systemd-udev-settle ... yes it is broken chroot ${TARGET} systemctl mask systemd-udev-settle [edited for formatting]

lima 9y ago

Maybe he didn't, but I did, and it works fine.

I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical storage in a single box. Linked together via infiniband, 40/100 GbE, and others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS). Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well. The greatest pain was the networking. Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking. Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases. My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works. I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over. Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.

lima 9y ago
> The greatest pain was the networking. Last time I checked, systemd - the init system - wasn't involved with networking at all. There's systemd-networkd, but that's an optional component and no major distro is actually using it as a default. RHEL 7 - which runs systemd - supports both NetworkManager and their legacy networking scripts. You said there were "myriad of shortcomings" and "egregious settings", some examples would help.
- hpcjoe 9y ago
  Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases. On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image. This is just a small sampling of the open source build system, that I grabbed from some of the configs we used. Yes, systemd is broken, but not irretrievably. It is fixable. # fix some systemd timeout brokenness sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf # fix systemd journaling. Yeah, really sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf # fix the INSANE logind.conf per user directory size ... hard code it to 256M sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf # # fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ... sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf # mask off systemd-udev-settle ... yes it is broken chroot ${TARGET} systemctl mask systemd-udev-settle [edited for formatting]