7 ms·

Broken by design: systemd

116 points by mmphosis 13y ago

rjzzleep 13y ago
it seems like the pid 1 issue has been discussed quite a few times in the past. see also here https://lwn.net/Articles/520892/ https://lwn.net/Articles/520892/ it's also discussed in the official debian debate https://wiki.debian.org/Debate/initsystem/systemd https://wiki.debian.org/Debate/initsystem/systemd can anyone explain to me why they haven't just changed systemd to do less work in pid 1? i've used systemd on my archlinux laptop for quite a while and was happy with how much faster the system boots. but i also found it to be too huge and too intertwined for basic maintenance tasks. but ofc that is just a personal opinion. suddenly i have to have a repertoire of systemd commands to do things i used to be able to do with basic unix tools
- rcxdude 13y ago
  I don't know. Given what I know about how cgroups and systemd works, I don't think there's any particular issue with having a thin shim and starting systemd as PID 2, and so getting some of the reliability gains this article talks about. But I don't have all that much knowledge of the details and there may be good technical reasons why that is a bad idea (the man page notes that it is possible via a command line flag, but that it is not supported and only intended for testing). There are some issues surrounding cgroups moving to a single-userspace-arbitrator model, but I don't know what requires this to be PID 1.
  - zanny 13y ago
    The article remarks its pid 1 to catch "bad" services that disown their parents and get defaulted to pid 1. But I figure if they switched the bulk of systemd to pid 2, any negligent services would get complained about until they were fixed.
    - dalias 13y ago
      Almost all daemons have a command line or config file option to skip "daemonizing" so that the parent can watch the PID correctly. For the few that don't, you can fix them, hack them (e.g. LD_PRELOAD wrappers to shutdown their bogus double-forking), seccomp-ptrace them, or simply wrap them with a legacy-style monitor that reads the pid file and forwards signals it receives to the real pid (this last option of course doesn't fix the race conditions inherent in such usage).
    - rcxdude 13y ago
      This is something systemd can already handle. Linux has APIs to change the 'reaper' process (PR_SET_CHILD_SUBREAPER) so that orphaned processes move to a non PID 1 process.
zimbatm 13y ago
systemd is like a second kernel in user-space.
- mmphosis 13y ago
  and, launchd is like a second mach kernel at a higher level. https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/Mach/Mach.html https://developer.apple.com/library/mac/documentation/Darwin...
  - teacup50 13y ago
    Except for the part where launchd doesn't implement anything other than traditional process/service launch and termination, and doesn't violate the layering boundaries between system components. In a launchd system, there's absolutely no dependency on launchd by userspace software.
- zanny 13y ago
  Which should be good. If the OPs arguments about how dangerous it is to empower and stuff a lot of functionality into pid 1, think of how running in kernel mode on a cpu gives you power when your code is in the kernel proper. So moving functionality out of kernel mode and into user mode is great. The only steps after that are moving as much functionality as possible to the user, and then to nobody. We do trend towards minimum permissions for the sake of security for a reason.
  - dalias 13y ago
    Functionality is not being moved out of the kernel and into userspace. It's being moved out of legacy freedesktop.org components (hald, ConsoleKit, PolicyKit, etc.) and into PID 1. Where should it be moved to? /dev/null.
    - nona 13y ago
      I think you need to look a bit more closely at what's actually going into PID 1. See for example: http://people.debian.org/~stapelberg/docs/systemd-dependencies.html http://people.debian.org/~stapelberg/docs/systemd-dependenci... I don't see any logind/hostnamed/journald or any of that stuff in PID1.
ploxiln 13y ago
I've used systemd on Arch Linux for about a year now. It wasn't the end of the world, I can deal with it, but it's not fun. I'm bothered by seemingly built-in units that have no corresponding unit files, which all default to enabled, including units for bootsplash stuff ("plymouth") that isn't used on arch linux. I'm very annoyed by mounts and such that don't even have units, files or otherwise, such as the "pstore" filesystem or autofs4. I'm super annoyed that since an upgrade half a year ago, the configuration option to disable use of some cgroup controllers was removed, and now any cgroup controllers supported by the kernel are used unconditionally. I have good reasons for not using some of these things, but even more than that, one of the main reasons I like to use Arch Linux is that I like to add what I need, rather than try to make sense of and clean up the clutter of a typical operating system later. The lack of control and default-everything-on of systemd is really very annoying, and I'm bothered by the strategies mentioned in this article by which systemd makes itself the one and only practical modern init system for linux.
- Fasebook 13y ago
  It seems to me that there is concerted effort to discredit alternative init implementations. Example: https://en.wikipedia.org/w/index.php?title=Daemontools&action=history https://en.wikipedia.org/w/index.php?title=Daemontools&actio... I can't imagine what agency with a history of undermining technical standards could possibly be involved.
  - pekk 13y ago
    Why don't you just say what you mean.
    - deleted 13y ago
      [deleted]
- Jasper_ 13y ago
  The only Plymouth-specific code I see in systemd is related to asking the user questions at boot-up time, like their mount password, and it works by sending a socket message to a socket plymouth opens, and successfully falls back if it doesn't exist. I don't see any plymouth-related services or unit files hardcoded. I do, however, see a NEWS-related entry from the v186 release: http://cgit.freedesktop.org/systemd/systemd/tree/NEWS#n1775 http://cgit.freedesktop.org/systemd/systemd/tree/NEWS#n1775
nona 13y ago
Not everything of systemd is in PID 1. And yes, what some people consider necessary and important or even essential features missing from sysvinit, others consider bloat. Just to ensure we're all talking about the same thing, here's an overview of systemd's footprint: http://people.debian.org/~stapelberg/docs/systemd-dependencies.html http://people.debian.org/~stapelberg/docs/systemd-dependenci... From the article: "On most machines, systemd will bring from 236 KiB to 550 KiB of additional libraries to be loaded into memory. When recompiling systemd without its optional dependencies (e.g. for embedded systems), this shrinks to 368 KiB in the worst case. The RSS (Resident Set Size) of systemd is 1.8 MiB, whereas sysvinit uses 0.8 MiB — both measured on typical Debian wheezy installations."
TheCondor 13y ago
Why don't the anti-systemd folks publish some actual attacks against that big ol' attack surface? I've been using it for maybe 18 months, it was different and different is some times scary but I have had no problems at all. Migrated some legacy init scripts and it's a piece of cake. It's fast, it works, it does what it claims to do. I've been shipping unix software mostly on linux for 15 years and there are very real reasons systemd, launchd, smf, upstart and other "do more than nothing" init systems exist and companies making money selling them write them. All of them are much much more simple for the administrator to use too...
- dalias 13y ago
  Folks on #musl found 2 integer overflow bugs in the UTF-8 handling code within seconds of checking out the source to read it. It's not clear to me where the code is called from and what inputs reach it, but that kind of bug does not inspire confidence in the safety of the project as a whole. I also quickly found a case where a potentially-null string pointer is passed to a logging function which might or might not be able to accept null string pointers. (I suspect it uses snprintf and therefore depends on the underlying C library's handling of this usage, which is undefined behavior.)
  - Jasper_ 13y ago
    In the interest of transparency, can you tell us exactly what you found? I looked around for a bit but couldn't spot anything.
- einhverfr 13y ago
  The argument is one of minimizing attack surface, not specific current attacks. I.e. it's a question of selecting the right technology security-wise and attack surface is a big part of that equation because with that surface area comes the strong likelihood that attacks will be found against the process over time. I use systemd because that's what my distro comes with. There are positives and negatives. On the whole though, I agree with much of the criticism, and I hope that a real alternative comes along which is more security and uptime friendly.
nly 13y ago
> systemd makes it impossible to upgrade without rebooting. This leads to "Linux" becoming the laughing stock of Windows fans, as happened with Ubuntu a long time ago. This irked me. It's just an unnecessary comparison. The ZDnet article linked is, and always was, completely fallacious in comparing an OS critical update channel to a fully-fledged distro package manager. Having just bought a brand new Windows laptop this weekend, I found myself spending no less than 2 hours tirelessly updating and rebooting Windows, and searching for, and installing, driver updates manually. It's 2014 and it still sucks. Windows update and OEM updaters are, in my experience, piss-poor and don't pick up half of the updates available to your average system. Frankly I think your average Linux desktop could easily risk doubling or tripling the number and frequency of reboots demanded, and you'd still not reach a significant fraction of the annoyance of Windows on that particular front.
- pekk 13y ago
  And all the servers? They are supposed to just keep rebooting all the time as if they were Windows desktops?
  - Jasper_ 13y ago
    If you upgrade systemd or the kernel, yes, you should reboot your machine. Ideally, it shouldn't take more than a minute or two.
    - josephkern 13y ago
      Obviously not talking about servers here. Those RAID can take more than a few minutes to spin-up.
      - btgeekboy 13y ago
        If you can't afford 3 minutes of downtime (how long a Dell PowerEdge R620 takes to start responding to ping again after a shutdown -r now), you really should reconsider running that service without a redundant system at the ready.
mwcampbell 13y ago
It seems to me that, in keeping with its long tradition of being conservative, Debian should put off switching its init system for at least another release, since systemd is by no means the clear winner from a strictly technical point of view.
- notori0us 13y ago
  Agreed here. Then, at that time, there should be a re-evaluation. Perhaps also at that time openRC will be a potential candidate and far less experimental.
falconfunction 13y ago
Was openrc ever a part of the discussion? I don't like the concept of a GUI/Redhats philosophy dictating system behavior when X isn't even installed.
joeyh 13y ago
AFAICS, the arguments about restarting pid 1 on upgrade being a problem also apply to sysvinit, which has been restarting itself on upgrade for 20 years.
- teacup50 13y ago
  Except that sysv init, having tiny attack surface, very rarely needs updates.
pdkl95 13y ago
I'll run systemd only after you pry openrc[1] from my cold, dead hands. FAR too much of the project makes exactly the wrong design choices. The worst of these flaws is the binding of previously-independent components discussed near the end of this article. This bundling Is not even necessary! The features that systemd gets right - an interesting daemon management tool, and providing a useful default start/stop mechanism ("4 lines of config instead of re-writing the same generic shell script") - can obviously be independent features. This mess should be limited to merely bad design in a particular software niche(s)., but now we're seeing the "my way or nothing" attitude spill over into existing user-level software. (e.g., Gnome[2] I would probably already be using part of it was properly modular. Instead, the bundle is growing even more monolithic and incompatible. When a similar "forced upgrade/incompatibility" happened to OpenGL (with ES dropping the traditional fixed-function pipeline, JWZ had a very good summary[3] of the the problem, from a "having to maintain old software" prespective: Let's say you have a well-specified system that is in wide use (a language, a library API, whatever) and because of changes in some substrate (operating systems, hardware, whatever) you find that you need to add a new way of doing things to it. The way you do this is, you add new features to the specification and you clearly document the version in which those features become supported. If there are old features that you would like to discourage the use of, then you mark them as obsolete -- but you do not remove them because thou shalt not break working code. If you don't agree with that, then please, get out of the software industry right now. Find another line of work. Please. Your users have code that works. Maybe the new APIs would serve them better. Maybe things would be so much more efficient if they updated their code to use the new API. Or maybe it doesn't matter to them and they just want working code to continue to be working code. At least until such a time as they need the new features, or new efficiency. Remember the First Rule of Optimization: DON'T. sigh - one of the big reasons behind my I left the world of Microsoft and Windows behind ~15 year ago was to be free of this kind of forced change... 1: Not giving up my .xinitrc/.Xresources or E16 theme, either. In all cases (along with openrc/sysvinit), they have over a decade's worth of bugfixes and tweaks. Any replacement would have to be very good to be worth the time/effort involved in switching. 2: to be fair, Gnome has been merging thing that shouldn't be merged all the way back to the beginning of the project, and is as much at fault here as systemd itself. 3: http://www.jwz.org/blog/2012/06/i-have-ported-xscreensaver-to-the-iphone/ http://www.jwz.org/blog/2012/06/i-have-ported-xscreensaver-t...
gwu78 13y ago
I like this writing. Solid. But, I'm curious: What are the specific deficiencies the author sees in daemontools? "Not ready for primetime." I mean, it's so typical for bloggers to say certain software is this or that (some generalisation), without giving details. I don't know much about systemd. I find Linux too chaotic, a moving target that moves a little too frequently for my tastes. It's changes like these that I find take too much energy to follow. I doubt it's a problem specific to Linux though. As a BSD user, I've been amazed with how much init.c has grown in size over the years. A variety of developers have touched it.

116 points by mmphosis 13y ago

it seems like the pid 1 issue has been discussed quite a few times in the past. see also here https://lwn.net/Articles/520892/ https://lwn.net/Articles/520892/ it's also discussed in the official debian debate https://wiki.debian.org/Debate/initsystem/systemd https://wiki.debian.org/Debate/initsystem/systemd can anyone explain to me why they haven't just changed systemd to do less work in pid 1? i've used systemd on my archlinux laptop for quite a while and was happy with how much faster the system boots. but i also found it to be too huge and too intertwined for basic maintenance tasks. but ofc that is just a personal opinion. suddenly i have to have a repertoire of systemd commands to do things i used to be able to do with basic unix tools

rcxdude 13y ago
I don't know. Given what I know about how cgroups and systemd works, I don't think there's any particular issue with having a thin shim and starting systemd as PID 2, and so getting some of the reliability gains this article talks about. But I don't have all that much knowledge of the details and there may be good technical reasons why that is a bad idea (the man page notes that it is possible via a command line flag, but that it is not supported and only intended for testing). There are some issues surrounding cgroups moving to a single-userspace-arbitrator model, but I don't know what requires this to be PID 1.
- zanny 13y ago
  The article remarks its pid 1 to catch "bad" services that disown their parents and get defaulted to pid 1. But I figure if they switched the bulk of systemd to pid 2, any negligent services would get complained about until they were fixed.
  - dalias 13y ago
    Almost all daemons have a command line or config file option to skip "daemonizing" so that the parent can watch the PID correctly. For the few that don't, you can fix them, hack them (e.g. LD_PRELOAD wrappers to shutdown their bogus double-forking), seccomp-ptrace them, or simply wrap them with a legacy-style monitor that reads the pid file and forwards signals it receives to the real pid (this last option of course doesn't fix the race conditions inherent in such usage).
  - rcxdude 13y ago
    This is something systemd can already handle. Linux has APIs to change the 'reaper' process (PR_SET_CHILD_SUBREAPER) so that orphaned processes move to a non PID 1 process.

zimbatm 13y ago

systemd is like a second kernel in user-space.

mmphosis 13y ago
and, launchd is like a second mach kernel at a higher level. https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/Mach/Mach.html https://developer.apple.com/library/mac/documentation/Darwin...
- teacup50 13y ago
  Except for the part where launchd doesn't implement anything other than traditional process/service launch and termination, and doesn't violate the layering boundaries between system components. In a launchd system, there's absolutely no dependency on launchd by userspace software.
zanny 13y ago
Which should be good. If the OPs arguments about how dangerous it is to empower and stuff a lot of functionality into pid 1, think of how running in kernel mode on a cpu gives you power when your code is in the kernel proper. So moving functionality out of kernel mode and into user mode is great. The only steps after that are moving as much functionality as possible to the user, and then to nobody. We do trend towards minimum permissions for the sake of security for a reason.
- dalias 13y ago
  Functionality is not being moved out of the kernel and into userspace. It's being moved out of legacy freedesktop.org components (hald, ConsoleKit, PolicyKit, etc.) and into PID 1. Where should it be moved to? /dev/null.
  - nona 13y ago
    I think you need to look a bit more closely at what's actually going into PID 1. See for example: http://people.debian.org/~stapelberg/docs/systemd-dependencies.html http://people.debian.org/~stapelberg/docs/systemd-dependenci... I don't see any logind/hostnamed/journald or any of that stuff in PID1.

ploxiln 13y ago

I've used systemd on Arch Linux for about a year now. It wasn't the end of the world, I can deal with it, but it's not fun. I'm bothered by seemingly built-in units that have no corresponding unit files, which all default to enabled, including units for bootsplash stuff ("plymouth") that isn't used on arch linux. I'm very annoyed by mounts and such that don't even have units, files or otherwise, such as the "pstore" filesystem or autofs4. I'm super annoyed that since an upgrade half a year ago, the configuration option to disable use of some cgroup controllers was removed, and now any cgroup controllers supported by the kernel are used unconditionally. I have good reasons for not using some of these things, but even more than that, one of the main reasons I like to use Arch Linux is that I like to add what I need, rather than try to make sense of and clean up the clutter of a typical operating system later. The lack of control and default-everything-on of systemd is really very annoying, and I'm bothered by the strategies mentioned in this article by which systemd makes itself the one and only practical modern init system for linux.

Fasebook 13y ago
It seems to me that there is concerted effort to discredit alternative init implementations. Example: https://en.wikipedia.org/w/index.php?title=Daemontools&action=history https://en.wikipedia.org/w/index.php?title=Daemontools&actio... I can't imagine what agency with a history of undermining technical standards could possibly be involved.
- pekk 13y ago
  Why don't you just say what you mean.
  - deleted 13y ago
    [deleted]
Jasper_ 13y ago
The only Plymouth-specific code I see in systemd is related to asking the user questions at boot-up time, like their mount password, and it works by sending a socket message to a socket plymouth opens, and successfully falls back if it doesn't exist. I don't see any plymouth-related services or unit files hardcoded. I do, however, see a NEWS-related entry from the v186 release: http://cgit.freedesktop.org/systemd/systemd/tree/NEWS#n1775 http://cgit.freedesktop.org/systemd/systemd/tree/NEWS#n1775

nona 13y ago

Not everything of systemd is in PID 1. And yes, what some people consider necessary and important or even essential features missing from sysvinit, others consider bloat. Just to ensure we're all talking about the same thing, here's an overview of systemd's footprint: http://people.debian.org/~stapelberg/docs/systemd-dependencies.html http://people.debian.org/~stapelberg/docs/systemd-dependenci... From the article: "On most machines, systemd will bring from 236 KiB to 550 KiB of additional libraries to be loaded into memory. When recompiling systemd without its optional dependencies (e.g. for embedded systems), this shrinks to 368 KiB in the worst case. The RSS (Resident Set Size) of systemd is 1.8 MiB, whereas sysvinit uses 0.8 MiB — both measured on typical Debian wheezy installations."

TheCondor 13y ago

Why don't the anti-systemd folks publish some actual attacks against that big ol' attack surface? I've been using it for maybe 18 months, it was different and different is some times scary but I have had no problems at all. Migrated some legacy init scripts and it's a piece of cake. It's fast, it works, it does what it claims to do. I've been shipping unix software mostly on linux for 15 years and there are very real reasons systemd, launchd, smf, upstart and other "do more than nothing" init systems exist and companies making money selling them write them. All of them are much much more simple for the administrator to use too...

dalias 13y ago
Folks on #musl found 2 integer overflow bugs in the UTF-8 handling code within seconds of checking out the source to read it. It's not clear to me where the code is called from and what inputs reach it, but that kind of bug does not inspire confidence in the safety of the project as a whole. I also quickly found a case where a potentially-null string pointer is passed to a logging function which might or might not be able to accept null string pointers. (I suspect it uses snprintf and therefore depends on the underlying C library's handling of this usage, which is undefined behavior.)
- Jasper_ 13y ago
  In the interest of transparency, can you tell us exactly what you found? I looked around for a bit but couldn't spot anything.
einhverfr 13y ago
The argument is one of minimizing attack surface, not specific current attacks. I.e. it's a question of selecting the right technology security-wise and attack surface is a big part of that equation because with that surface area comes the strong likelihood that attacks will be found against the process over time. I use systemd because that's what my distro comes with. There are positives and negatives. On the whole though, I agree with much of the criticism, and I hope that a real alternative comes along which is more security and uptime friendly.

nly 13y ago

> systemd makes it impossible to upgrade without rebooting. This leads to "Linux" becoming the laughing stock of Windows fans, as happened with Ubuntu a long time ago. This irked me. It's just an unnecessary comparison. The ZDnet article linked is, and always was, completely fallacious in comparing an OS critical update channel to a fully-fledged distro package manager. Having just bought a brand new Windows laptop this weekend, I found myself spending no less than 2 hours tirelessly updating and rebooting Windows, and searching for, and installing, driver updates manually. It's 2014 and it still sucks. Windows update and OEM updaters are, in my experience, piss-poor and don't pick up half of the updates available to your average system. Frankly I think your average Linux desktop could easily risk doubling or tripling the number and frequency of reboots demanded, and you'd still not reach a significant fraction of the annoyance of Windows on that particular front.

pekk 13y ago
And all the servers? They are supposed to just keep rebooting all the time as if they were Windows desktops?
- Jasper_ 13y ago
  If you upgrade systemd or the kernel, yes, you should reboot your machine. Ideally, it shouldn't take more than a minute or two.
  - josephkern 13y ago
    Obviously not talking about servers here. Those RAID can take more than a few minutes to spin-up.
    - btgeekboy 13y ago
      If you can't afford 3 minutes of downtime (how long a Dell PowerEdge R620 takes to start responding to ping again after a shutdown -r now), you really should reconsider running that service without a redundant system at the ready.

mwcampbell 13y ago

It seems to me that, in keeping with its long tradition of being conservative, Debian should put off switching its init system for at least another release, since systemd is by no means the clear winner from a strictly technical point of view.

notori0us 13y ago
Agreed here. Then, at that time, there should be a re-evaluation. Perhaps also at that time openRC will be a potential candidate and far less experimental.

falconfunction 13y ago

Was openrc ever a part of the discussion? I don't like the concept of a GUI/Redhats philosophy dictating system behavior when X isn't even installed.

joeyh 13y ago

AFAICS, the arguments about restarting pid 1 on upgrade being a problem also apply to sysvinit, which has been restarting itself on upgrade for 20 years.

teacup50 13y ago
Except that sysv init, having tiny attack surface, very rarely needs updates.

pdkl95 13y ago

I'll run systemd only after you pry openrc[1] from my cold, dead hands. FAR too much of the project makes exactly the wrong design choices. The worst of these flaws is the binding of previously-independent components discussed near the end of this article. This bundling Is not even necessary! The features that systemd gets right - an interesting daemon management tool, and providing a useful default start/stop mechanism ("4 lines of config instead of re-writing the same generic shell script") - can obviously be independent features. This mess should be limited to merely bad design in a particular software niche(s)., but now we're seeing the "my way or nothing" attitude spill over into existing user-level software. (e.g., Gnome[2] I would probably already be using part of it was properly modular. Instead, the bundle is growing even more monolithic and incompatible. When a similar "forced upgrade/incompatibility" happened to OpenGL (with ES dropping the traditional fixed-function pipeline, JWZ had a very good summary[3] of the the problem, from a "having to maintain old software" prespective: Let's say you have a well-specified system that is in wide use (a language, a library API, whatever) and because of changes in some substrate (operating systems, hardware, whatever) you find that you need to add a new way of doing things to it. The way you do this is, you add new features to the specification and you clearly document the version in which those features become supported. If there are old features that you would like to discourage the use of, then you mark them as obsolete -- but you do not remove them because thou shalt not break working code. If you don't agree with that, then please, get out of the software industry right now. Find another line of work. Please. Your users have code that works. Maybe the new APIs would serve them better. Maybe things would be so much more efficient if they updated their code to use the new API. Or maybe it doesn't matter to them and they just want working code to continue to be working code. At least until such a time as they need the new features, or new efficiency. Remember the First Rule of Optimization: DON'T. sigh - one of the big reasons behind my I left the world of Microsoft and Windows behind ~15 year ago was to be free of this kind of forced change... 1: Not giving up my .xinitrc/.Xresources or E16 theme, either. In all cases (along with openrc/sysvinit), they have over a decade's worth of bugfixes and tweaks. Any replacement would have to be very good to be worth the time/effort involved in switching. 2: to be fair, Gnome has been merging thing that shouldn't be merged all the way back to the beginning of the project, and is as much at fault here as systemd itself. 3: http://www.jwz.org/blog/2012/06/i-have-ported-xscreensaver-to-the-iphone/ http://www.jwz.org/blog/2012/06/i-have-ported-xscreensaver-t...

gwu78 13y ago

I like this writing. Solid. But, I'm curious: What are the specific deficiencies the author sees in daemontools? "Not ready for primetime." I mean, it's so typical for bloggers to say certain software is this or that (some generalisation), without giving details. I don't know much about systemd. I find Linux too chaotic, a moving target that moves a little too frequently for my tastes. It's changes like these that I find take too much energy to follow. I doubt it's a problem specific to Linux though. As a BSD user, I've been amazed with how much init.c has grown in size over the years. A variety of developers have touched it.