9 ms·

Structural and semantic deficiencies in the systemd architecture

183 points by vezzy-fnord 11y ago

dang 11y ago
Since it looks like this article is trying to treat a controversial topic substantively, let's see if we can do the same. If you comment, please stick to the substance and avoid dismissive comments and rudeness. We should all do so anyway, but when the topic is polarized it can take more conscious effort. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
ncraun 11y ago
Very interesting article, refreshing as well to see a criticism of systemd based on technical details. Unlike most other criticisms of systemd, this author does not resort to personal attacks and insults against Lennart Poettering or fear-mongering appeals to tradition. I still think systemd is an improvement over the previous sysvinit, but the author seems to have found some real conceptual flaws with systemd that are somewhat unsettling. I hope this article can start some actually useful conversations about init systems that do not fall into the pitfalls of FUD and negativity that seem to have surrounded this topic in the past.
- striking 11y ago
  My explanation for the lack of technical (rather than political) articles examining systemd is that it's very difficult to wrap your mind around because it carries so many ideas. A quote from an article linked within the linked article: > All of the above definitions are technically correct. None of them are complete. > It is thus that debates are doomed to go nowhere, as no one can agree on a single definition of systemd, and the developers have been largely unhelpful in that matter. Vagueness can be a strength, in that it allows for people to divide and conquer, but it is also unproductive and leads to unnecessary bloodshed. Slowly, surely, people will begin to understand what systemd is. Only then will people be able to figure out whether they want it or not.
ausjke 11y ago
Read it through but hard to grasp all the details. One thing I'm sure is that for non-systemd "legacy" init systems I know what I'm doing, for systemd I most of the time don't know what's going on there, it's still like a blind art after using it for a while, overly abstracted might be one reason for that.
- kbenson 11y ago
  I think part of the reason for this may be that some types of systems lend themselves to exploration and gradual learning through use, while others, being very tightly coupled, generally require a larger upfront investment in learning how they function as a whole, since any one subcomponent is only marginally useful without the rest. This is largely how I remembered studying for Microsoft certification back in the day to be. There was only so much you could easily learn from playing around with the system. At some point, well written technical documentation was, if not required, extremely helpful in making connections between how systems worked together, or explaining exactly what a component was or was not capable of. I don't think this is inherently worse or better, just different, and with its own advantages and disadvantages. For example, people who went through this training seemed to have a very high base-level of understanding of how to configure these systems, what they could or could not do, and how to troubleshoot them, but not necessarily how they functioned in every case. Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. The average person responsible for using these will have vastly different levels of skill and knowledge about them, but with that comes how it actually functions, because that's needed to troubleshoot problems, since it's so exposed. The feeling I get mostly when reading about systemd is that it's very much in the vein of MS products. Like I said, that isn't necessarily a bad thing, but it is very different than a lot of UNIX admins are used to.
  - bgilroy26 11y ago
    >Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. Can you tease out this distinction a bit more? It reads like you are saying, due to their nature, people learned how to use Unix service handlers, without a good understanding of how they were designed. It could also mean that people learned how Unix service handlers worked by using them, through trial and error, because the documentation was scattered and not entirely comprehensive.
nextos 11y ago
The major problem right now if one decides to use a systemd alternative is that most packages are shipping with systemd unit files, but will obviously lack support for other init implementations. I like dmd, implemented in Scheme, which is part of the GuixSD distribution.
- baghira 11y ago
  1. The solution to this is to write a translator from systemd units (those of .service type) to the desired kind of unit. I haven't looked into dmd, but this should be possible at least for the majority of units. 2. This was going to be a problem in any case i.e. had most distros switched to openRC one would still have been faced with the task of producing a suitable collection of service configuration files for dmd and then adapting them to dmd's specfic featureset.
  - mercurial 11y ago
    Yeah, well, if it's to go from unit files to Guile code, that's not what I'd consider an improvement. Having configuration files for use cases where the requirements are simple enough is nice. Of course, when they're not, you end up with an abomination like MSBuild, but that's not the case here.
    - baghira 11y ago
      True, but if that is the case you won't be running dmd in the first place. And automated conversion/runtime translation will be a possibility even with different service managers, e.g. systemXVI. I'm not really convinced by the second, in that I believe that nothing in the article proves that service management is something too complex to be done via configuration files.
geofft 11y ago
This is a very detailed article, and I appreciate it and will need to study it carefully. But I think that the major argument it is making is "systemd is a mediocre service manager; similar tools existed before systemd, and were better." That is a very different argument from "we would be better off with sysvinit," which seems to be the primary argument I see in practice: if you're arguing things like "do one thing and do it well," "shell scripts are great and debuggable," "init should reap children and nothing else," "there's too much code in pid1," "I don't want to learn something new," etc., you're making that form of argument. If there is general consensus from studied opponents of systemd that something systemd-like is an obviously good idea, great. Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Which might even be based on an existing system like s6. But then the path forward has to be either improving systemd or replacing it, not continuing with sysvinit. I am in practice a pro-systemd person, but that's only because my realistic alternatives, for my use case, are sysvinit and Upstart. I have no idea how I'd start to build a system that used something like s6. I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades. If the general consensus is that a systemd-style service manager is great, but it should not be pid 1, that's an argument worth making, but I don't see much focus in this article on that. If the general consensus is that everything can be done with sysvinit, this article doesn't make that case at all. (But I suspect that's not the general consensus.)
- lobster_johnson 11y ago
  Talking about "general concensus" is hard because there are a lot of different viewpoints, and depends on how much of the debate one is exposed to. I have seen people compare systemd unfavourably to daemontools, s6, runit, Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative. A lot of people are unhappy with SysV, but if they don't like systemd, they would probably consider sticking with SysV to be the lesser of two evils. > Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Unfortunately, I think that for those who don't like systemd, their criticisms (its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of yarn contrary to Unix philosophy about small, composable programs cooperating over standard protocols) cannot be solved by improving systemd itself, at least not without a huge undertaking to redesign and break it apart, somehow.
hueving 11y ago
The article that lies herein, was, in a manner of speaking (but not literally speaking - 'in a manner of speaking' is just a phrase), so overly verbose in many places that it drowns the reader in verbal diarrhea as thick as pea soup. That is to say, I, as a reader, felt that the author (of this article) could have made these points in a more reachable, succinct fashion.
- vezzy-fnord 11y ago
  I would, as a general principle, be cautious (in the sense of not being hasty) to dismiss in a manner reminiscent of Parkinson's law of triviality (insofar as the latter pertains to the bikeshed principle) the literary style of the article in question, since the common interpretation of excessive verbosity is flawed, in this case being seldom more than a preference with little in the way of any concrete justification that would be of direct relevance to the topic at hand, the topic being an architectural treatise. Any reservations about succinctness, or a perceived lack thereof, must therefore be qualified and left over for further analysis to draw proper and conceptually robust remarks given the evidence presented. (Yes, the writing style is somewhat obtuse. You'll have to excuse that. The topic itself is obscure, so I suppose it fits.)
ised 11y ago
"a controversial topic" My observation: The "discussion" never seems to focus on systemd. Instead it usually turns to comments on the former sysvinit system or other init systems. That avoidance is perhaps something to ponder. And maybe it's why this author felt the urge to put some focus on systemd itself. Love the quote from djb. In sum, the best interface is no interface. Parsing amounts to high margin for error and often an incredible nuisance. In another thread today I wrote about the command line interface and "expecting a reponse". Truthfully, djb's utilities that simply return an exit value and no output are the best ones I have ever used. (It seems djb himself is a systemd user. Not sure what if anything that means.) Imagine if the information dissiminated via www was as easy to "parse" as text0. Writing a simple "web browser" might be easy enough that programmers would not need to be paid to do it.
- dang 11y ago
  > The "discussion" never seems to focus on systemd All the more reason to try making this one better. (We detached this comment from https://news.ycombinator.com/item?id=10370446 https://news.ycombinator.com/item?id=10370446 and marked it off-topic.)
tadfisher 11y ago
My response from another discussion on this article: Much of these deficiencies boil down to the design criteria for systemd, which is basically to avoid forcing the system administrator to perform dependency graph resolution. This was a direct response to Upstart, which pushed this work to the administrator (or distro packager) by design. Thus the declarative syntax for unit files, which do not explicitly control dependency resolution, parallelization, or startup order, which is simply not necessary or cared about for 99% of use cases. The 1% case there is for situations where you should not use a prebaked init system in the first place, such as when you absolutely need deterministic boots, i.e. in an embedded or realtime context. > In systemd, however, the execution state is not based on explicit chain loading, but on serializing unit file options to a private ExecContext structure overlayed into Unit objects and set for service unit types. This is not a problem with systemd's design. This is a problem with software written without the assumption of reliable process supervision, which is a reasonable assumption given the history of UNIX daemonization. systemd takes the (apparently controversial) stance that processes should not manage their own execution environment, or that of their dependencies. It provides numerous prebuilt hacks to make things work for broken software, but nobody should expect systemd to design for 100% of situations where processes do Weird Shit™. And yes, I'm including "delegated service restarters" under this umbrella. The need for something like this absolutely smells of poor design. It also flies in the face of the author's preference for deterministic execution behavior, because by definition your process will execute differently depending on if it was started or restarted. The link to "systemd house of horrors" [1] only proves this point. > A supervisor insisting on programmatic accommodations from service writers is not the most desirable state of affairs. A rarely discussed alternative to the two common approaches is, again as with startup speed optimization minus parallelism, checkpoint/restore: checkpoint a process image from a point where initialization is known to be complete and overlay it on startup, using a tool such as DMTCP or CRIU. I don't know about anyone else, but this sounds like an absolute nightmare. For one, it relies on the service exposing enough information to be checkpointed at the right time; at which point, you might as well call sd_notify or hook in a script to do it for you, so this is not a reduction in requiring programmatic accommodations from service writers. Two, process-snapshotting is not a panacea, and is a nightmare to debug when it breaks your expectations; you'll need to test all sorts of process state, including open pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which point it's best to just let the service notify you directly when the process is done. Which is a facility that systemd provides. > Of course we also have the issue of circular dependencies in the systemd architecture itself. We have the init, process manager, process supervisor, cgroup writer, local service tools, the Unit object structure (which might benefit from being made a protocol), timers, mounts, automounts and swaps all in the same module with ill-defined boundaries. A worthy criticism. I would also like to see a bit more modularization from systemd, but I can also see the point of systemd developers not expending effort to do so. For one, it would drastically increase the number of contexts for testing (i.e. can we verify the service manager works without the cgroups module loaded, etc). I think systemd is at the point where it's rock-solid for all of its use cases, and a pluggable/composable architecture would have delayed this situation for not very much benefit. In short, systemd imposes its opinions about process init and supervision. Whether or not this is reasonable is up for debate. However, it is very useful to have both a strong opinion and a solid implementation of this opinion. IMO, the idealized world that systemd supports (i.e. non-forking services, initialization notification, strict execution environment and dependency declaration, centralized and opaque init/dependency-resolution/parallelization) is a vast improvement over the status quo and actually improves the lives of service writers (by removing an entire class of responsibilities), so I think it's a reasonable opinion and thus I think systemd has a reasonable design and architecture. There are many known technical deficiencies with the implementation that are mostly being worked on, but I'm not seeing anything damning of the whole idea. [1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/systemd-house-of-horror/ http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...
colin_mccabe 11y ago
tl;dr: the author argues that: * systemd is more similar to a traditional job scheduler than a traditional init system. * systemd should not parse text files in pid1 because it might be a security risk. * Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified. * "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex. * Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?) * Systemd does not have a plugin system * Journald is criticized because it does too much I would counter that: * I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone. * The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism. * I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making. * I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense? * I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible. * Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.) * I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck") I enjoyed reading this, and it's nice to see some more reasoned criticism of systemd. I feel like the prose got a little bit purple at times. We had to spend 10 paragraphs "descend[ing] into an inferno with the same dead horse talking points", "culminating into some rather heterodox conclusions", and "progressively introducing and elucidating on concepts, applying some a priori reasoning at times, and backtracking to derive conclusions or reiterate on prior stated knowledge" before we even saw a word of argument! This needed an editor...
adekok 11y ago
My take-away is the following: * systemd tries to solve a complicated problem * systemd has a lot of complexity * some of that complexity seems unnecessary * parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad. * systemd manages states and dependencies (this is good) * some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad) * as a result of the above point, systemd behavior is hard to debug, and hard to understand * another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles. * The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad) ... etc. i.e. systemd is an engineering solution. It pretty much works, but it has inherent design / implementation issues which cause real-world problems.
- Jweb_Guru 11y ago
  This article pretty conclusively convinced me that systemd is actually good, if these are the strongest arguments against it. Most of the dependency-related stuff (process order related, cyclicstuff) seems eminently fixable. I found his arguments about dependency management having to be linearizable highly unconvincing.
  - sagichmal 11y ago
    > I found his arguments about dependency management having to be linearizable highly unconvincing. What? Can you expand on this? To me it rang as a well-reasoned, and pretty devastating, critique.
  - 9mit3t2m9h9a 11y ago
    These are the strongest argument against its core service management logic; those that explain why it is not better than Upstart (or daemontools or s6+s6-rc …) And systemd causes problems not only by this, but also by its bundling of similarly mediocre or worse other components.
chris_wot 11y ago
Thank you for doing this - I'm going to read this with a lot of interest :-)
- vezzy-fnord 11y ago
  Do return with commentary.

183 points by vezzy-fnord 11y ago

dang 11y ago

Since it looks like this article is trying to treat a controversial topic substantively, let's see if we can do the same. If you comment, please stick to the substance and avoid dismissive comments and rudeness. We should all do so anyway, but when the topic is polarized it can take more conscious effort. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

deleted 11y ago
[deleted]
deleted 11y ago
[deleted]

ncraun 11y ago

Very interesting article, refreshing as well to see a criticism of systemd based on technical details. Unlike most other criticisms of systemd, this author does not resort to personal attacks and insults against Lennart Poettering or fear-mongering appeals to tradition. I still think systemd is an improvement over the previous sysvinit, but the author seems to have found some real conceptual flaws with systemd that are somewhat unsettling. I hope this article can start some actually useful conversations about init systems that do not fall into the pitfalls of FUD and negativity that seem to have surrounded this topic in the past.

striking 11y ago
My explanation for the lack of technical (rather than political) articles examining systemd is that it's very difficult to wrap your mind around because it carries so many ideas. A quote from an article linked within the linked article: > All of the above definitions are technically correct. None of them are complete. > It is thus that debates are doomed to go nowhere, as no one can agree on a single definition of systemd, and the developers have been largely unhelpful in that matter. Vagueness can be a strength, in that it allows for people to divide and conquer, but it is also unproductive and leads to unnecessary bloodshed. Slowly, surely, people will begin to understand what systemd is. Only then will people be able to figure out whether they want it or not.

ausjke 11y ago

Read it through but hard to grasp all the details. One thing I'm sure is that for non-systemd "legacy" init systems I know what I'm doing, for systemd I most of the time don't know what's going on there, it's still like a blind art after using it for a while, overly abstracted might be one reason for that.

kbenson 11y ago
I think part of the reason for this may be that some types of systems lend themselves to exploration and gradual learning through use, while others, being very tightly coupled, generally require a larger upfront investment in learning how they function as a whole, since any one subcomponent is only marginally useful without the rest. This is largely how I remembered studying for Microsoft certification back in the day to be. There was only so much you could easily learn from playing around with the system. At some point, well written technical documentation was, if not required, extremely helpful in making connections between how systems worked together, or explaining exactly what a component was or was not capable of. I don't think this is inherently worse or better, just different, and with its own advantages and disadvantages. For example, people who went through this training seemed to have a very high base-level of understanding of how to configure these systems, what they could or could not do, and how to troubleshoot them, but not necessarily how they functioned in every case. Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. The average person responsible for using these will have vastly different levels of skill and knowledge about them, but with that comes how it actually functions, because that's needed to troubleshoot problems, since it's so exposed. The feeling I get mostly when reading about systemd is that it's very much in the vein of MS products. Like I said, that isn't necessarily a bad thing, but it is very different than a lot of UNIX admins are used to.
- bgilroy26 11y ago
  >Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. Can you tease out this distinction a bit more? It reads like you are saying, due to their nature, people learned how to use Unix service handlers, without a good understanding of how they were designed. It could also mean that people learned how Unix service handlers worked by using them, through trial and error, because the documentation was scattered and not entirely comprehensive.

nextos 11y ago

The major problem right now if one decides to use a systemd alternative is that most packages are shipping with systemd unit files, but will obviously lack support for other init implementations. I like dmd, implemented in Scheme, which is part of the GuixSD distribution.

baghira 11y ago
1. The solution to this is to write a translator from systemd units (those of .service type) to the desired kind of unit. I haven't looked into dmd, but this should be possible at least for the majority of units. 2. This was going to be a problem in any case i.e. had most distros switched to openRC one would still have been faced with the task of producing a suitable collection of service configuration files for dmd and then adapting them to dmd's specfic featureset.
- mercurial 11y ago
  Yeah, well, if it's to go from unit files to Guile code, that's not what I'd consider an improvement. Having configuration files for use cases where the requirements are simple enough is nice. Of course, when they're not, you end up with an abomination like MSBuild, but that's not the case here.
  - baghira 11y ago
    True, but if that is the case you won't be running dmd in the first place. And automated conversion/runtime translation will be a possibility even with different service managers, e.g. systemXVI. I'm not really convinced by the second, in that I believe that nothing in the article proves that service management is something too complex to be done via configuration files.

geofft 11y ago

This is a very detailed article, and I appreciate it and will need to study it carefully. But I think that the major argument it is making is "systemd is a mediocre service manager; similar tools existed before systemd, and were better." That is a very different argument from "we would be better off with sysvinit," which seems to be the primary argument I see in practice: if you're arguing things like "do one thing and do it well," "shell scripts are great and debuggable," "init should reap children and nothing else," "there's too much code in pid1," "I don't want to learn something new," etc., you're making that form of argument. If there is general consensus from studied opponents of systemd that something systemd-like is an obviously good idea, great. Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Which might even be based on an existing system like s6. But then the path forward has to be either improving systemd or replacing it, not continuing with sysvinit. I am in practice a pro-systemd person, but that's only because my realistic alternatives, for my use case, are sysvinit and Upstart. I have no idea how I'd start to build a system that used something like s6. I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades. If the general consensus is that a systemd-style service manager is great, but it should not be pid 1, that's an argument worth making, but I don't see much focus in this article on that. If the general consensus is that everything can be done with sysvinit, this article doesn't make that case at all. (But I suspect that's not the general consensus.)

lobster_johnson 11y ago
Talking about "general concensus" is hard because there are a lot of different viewpoints, and depends on how much of the debate one is exposed to. I have seen people compare systemd unfavourably to daemontools, s6, runit, Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative. A lot of people are unhappy with SysV, but if they don't like systemd, they would probably consider sticking with SysV to be the lesser of two evils. > Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Unfortunately, I think that for those who don't like systemd, their criticisms (its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of yarn contrary to Unix philosophy about small, composable programs cooperating over standard protocols) cannot be solved by improving systemd itself, at least not without a huge undertaking to redesign and break it apart, somehow.

hueving 11y ago

The article that lies herein, was, in a manner of speaking (but not literally speaking - 'in a manner of speaking' is just a phrase), so overly verbose in many places that it drowns the reader in verbal diarrhea as thick as pea soup. That is to say, I, as a reader, felt that the author (of this article) could have made these points in a more reachable, succinct fashion.

vezzy-fnord 11y ago
I would, as a general principle, be cautious (in the sense of not being hasty) to dismiss in a manner reminiscent of Parkinson's law of triviality (insofar as the latter pertains to the bikeshed principle) the literary style of the article in question, since the common interpretation of excessive verbosity is flawed, in this case being seldom more than a preference with little in the way of any concrete justification that would be of direct relevance to the topic at hand, the topic being an architectural treatise. Any reservations about succinctness, or a perceived lack thereof, must therefore be qualified and left over for further analysis to draw proper and conceptually robust remarks given the evidence presented. (Yes, the writing style is somewhat obtuse. You'll have to excuse that. The topic itself is obscure, so I suppose it fits.)

ised 11y ago

"a controversial topic" My observation: The "discussion" never seems to focus on systemd. Instead it usually turns to comments on the former sysvinit system or other init systems. That avoidance is perhaps something to ponder. And maybe it's why this author felt the urge to put some focus on systemd itself. Love the quote from djb. In sum, the best interface is no interface. Parsing amounts to high margin for error and often an incredible nuisance. In another thread today I wrote about the command line interface and "expecting a reponse". Truthfully, djb's utilities that simply return an exit value and no output are the best ones I have ever used. (It seems djb himself is a systemd user. Not sure what if anything that means.) Imagine if the information dissiminated via www was as easy to "parse" as text0. Writing a simple "web browser" might be easy enough that programmers would not need to be paid to do it.

dang 11y ago
> The "discussion" never seems to focus on systemd All the more reason to try making this one better. (We detached this comment from https://news.ycombinator.com/item?id=10370446 https://news.ycombinator.com/item?id=10370446 and marked it off-topic.)

tadfisher 11y ago

My response from another discussion on this article: Much of these deficiencies boil down to the design criteria for systemd, which is basically to avoid forcing the system administrator to perform dependency graph resolution. This was a direct response to Upstart, which pushed this work to the administrator (or distro packager) by design. Thus the declarative syntax for unit files, which do not explicitly control dependency resolution, parallelization, or startup order, which is simply not necessary or cared about for 99% of use cases. The 1% case there is for situations where you should not use a prebaked init system in the first place, such as when you absolutely need deterministic boots, i.e. in an embedded or realtime context. > In systemd, however, the execution state is not based on explicit chain loading, but on serializing unit file options to a private ExecContext structure overlayed into Unit objects and set for service unit types. This is not a problem with systemd's design. This is a problem with software written without the assumption of reliable process supervision, which is a reasonable assumption given the history of UNIX daemonization. systemd takes the (apparently controversial) stance that processes should not manage their own execution environment, or that of their dependencies. It provides numerous prebuilt hacks to make things work for broken software, but nobody should expect systemd to design for 100% of situations where processes do Weird Shit™. And yes, I'm including "delegated service restarters" under this umbrella. The need for something like this absolutely smells of poor design. It also flies in the face of the author's preference for deterministic execution behavior, because by definition your process will execute differently depending on if it was started or restarted. The link to "systemd house of horrors" [1] only proves this point. > A supervisor insisting on programmatic accommodations from service writers is not the most desirable state of affairs. A rarely discussed alternative to the two common approaches is, again as with startup speed optimization minus parallelism, checkpoint/restore: checkpoint a process image from a point where initialization is known to be complete and overlay it on startup, using a tool such as DMTCP or CRIU. I don't know about anyone else, but this sounds like an absolute nightmare. For one, it relies on the service exposing enough information to be checkpointed at the right time; at which point, you might as well call sd_notify or hook in a script to do it for you, so this is not a reduction in requiring programmatic accommodations from service writers. Two, process-snapshotting is not a panacea, and is a nightmare to debug when it breaks your expectations; you'll need to test all sorts of process state, including open pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which point it's best to just let the service notify you directly when the process is done. Which is a facility that systemd provides. > Of course we also have the issue of circular dependencies in the systemd architecture itself. We have the init, process manager, process supervisor, cgroup writer, local service tools, the Unit object structure (which might benefit from being made a protocol), timers, mounts, automounts and swaps all in the same module with ill-defined boundaries. A worthy criticism. I would also like to see a bit more modularization from systemd, but I can also see the point of systemd developers not expending effort to do so. For one, it would drastically increase the number of contexts for testing (i.e. can we verify the service manager works without the cgroups module loaded, etc). I think systemd is at the point where it's rock-solid for all of its use cases, and a pluggable/composable architecture would have delayed this situation for not very much benefit. In short, systemd imposes its opinions about process init and supervision. Whether or not this is reasonable is up for debate. However, it is very useful to have both a strong opinion and a solid implementation of this opinion. IMO, the idealized world that systemd supports (i.e. non-forking services, initialization notification, strict execution environment and dependency declaration, centralized and opaque init/dependency-resolution/parallelization) is a vast improvement over the status quo and actually improves the lives of service writers (by removing an entire class of responsibilities), so I think it's a reasonable opinion and thus I think systemd has a reasonable design and architecture. There are many known technical deficiencies with the implementation that are mostly being worked on, but I'm not seeing anything damning of the whole idea. [1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/systemd-house-of-horror/ http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...

colin_mccabe 11y ago

tl;dr: the author argues that: * systemd is more similar to a traditional job scheduler than a traditional init system. * systemd should not parse text files in pid1 because it might be a security risk. * Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified. * "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex. * Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?) * Systemd does not have a plugin system * Journald is criticized because it does too much I would counter that: * I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone. * The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism. * I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making. * I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense? * I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible. * Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.) * I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck") I enjoyed reading this, and it's nice to see some more reasoned criticism of systemd. I feel like the prose got a little bit purple at times. We had to spend 10 paragraphs "descend[ing] into an inferno with the same dead horse talking points", "culminating into some rather heterodox conclusions", and "progressively introducing and elucidating on concepts, applying some a priori reasoning at times, and backtracking to derive conclusions or reiterate on prior stated knowledge" before we even saw a word of argument! This needed an editor...

adekok 11y ago

My take-away is the following: * systemd tries to solve a complicated problem * systemd has a lot of complexity * some of that complexity seems unnecessary * parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad. * systemd manages states and dependencies (this is good) * some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad) * as a result of the above point, systemd behavior is hard to debug, and hard to understand * another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles. * The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad) ... etc. i.e. systemd is an engineering solution. It pretty much works, but it has inherent design / implementation issues which cause real-world problems.

Jweb_Guru 11y ago
This article pretty conclusively convinced me that systemd is actually good, if these are the strongest arguments against it. Most of the dependency-related stuff (process order related, cyclicstuff) seems eminently fixable. I found his arguments about dependency management having to be linearizable highly unconvincing.
- sagichmal 11y ago
  > I found his arguments about dependency management having to be linearizable highly unconvincing. What? Can you expand on this? To me it rang as a well-reasoned, and pretty devastating, critique.
- 9mit3t2m9h9a 11y ago
  These are the strongest argument against its core service management logic; those that explain why it is not better than Upstart (or daemontools or s6+s6-rc …) And systemd causes problems not only by this, but also by its bundling of similarly mediocre or worse other components.

chris_wot 11y ago

Thank you for doing this - I'm going to read this with a lot of interest :-)

vezzy-fnord 11y ago
Do return with commentary.