29 ms·

Systemd, ten years later: a historical and technical retrospective

381 points by vezzy-fnord 6y ago

lidHanteyk 6y ago
The author sounds distressingly prescient with their prediction that BPF will be the next venue for this sort of farce.
nn3 6y ago
The dependency problems he's describing sound like they could be relatively easily fixed with some new systemd keywords? Of course it would take some time to migrate existing unit files. It doesn't sound like a fundamental critique. Would be nice to turn this into a constructive proposal to fix systemd.
- kelnos 6y ago
  It feels like a lot of the existing keywords were added in order to paper over previously-discovered dependency issues. And the result has been a) confusion, and b) more dependency issues. I don't think adding more is going to help the matter.
meddlepal 6y ago
Oh good, the monthly HN systemd 5 minute hate is upon us once again.
- gjvc 6y ago
  It's worth considering why the topic makes such a regular appearance. I suggest it might be because that so many users of this website find it to be a remarkable example of not only how something so widely unpopular can become so widely established, but also being something that they have to contend with frequently.
  - sneak 6y ago
    It would appear from distribution adoption that systemd is a lot more popular than it is unpopular.
    - dralley 6y ago
      One of the reasons distributions appreciate systemd because systemd unit files are easy to write and can easily be written and maintained upstream and used with very few if any modifications downstream. Your average HN user doesn't see or particularly care that it makes things easier for distro maintainers, so it's much easier to push the corporate conspiracy angle than to accept that a lot of the hand-wringing takes huge amounts not-very-fun volunteer labor for granted.
      - rleigh 6y ago
        This is one thing which is often claimed, but to my mind is poorly justified. As a (Debian) package maintainer for over a decade, this used to be a non-issue. They were write and forget. The average package was just some simple boilerplate.
        esarbe 6y ago
        I love not having to maintain half a dozen init scripts for three distributions but just one unit file for a well defined service manager.
      - bandrami 6y ago
        As a Slackware and OpenBSD user I've really never understood this complaint. The init scripts they ship with are neither difficult to understand nor hard to read, and new ones are very easy to write. Maybe the LSB stuff was awful; I never used it, and for that matter I can't think of a reason I would want to. It's the old FreeDesktop.org issue: it's a probably-interesting solution to a problem I just don't have.
    - bzb3 6y ago
      Red Hat made systemd a hard dependency for gnome, forcing all distros that have no manpower to patch gnome to use systemd.
  - Nextgrid 6y ago
    Is it actually unpopular or is there just a vocal minority constantly complaining about it while a much larger part of the user base is happily using it?
    - riskyfive 6y ago
      It's unpopular. It has funding. No other project in the space has funding, so it wins by default.
      - asdf21 6y ago
        Sounds like a backdoor to me.
      - gjvc 6y ago
        Exactly. This is the measure of the situation. Well put.
    - meddlepal 6y ago
      Mostly just unpopular here from what I've seen in industry as a programmer with lots of sysadmin, ops, and infra engineering experience. The usual arguments are it is not "Unixy" and then fall back on original designer intent and philosophical attacks, or worse ad hominems against Red Hat or Linus Pottering.
gjvc 6y ago
If you don't like something, put your energy into implementing it the way you do. With any luck, the ensuing distraction will be more effective at weakening the original than simply complaining about it.
- Karunamon 6y ago
  Sadly, systemd has corporate backing, which is a very hard fight to win for a ragtag bunch of nerds with dayjobs who have the completely unreasonable expectation of not having their shit broken, regardless of how much energy we have. Money and full-time devs beats energy every day of the week.
  - gjvc 6y ago
    My point was that there is nothing (no competing implementation) to be judged against, so it rather wins de facto.
    - dijit 6y ago
      This is like declaring McDonalds burgers the best because small burger joints can't offer Burgers _and also_ ice creams, salads, chicken and toys for children. If we're talking about init; there are actually better inits already available but you'll never hear about them (runit, for instance) precisely because you don't just leave systemd. You leave the systemd ecosystem. This is the main point of the article: systemd as a "job system" is actually quite bad, but the ecosystem has some value in some places and you have to buy the shit with the pig.
whereistimbo 6y ago
vezzy-fnord! I have always enjoyed your commentary and perspective of operating system, it's nice to see you back again!
- vezzy-fnord 6y ago
  > it's nice to see you back again! Not for long, probably. I drifted out of this sphere years ago, and came back specifically for systemd's 10 year anniversary, as I felt obligated to at least do that. Judging by the tone of the comments, 10 years later is still too soon to discuss systemd dispassionately, but when I come back in 10 more years I'll see if things have changed.
  - whereistimbo 6y ago
    Would you like to tell me what OS are you currently using? Are you still keeping pace on OS research?
  - majewsky 6y ago
    Honest question: Which relevant technology is ever discussed dispassionately?
    - vezzy-fnord 6y ago
      If we can't ever perfectly adhere to an ideal, then I guess we ought to go full on with the connivery and deceit, then. Regardless, I did have at least some expectation that people would actually focus on the original post and not brawl in the comment section over tangents that often aren't even part of it. Oh, well. And hey, if other people down the road read this and it makes something click for them, I'll be happy with that.
      - btrask 6y ago
        Thank you for the article! It's too bad that your conclusion, that systemd doesn't/shouldn't matter, wasn't able to be grasped by the community.
deleted 6y ago
[deleted]
bepvte 6y ago
Interesting comparisons of a GNOME volunteer to Stalin. Also found it very interesting that the author put inclusivity in quotes.
- vezzy-fnord 6y ago
  Oh, but it is a good comparison. Many people have compared free software to communism. I always used to dismiss it. But, in a way, they were right. Both are movements with a self-conceited historicist endeavor to eliminate exploiting classes ('it is inevitable that the tendency for the rate of profit to fall to produce immiseration as capitalists can no longer profitably invest, leading to a proletarian seizure of power' versus 'it is inevitable that proprietary walled garden development models stagnate as they're overtaken by decentralized communities pooling their knowledge and labor into the most optimal end' and create a perfectly equitable free association of producers no longer having their surplus value be seized by exploiters. One degenerates into a repressive bureaucratic collectivist (I'm not a Trotskyist, but it's a good term) mechanism for reproducing surplus through either direct requisition or coercive planning. The other degenerates into a way for large-scale cloud providers to appropriate the free labor of volunteers so as to reinvest it back in their walled gardens. Stalin represented the Leninist wing against ultra-leftists, anarchists and other factions who were against tight top-down organization in favor of 'spontaneous' organizing, and the GNOME/Freedesktop/Red Hat nexus is against the faction in free software that insists on a loosely coordinated and loosely coupled bazaar without a central vision. But in no sense has systemd abolished this, it has only reduced a few nodes at best. EDIT: Actually I should point out that 'free software' per se is a deontological argument, and it's 'open source' specifically which specifically uses these historicist arguments to justify itself. But then open source was unabashedly an effort for corporate sanitization of free software, which most people forgot after it mostly crowded out 'free software.'
ratsmack 6y ago
>systemd still remains poorly understood and understudied from both a technical and social level despite paradoxically having disproportionate levels of attention focused on it. This statement makes no sense. It is well understood and has been studied and critiqued by many, including me. Also, I don't know what "social level" has to do with anything here. It was just created for the commercial aspect of Linux and forced into the relevant distro's by commercial interests... it's just that simple.
- JChase2 6y ago
  I'd say software doesn't always win out on it's merits, there's almost definitely some social aspects regarding what does and does not get adopted. I do agree the commercial interests probably played the largest role, though. I think beyond the initial knee-jerk one might have towards this it's pretty comprehensive and there's interesting points here that are at least worth exploring.
  - acdha 6y ago
    Aren’t those social aspects part of their merits? A big part of why systemd won widespread adoption was by stepping up – the number of people working on the others wasn’t enough to be more competitive. I lost track of the number of Upstart bugs we avoided by switching to systemd, and that had the backing of one of the most popular Linux distributions.
    - JChase2 6y ago
      I suppose so ya, can't sustain without a community or company unless you want to do all the work yourself.
      - acdha 6y ago
        Another factor I was thinking is that almost everything in systemd is there because someone saw a need, even if it’s a niche – things like mounts aren’t a big deal for many people but the people who had something which wasn’t well served before wanted something better in a new system. Trying to replace a core component will flush out a lot of those smaller communities.
acdha 6y ago
I think this really sums this article up: > One thing I’m certain of is that this shift cannot emerge from dilettantes, outsiders and proverbial basement hackers. One does not unseat a platform without already being part of the patriciate that calls the shots on what gets integrated where across the largest nodes in the ecosystem. It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work. An init system is harder than it might seem at first and requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators. If you’re not going to invest that level of engineering time, I don’t see how it’s reasonable to expect an equal voting share with those who do.
- throw0101a 6y ago
  > An init system is harder than it might seem at first […] systemd-as-init-system is not the problem. systemd-as-kitchen-sink is the problem. It's the tight coupling that annoys many people. Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?) What does systemd 245's systemd-homed have to do with an init replacement? * https://www.techrepublic.com/article/linux-home-directory-management-is-about-to-undergo-major-change/ https://www.techrepublic.com/article/linux-home-directory-ma...
  - ArchD 6y ago
    Yeah, don't get me started on the fragile glass test tube called systemd-resolved that fails in mysterious ways when I use wicd on Ubuntu and the WiFi gets momentarily disconnected. I don't see the sense behind treating name resolution as something so special that it has to be part of an init system. I like the overall theoretical concept of systemd but it has ugly implementation details like this.
    - Spivak 6y ago
zozbot234 6y ago
Systemd is an amazingly clear example of the second-system effect, as described in The Mythical Man Month. I fully expect it to be replaced down the line by something dramatically simpler and more intuitive, but that might take some time. Nonetheless, it does seem to solve some real problems with the earlier, rc-scripts approach.
- 0zymandiass 6y ago
  It's dramatically simpler and more intuitive than what it replaced, so I'm super excited when someone comes up with something even better!
  - downerending 6y ago
    Systemd has many pluses, but simplicity and intuitiveness are most certainly not among them.
    - josteink 6y ago
      Simple... for what? I’d like to setup a service, which depends on another service, and which must always be running, and if it goes down, it needs to have all forked processes killed, must be restarted, and it needs to run as a specific user. With systemd? 5 lines or so of boilerplate, independent of distro. I don’t know about you, but I call that simple.
      - kelnos 6y ago
        I think this subthread is talking about the complexity of a system's implementation, not that of its user interface.
        josteink 6y ago
        As a user, systemd wins for me hands down, because it is simple to use. That it masks complexity away from me is a feature, not a bug. This is also why distros is implementing it almost everywhere: It makes their job easier. Now if you are measuring the complexity of the system's (full) implementation, you can't really compare systemd to to SysV-init. You need to compare a systemd-based system to a SysV-init based system. And then you need to account for the 100s of inconsistently written shell-scripts, all the code distributed in the mess of third party dependencies (sh, bash, Perl, Awk, Python, su, supervisord, cron, etc) to ensure that the functionality of the systems you are comparing is really somewhat equal, not to mention the added complexity of the glue between all those components, and how it may fail. And when you do that comparison, I do believe you will end up concluding that 1. full systems are somewhat complex, for both systems, and 2. that systemd-based systems are not fundamentally more complex, and 3. systemd-based systems have the benefit of a single, centralized implementation for all these core functions, so that developers don't have to reimplement them inconsistently, and possibly buggy. With systemd, for better or worse, either your entire init-chain is broken or it isn't, so when it works, you know it works. With SysV-init you have no such guarantees.
ttctciyf 6y ago
As a somewhat dilettante and casual home sysadmin (currently) I was messing around with screen on a box downstairs and ran into this: https://www.reddit.com/r/programming/comments/4ldewx/systemd_kills_screen_and_tmux_by_default_on_logout/ https://www.reddit.com/r/programming/comments/4ldewx/systemd... Namely that systemd doesn't allow persistent processes started from the shell by default, preferring to terminate them when the user logs out. This would include processes like "screen" whose entire raison d'etre is to persist after the user logs out. (Well, it has other uses, but this is the main one IMO.) The stated workarounds - fiddling with some options like "KillUserProcesses=no" in logind.conf &co. - have so far failed. I don't know whether this situation is a problem with systemd or the distro, but it seems very much a problem with the culture summarised by the top commenter in the above thread, of (paraphrasing) glibly breaking existing workflows then casually brushing away criticism with arguments often boiling down to: "this is the right way, I don't care about tradition or protecting 'incorrect' usage."
- zozbot234 6y ago
  This is yet another example of pointless incidental complexity in systemd. The whole point of "nohup"-based tools like tmux and screen is to cleanly separate the management of user sessions from the incidental mechanism of whether a remote connection is being closed (the 'HUP' in nohup is short for "hang up" i.e. close a [possibly remote] connection). Systemd should simply acknowledge this fact and keep the user session going when a program has been launched under nohup, instead it tightly couples its own "session" concept to the remote connection and then adds a totally ad-hoc, hacked-together feature called EnableLinger to somehow make nohup work anyway. It's amazing.
  - pknopf 6y ago
    I'm not seeing a lot of hope here. There is an incredible investment in systemd by distributions, there is no turning back. Help me..
    - BeetleB 6y ago
      Gentoo is reaching out to you.
nine_k 6y ago
My beef with systemd is not its reinventing things. That part may actually be good. One problem is that a number of reinventions were poorly made. E.g. the log format. Yes, unstructured logs have a ton of drawbacks. Can we take some ridiculously well-tested, reliable embedded database or serialization format (like sqlite) and use it as log storage? Alas. Another problem is the "I know better" attitude. Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior. No wonder I don't run systemd on my desktops / laptops.
- m45t3r 6y ago
  > Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior. All those examples you gave are possible to disable in systemd, and in the case of DNS and user process after logout, they are not even the default. : Killing of user process are enabled by default in upstream but disabled by default in every distro that I know. And systemd-resolved, even if it is build by default, is not enabled by default even in upstream.
  - nine_k 6y ago
    Good thing sanity prevails in distro maintainers! But not in upstream, as you duly note. I also remember that even 2-3 years ago the situation was not as nice yet. I think what became systemd could have been great software, if not for the, mmm, cavalier attitude of its creators (not necessarily personal; it might be amended by RH corporate deadlines).
    - hnarn 6y ago
      "Not in upstream" doesn't seem to correspond with the comment you replied to.
wired_devil 6y ago
systemd is trash
aganame 6y ago
I wish systemd haters would make a public petition to remove systemd from this world. I would absolutely use such a list to make sure I never accidentally hire any one of them for any system administration positions.
- dman 6y ago
  As a person who does not work as a system administrator professionally but does use self administered linux machines as workstations systemd has not brought any new productivity gains but it sure has increased the learning curve.
  - aganame 6y ago
    As a person for whom Linux and BSDs have been a hobby since 1996, I have no idea what you're talking about.
    - jacquesm 6y ago
      A hobbyist making hiring decisions and maintaining a blacklist of professionals who bitch about a change they had no control over? You're not helping.
      - deleted 6y ago
        [deleted]
    - tux1968 6y ago
      One small example that i've struggled with is logging. It used to be I could reuse my editor knowledge to search and read log files. Now every time I want to look at a log I have to re-read the man pages to figure out the proper incantation.
      - aganame 6y ago
        Learning new tricks gets harder when we get older. Is that the software’s fault?
gorgoiler 6y ago
When writing software, it’s comparatively easy to come up with grand new ideas and turn them into lines of code and files filled with modules. What’s much harder is (1) to expand ones code without coupling everything together (2) present it in a persuasive way that naturally builds a user base. SystemD feels like the new grad hire who decided the first thing the would do after joining — the opening power play — is to convince the PHB we should rewrite everything in Haskell. At this point the gentler folk start quietly honing their CVs and sneaking their personal belongings off their desks, one night at a time, in the hope no one notices they are all about to quit. It felt like Unix systems administration was a haven from this sort of politics — I don’t remember tmux, git, or python trying to push behavior changes on upstream components of the OS[1]. I’m sure it’s why Unix attracted a certain type of personality for so many years and I’m sad to see that changing. [1] If you are thinking “but those are command line utilities, whereas the init process is a much more specialized case” then I’d encourage you to revisit the Unix principles of every component being a small and simple program! Even init, cron, and login! It’s not some grand ideal to be zealously adhered to: it’s the principles of small components that mean the same OS can run on a quad core Xeon as runs on a $45 network switch.
- theamk 6y ago
  I think you underestimate how badly the old sysvinit sucked in the world with many good distribution-provided packages. It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages: Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script. This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate. Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.
noitpmeder 6y ago
My worst gripe with systemd is that I cannot find reliable docs for older versions.
- nineteen999 6y ago
  Right. All we need now is systemd-selfupdated, so that it constantly keeps itself up to date, even through egress filtering firewalls.
- fsh 6y ago
  Your manpages should match the installed version.
reacharavindh 6y ago
God I wish more people used VoidLinux and packaged more and more stuff without needing Systemd. Systemd may be good for some, bad for some. But, it certainly should not be the only way to get things done. We need alternatives. It’s as if web developers were told now that there is IE, that is prevalent, we can write web apps that only work there ;-)
nyanpasu64 6y ago
I don't fully understand systemd, but the technical critique looks interesting and seems to indicate systemd is poorly designed (but I don't understand it all). Anyone else commenting on that?
- JdeBP 6y ago
  Enjoy https://blog.darknedgy.net/technology/2015/10/11/0/ https://blog.darknedgy.net/technology/2015/10/11/0/ , by the same author.
k_bx 6y ago
Can I just say, from a non-sysadmin perspective, how happy am I that these days I don't have to install something like Supervisord and just use the systemd + journald toolchain. Setup script is just one cp and few systemctl commands (start/stop/enable).
- dijit 6y ago
  Yeah, that's nice. I think there's a lot of people who get caught up in the merits of systemd and assume that there's nothing better. Or that the alternative is the old sysvinit which was in dire need of being replaced- but that's not true. Process supervision is totally feasible in init (just look at runit, Solaris' SMF and MacOS's launchd) even Windows has one.
mr_isomies 6y ago
Getting 10 engineers to agree on how to implement a padding function is nearly impossible. Was there ever hope to get thousands to agree at something that front and center? There was literally decades to come up with a solution. I'm sure things where proposed during that time. What systemd did right was get (or coerce) buy-in from enough entities & people to get a big enough purchase to grow. For me, that's the takeaway. It's not on the engineering merit of their solution.
dvfjsdhgfv 6y ago
A key quote: > As it turns out, there was a little-known Freedesktop-affiliated project called xdg-hostname in 2009 which was a start towards creating such D-Bus “utility daemons” for use in desktop widgets and others, all as a standalone project. Had this been followed through instead of having the utility daemons end up being part of the systemd source tree, a good deal of political acrimony could have been avoided – though at the cost of reducing systemd’s leverage. That is, if they used xdg-hostname instead, SystemD wouldn't be tightly coupled with Gnome, so users could choose between SystemD and InitV, and all this mess could have been avoided.
Stierlitz 6y ago
TLDR: down with sYstemd :]
arh68 6y ago
PulseAudio is the only program I've seen really stress my Ryzen. I recommend fixing the default sampling config if you have similar issues. This is a really entertaining writeup, given the subject matter.
Koshkin 6y ago
My biggest gripe with it is that its name is not ‘sysd.’
keymone 6y ago
One thing I’m surprised about in the init wars of past decade(s?) is that in almost half a century of Unix nobody got up and said “why don’t we, instead of bickering about implementation details, come up with single format to declare desired system startup behavior, and then users can pick whichever implementation they like to read and execute that format”? Can’t be impossible to distill some basics about service startup configuration and dependencies, can it? Then make it extensible and cry about differences in extension support like we do about per-browser css options, but at least converge on some basics.
AndrewStephens 6y ago
I do really appreciate this type of long form, researched blog post. I have no strong opinions on systemd, I've made simple service units and it has worked well enough. Having the history behind the design is a great aid to understanding.
- JdeBP 6y ago
  Then you will probably enjoy these: * https://blog.darknedgy.net/technology/2015/10/11/0/ https://blog.darknedgy.net/technology/2015/10/11/0/ * https://blog.darknedgy.net/technology/2015/09/05/0/ https://blog.darknedgy.net/technology/2015/09/05/0/ * https://blog.darknedgy.net/technology/2016/01/01/0/ https://blog.darknedgy.net/technology/2016/01/01/0/ * https://blog.afoolishmanifesto.com/tags/supervisors/ https://blog.afoolishmanifesto.com/tags/supervisors/
bandrami 6y ago
10 years ago I never had shutdowns hang with "Stop job waiting for PID 10834: 30 seconds... 90 seconds..." I do now. Regressions are a bad thing.
bozobuttz 6y ago
Systemd is an annoying turd. Still hate dealing with it to this day. I have to refer to the man pages every. single. time.

381 points by vezzy-fnord 6y ago

The author sounds distressingly prescient with their prediction that BPF will be the next venue for this sort of farce.

nn3 6y ago

The dependency problems he's describing sound like they could be relatively easily fixed with some new systemd keywords? Of course it would take some time to migrate existing unit files. It doesn't sound like a fundamental critique. Would be nice to turn this into a constructive proposal to fix systemd.

kelnos 6y ago
It feels like a lot of the existing keywords were added in order to paper over previously-discovered dependency issues. And the result has been a) confusion, and b) more dependency issues. I don't think adding more is going to help the matter.

meddlepal 6y ago

Oh good, the monthly HN systemd 5 minute hate is upon us once again.

gjvc 6y ago
It's worth considering why the topic makes such a regular appearance. I suggest it might be because that so many users of this website find it to be a remarkable example of not only how something so widely unpopular can become so widely established, but also being something that they have to contend with frequently.
- sneak 6y ago
  It would appear from distribution adoption that systemd is a lot more popular than it is unpopular.
  - dralley 6y ago
    One of the reasons distributions appreciate systemd because systemd unit files are easy to write and can easily be written and maintained upstream and used with very few if any modifications downstream. Your average HN user doesn't see or particularly care that it makes things easier for distro maintainers, so it's much easier to push the corporate conspiracy angle than to accept that a lot of the hand-wringing takes huge amounts not-very-fun volunteer labor for granted.
    - rleigh 6y ago
      This is one thing which is often claimed, but to my mind is poorly justified. As a (Debian) package maintainer for over a decade, this used to be a non-issue. They were write and forget. The average package was just some simple boilerplate.
      - esarbe 6y ago
        I love not having to maintain half a dozen init scripts for three distributions but just one unit file for a well defined service manager.
    - bandrami 6y ago
      As a Slackware and OpenBSD user I've really never understood this complaint. The init scripts they ship with are neither difficult to understand nor hard to read, and new ones are very easy to write. Maybe the LSB stuff was awful; I never used it, and for that matter I can't think of a reason I would want to. It's the old FreeDesktop.org issue: it's a probably-interesting solution to a problem I just don't have.
  - bzb3 6y ago
    Red Hat made systemd a hard dependency for gnome, forcing all distros that have no manpower to patch gnome to use systemd.
- Nextgrid 6y ago
  Is it actually unpopular or is there just a vocal minority constantly complaining about it while a much larger part of the user base is happily using it?
  - riskyfive 6y ago
    It's unpopular. It has funding. No other project in the space has funding, so it wins by default.
    - asdf21 6y ago
      Sounds like a backdoor to me.
    - gjvc 6y ago
      Exactly. This is the measure of the situation. Well put.
  - meddlepal 6y ago
    Mostly just unpopular here from what I've seen in industry as a programmer with lots of sysadmin, ops, and infra engineering experience. The usual arguments are it is not "Unixy" and then fall back on original designer intent and philosophical attacks, or worse ad hominems against Red Hat or Linus Pottering.

gjvc 6y ago

If you don't like something, put your energy into implementing it the way you do. With any luck, the ensuing distraction will be more effective at weakening the original than simply complaining about it.

Karunamon 6y ago
Sadly, systemd has corporate backing, which is a very hard fight to win for a ragtag bunch of nerds with dayjobs who have the completely unreasonable expectation of not having their shit broken, regardless of how much energy we have. Money and full-time devs beats energy every day of the week.
- gjvc 6y ago
  My point was that there is nothing (no competing implementation) to be judged against, so it rather wins de facto.
  - dijit 6y ago
    This is like declaring McDonalds burgers the best because small burger joints can't offer Burgers _and also_ ice creams, salads, chicken and toys for children. If we're talking about init; there are actually better inits already available but you'll never hear about them (runit, for instance) precisely because you don't just leave systemd. You leave the systemd ecosystem. This is the main point of the article: systemd as a "job system" is actually quite bad, but the ecosystem has some value in some places and you have to buy the shit with the pig.

whereistimbo 6y ago

vezzy-fnord! I have always enjoyed your commentary and perspective of operating system, it's nice to see you back again!

vezzy-fnord 6y ago
> it's nice to see you back again! Not for long, probably. I drifted out of this sphere years ago, and came back specifically for systemd's 10 year anniversary, as I felt obligated to at least do that. Judging by the tone of the comments, 10 years later is still too soon to discuss systemd dispassionately, but when I come back in 10 more years I'll see if things have changed.
- whereistimbo 6y ago
  Would you like to tell me what OS are you currently using? Are you still keeping pace on OS research?
- majewsky 6y ago
  Honest question: Which relevant technology is ever discussed dispassionately?
  - vezzy-fnord 6y ago
    If we can't ever perfectly adhere to an ideal, then I guess we ought to go full on with the connivery and deceit, then. Regardless, I did have at least some expectation that people would actually focus on the original post and not brawl in the comment section over tangents that often aren't even part of it. Oh, well. And hey, if other people down the road read this and it makes something click for them, I'll be happy with that.
    - btrask 6y ago
      Thank you for the article! It's too bad that your conclusion, that systemd doesn't/shouldn't matter, wasn't able to be grasped by the community.

deleted 6y ago

[deleted]

bepvte 6y ago

Interesting comparisons of a GNOME volunteer to Stalin. Also found it very interesting that the author put inclusivity in quotes.

vezzy-fnord 6y ago
Oh, but it is a good comparison. Many people have compared free software to communism. I always used to dismiss it. But, in a way, they were right. Both are movements with a self-conceited historicist endeavor to eliminate exploiting classes ('it is inevitable that the tendency for the rate of profit to fall to produce immiseration as capitalists can no longer profitably invest, leading to a proletarian seizure of power' versus 'it is inevitable that proprietary walled garden development models stagnate as they're overtaken by decentralized communities pooling their knowledge and labor into the most optimal end' and create a perfectly equitable free association of producers no longer having their surplus value be seized by exploiters. One degenerates into a repressive bureaucratic collectivist (I'm not a Trotskyist, but it's a good term) mechanism for reproducing surplus through either direct requisition or coercive planning. The other degenerates into a way for large-scale cloud providers to appropriate the free labor of volunteers so as to reinvest it back in their walled gardens. Stalin represented the Leninist wing against ultra-leftists, anarchists and other factions who were against tight top-down organization in favor of 'spontaneous' organizing, and the GNOME/Freedesktop/Red Hat nexus is against the faction in free software that insists on a loosely coordinated and loosely coupled bazaar without a central vision. But in no sense has systemd abolished this, it has only reduced a few nodes at best. EDIT: Actually I should point out that 'free software' per se is a deontological argument, and it's 'open source' specifically which specifically uses these historicist arguments to justify itself. But then open source was unabashedly an effort for corporate sanitization of free software, which most people forgot after it mostly crowded out 'free software.'

ratsmack 6y ago

>systemd still remains poorly understood and understudied from both a technical and social level despite paradoxically having disproportionate levels of attention focused on it. This statement makes no sense. It is well understood and has been studied and critiqued by many, including me. Also, I don't know what "social level" has to do with anything here. It was just created for the commercial aspect of Linux and forced into the relevant distro's by commercial interests... it's just that simple.

JChase2 6y ago
I'd say software doesn't always win out on it's merits, there's almost definitely some social aspects regarding what does and does not get adopted. I do agree the commercial interests probably played the largest role, though. I think beyond the initial knee-jerk one might have towards this it's pretty comprehensive and there's interesting points here that are at least worth exploring.
- acdha 6y ago
  Aren’t those social aspects part of their merits? A big part of why systemd won widespread adoption was by stepping up – the number of people working on the others wasn’t enough to be more competitive. I lost track of the number of Upstart bugs we avoided by switching to systemd, and that had the backing of one of the most popular Linux distributions.
  - JChase2 6y ago
    I suppose so ya, can't sustain without a community or company unless you want to do all the work yourself.
    - acdha 6y ago
      Another factor I was thinking is that almost everything in systemd is there because someone saw a need, even if it’s a niche – things like mounts aren’t a big deal for many people but the people who had something which wasn’t well served before wanted something better in a new system. Trying to replace a core component will flush out a lot of those smaller communities.

acdha 6y ago

I think this really sums this article up: > One thing I’m certain of is that this shift cannot emerge from dilettantes, outsiders and proverbial basement hackers. One does not unseat a platform without already being part of the patriciate that calls the shots on what gets integrated where across the largest nodes in the ecosystem. It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work. An init system is harder than it might seem at first and requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators. If you’re not going to invest that level of engineering time, I don’t see how it’s reasonable to expect an equal voting share with those who do.

throw0101a 6y ago
> An init system is harder than it might seem at first […] systemd-as-init-system is not the problem. systemd-as-kitchen-sink is the problem. It's the tight coupling that annoys many people. Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?) What does systemd 245's systemd-homed have to do with an init replacement? * https://www.techrepublic.com/article/linux-home-directory-management-is-about-to-undergo-major-change/ https://www.techrepublic.com/article/linux-home-directory-ma...
- ArchD 6y ago
  Yeah, don't get me started on the fragile glass test tube called systemd-resolved that fails in mysterious ways when I use wicd on Ubuntu and the WiFi gets momentarily disconnected. I don't see the sense behind treating name resolution as something so special that it has to be part of an init system. I like the overall theoretical concept of systemd but it has ugly implementation details like this.
  - Spivak 6y ago

zozbot234 6y ago

Systemd is an amazingly clear example of the second-system effect, as described in The Mythical Man Month. I fully expect it to be replaced down the line by something dramatically simpler and more intuitive, but that might take some time. Nonetheless, it does seem to solve some real problems with the earlier, rc-scripts approach.

0zymandiass 6y ago
It's dramatically simpler and more intuitive than what it replaced, so I'm super excited when someone comes up with something even better!
- downerending 6y ago
  Systemd has many pluses, but simplicity and intuitiveness are most certainly not among them.
  - josteink 6y ago
    Simple... for what? I’d like to setup a service, which depends on another service, and which must always be running, and if it goes down, it needs to have all forked processes killed, must be restarted, and it needs to run as a specific user. With systemd? 5 lines or so of boilerplate, independent of distro. I don’t know about you, but I call that simple.
    - kelnos 6y ago
      I think this subthread is talking about the complexity of a system's implementation, not that of its user interface.
      - josteink 6y ago
        As a user, systemd wins for me hands down, because it is simple to use. That it masks complexity away from me is a feature, not a bug. This is also why distros is implementing it almost everywhere: It makes their job easier. Now if you are measuring the complexity of the system's (full) implementation, you can't really compare systemd to to SysV-init. You need to compare a systemd-based system to a SysV-init based system. And then you need to account for the 100s of inconsistently written shell-scripts, all the code distributed in the mess of third party dependencies (sh, bash, Perl, Awk, Python, su, supervisord, cron, etc) to ensure that the functionality of the systems you are comparing is really somewhat equal, not to mention the added complexity of the glue between all those components, and how it may fail. And when you do that comparison, I do believe you will end up concluding that 1. full systems are somewhat complex, for both systems, and 2. that systemd-based systems are not fundamentally more complex, and 3. systemd-based systems have the benefit of a single, centralized implementation for all these core functions, so that developers don't have to reimplement them inconsistently, and possibly buggy. With systemd, for better or worse, either your entire init-chain is broken or it isn't, so when it works, you know it works. With SysV-init you have no such guarantees.

ttctciyf 6y ago

As a somewhat dilettante and casual home sysadmin (currently) I was messing around with screen on a box downstairs and ran into this: https://www.reddit.com/r/programming/comments/4ldewx/systemd_kills_screen_and_tmux_by_default_on_logout/ https://www.reddit.com/r/programming/comments/4ldewx/systemd... Namely that systemd doesn't allow persistent processes started from the shell by default, preferring to terminate them when the user logs out. This would include processes like "screen" whose entire raison d'etre is to persist after the user logs out. (Well, it has other uses, but this is the main one IMO.) The stated workarounds - fiddling with some options like "KillUserProcesses=no" in logind.conf &co. - have so far failed. I don't know whether this situation is a problem with systemd or the distro, but it seems very much a problem with the culture summarised by the top commenter in the above thread, of (paraphrasing) glibly breaking existing workflows then casually brushing away criticism with arguments often boiling down to: "this is the right way, I don't care about tradition or protecting 'incorrect' usage."

zozbot234 6y ago
This is yet another example of pointless incidental complexity in systemd. The whole point of "nohup"-based tools like tmux and screen is to cleanly separate the management of user sessions from the incidental mechanism of whether a remote connection is being closed (the 'HUP' in nohup is short for "hang up" i.e. close a [possibly remote] connection). Systemd should simply acknowledge this fact and keep the user session going when a program has been launched under nohup, instead it tightly couples its own "session" concept to the remote connection and then adds a totally ad-hoc, hacked-together feature called EnableLinger to somehow make nohup work anyway. It's amazing.
- pknopf 6y ago
  I'm not seeing a lot of hope here. There is an incredible investment in systemd by distributions, there is no turning back. Help me..
  - BeetleB 6y ago
    Gentoo is reaching out to you.

nine_k 6y ago

My beef with systemd is not its reinventing things. That part may actually be good. One problem is that a number of reinventions were poorly made. E.g. the log format. Yes, unstructured logs have a ton of drawbacks. Can we take some ridiculously well-tested, reliable embedded database or serialization format (like sqlite) and use it as log storage? Alas. Another problem is the "I know better" attitude. Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior. No wonder I don't run systemd on my desktops / laptops.

m45t3r 6y ago
> Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior. All those examples you gave are possible to disable in systemd, and in the case of DNS and user process after logout, they are not even the default. : Killing of user process are enabled by default in upstream but disabled by default in every distro that I know. And systemd-resolved, even if it is build by default, is not enabled by default even in upstream.
- nine_k 6y ago
  Good thing sanity prevails in distro maintainers! But not in upstream, as you duly note. I also remember that even 2-3 years ago the situation was not as nice yet. I think what became systemd could have been great software, if not for the, mmm, cavalier attitude of its creators (not necessarily personal; it might be amended by RH corporate deadlines).
  - hnarn 6y ago
    "Not in upstream" doesn't seem to correspond with the comment you replied to.

wired_devil 6y ago

systemd is trash

aganame 6y ago

I wish systemd haters would make a public petition to remove systemd from this world. I would absolutely use such a list to make sure I never accidentally hire any one of them for any system administration positions.

dman 6y ago
As a person who does not work as a system administrator professionally but does use self administered linux machines as workstations systemd has not brought any new productivity gains but it sure has increased the learning curve.
- aganame 6y ago
  As a person for whom Linux and BSDs have been a hobby since 1996, I have no idea what you're talking about.
  - jacquesm 6y ago
    A hobbyist making hiring decisions and maintaining a blacklist of professionals who bitch about a change they had no control over? You're not helping.
    - deleted 6y ago
      [deleted]
  - tux1968 6y ago
    One small example that i've struggled with is logging. It used to be I could reuse my editor knowledge to search and read log files. Now every time I want to look at a log I have to re-read the man pages to figure out the proper incantation.
    - aganame 6y ago
      Learning new tricks gets harder when we get older. Is that the software’s fault?

gorgoiler 6y ago

When writing software, it’s comparatively easy to come up with grand new ideas and turn them into lines of code and files filled with modules. What’s much harder is (1) to expand ones code without coupling everything together (2) present it in a persuasive way that naturally builds a user base. SystemD feels like the new grad hire who decided the first thing the would do after joining — the opening power play — is to convince the PHB we should rewrite everything in Haskell. At this point the gentler folk start quietly honing their CVs and sneaking their personal belongings off their desks, one night at a time, in the hope no one notices they are all about to quit. It felt like Unix systems administration was a haven from this sort of politics — I don’t remember tmux, git, or python trying to push behavior changes on upstream components of the OS[1]. I’m sure it’s why Unix attracted a certain type of personality for so many years and I’m sad to see that changing. [1] If you are thinking “but those are command line utilities, whereas the init process is a much more specialized case” then I’d encourage you to revisit the Unix principles of every component being a small and simple program! Even init, cron, and login! It’s not some grand ideal to be zealously adhered to: it’s the principles of small components that mean the same OS can run on a quad core Xeon as runs on a $45 network switch.

theamk 6y ago
I think you underestimate how badly the old sysvinit sucked in the world with many good distribution-provided packages. It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages: Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script. This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate. Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.

noitpmeder 6y ago

My worst gripe with systemd is that I cannot find reliable docs for older versions.

nineteen999 6y ago
Right. All we need now is systemd-selfupdated, so that it constantly keeps itself up to date, even through egress filtering firewalls.
fsh 6y ago
Your manpages should match the installed version.

reacharavindh 6y ago

God I wish more people used VoidLinux and packaged more and more stuff without needing Systemd. Systemd may be good for some, bad for some. But, it certainly should not be the only way to get things done. We need alternatives. It’s as if web developers were told now that there is IE, that is prevalent, we can write web apps that only work there ;-)

nyanpasu64 6y ago

I don't fully understand systemd, but the technical critique looks interesting and seems to indicate systemd is poorly designed (but I don't understand it all). Anyone else commenting on that?

JdeBP 6y ago
Enjoy https://blog.darknedgy.net/technology/2015/10/11/0/ https://blog.darknedgy.net/technology/2015/10/11/0/ , by the same author.

k_bx 6y ago

Can I just say, from a non-sysadmin perspective, how happy am I that these days I don't have to install something like Supervisord and just use the systemd + journald toolchain. Setup script is just one cp and few systemctl commands (start/stop/enable).

dijit 6y ago
Yeah, that's nice. I think there's a lot of people who get caught up in the merits of systemd and assume that there's nothing better. Or that the alternative is the old sysvinit which was in dire need of being replaced- but that's not true. Process supervision is totally feasible in init (just look at runit, Solaris' SMF and MacOS's launchd) even Windows has one.

mr_isomies 6y ago

Getting 10 engineers to agree on how to implement a padding function is nearly impossible. Was there ever hope to get thousands to agree at something that front and center? There was literally decades to come up with a solution. I'm sure things where proposed during that time. What systemd did right was get (or coerce) buy-in from enough entities & people to get a big enough purchase to grow. For me, that's the takeaway. It's not on the engineering merit of their solution.

dvfjsdhgfv 6y ago

A key quote: > As it turns out, there was a little-known Freedesktop-affiliated project called xdg-hostname in 2009 which was a start towards creating such D-Bus “utility daemons” for use in desktop widgets and others, all as a standalone project. Had this been followed through instead of having the utility daemons end up being part of the systemd source tree, a good deal of political acrimony could have been avoided – though at the cost of reducing systemd’s leverage. That is, if they used xdg-hostname instead, SystemD wouldn't be tightly coupled with Gnome, so users could choose between SystemD and InitV, and all this mess could have been avoided.

Stierlitz 6y ago

TLDR: down with sYstemd :]

arh68 6y ago

PulseAudio is the only program I've seen really stress my Ryzen. I recommend fixing the default sampling config if you have similar issues. This is a really entertaining writeup, given the subject matter.

Koshkin 6y ago

My biggest gripe with it is that its name is not ‘sysd.’

keymone 6y ago

One thing I’m surprised about in the init wars of past decade(s?) is that in almost half a century of Unix nobody got up and said “why don’t we, instead of bickering about implementation details, come up with single format to declare desired system startup behavior, and then users can pick whichever implementation they like to read and execute that format”? Can’t be impossible to distill some basics about service startup configuration and dependencies, can it? Then make it extensible and cry about differences in extension support like we do about per-browser css options, but at least converge on some basics.

AndrewStephens 6y ago

I do really appreciate this type of long form, researched blog post. I have no strong opinions on systemd, I've made simple service units and it has worked well enough. Having the history behind the design is a great aid to understanding.

JdeBP 6y ago
Then you will probably enjoy these: * https://blog.darknedgy.net/technology/2015/10/11/0/ https://blog.darknedgy.net/technology/2015/10/11/0/ * https://blog.darknedgy.net/technology/2015/09/05/0/ https://blog.darknedgy.net/technology/2015/09/05/0/ * https://blog.darknedgy.net/technology/2016/01/01/0/ https://blog.darknedgy.net/technology/2016/01/01/0/ * https://blog.afoolishmanifesto.com/tags/supervisors/ https://blog.afoolishmanifesto.com/tags/supervisors/

bandrami 6y ago

10 years ago I never had shutdowns hang with "Stop job waiting for PID 10834: 30 seconds... 90 seconds..." I do now. Regressions are a bad thing.

bozobuttz 6y ago

Systemd is an annoying turd. Still hate dealing with it to this day. I have to refer to the man pages every. single. time.