11 ms·
I think you underestimate how badly the old sysvinit sucked in the world with many good distribution-provided packages. It was pretty good in the good, old day
by theamk 7y ago
I think you underestimate how badly the old sysvinit sucked in the world with many good distribution-provided packages.
It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages:
Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script.
This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate.
Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.
- AnthonyMouse 7y agoWhat you are missing is that it is possible for two things to be true at once. It is possible for sysvinit to suck and for systemd to suck in entirely different and potentially much worse ways. It's as though someone is complaining about having to live under the USSR and how bad things are, and then someone responds that the German rocket program has been dismantled and the British never really had much of one and the Americans are doing okay but the fact is that the USSR was the first to put a man into space. As if that justifies everything else they did.
- cycloptic 7y agoSysvinit is still around, you can always go back to it if you really want. The package is even still in debian.
- AnthonyMouse 7y agoThe problem is not that sysvinit doesn't exist, it's that other packages now have dependencies on systemd, so it doesn't just swap out.
- cycloptic 7y agoI don't see how that is the problem. It's exactly what you asked for. If you're replacing a low-level component with something else that hasn't been updated in years and doesn't implement features that other packages do then yeah, you have to be prepared to accept that some newer things will break, and you'll have to revert a lot of other packages to older versions too. What else can we do? This isn't even a systemd-specific thing.
- AnthonyMouse 7y agoThey didn't used to break if you wanted to switch from one init system to another. What you could do is to reduce unnecessary interdependencies. And where there is inherently interaction between separate packages, use a stable interface and publish the standard so that alternative implementations can replace a single package in a standard way without breaking several others.
- cycloptic 7y agoYes, they did. It they didn't break it was because those inits were compatible with sysvinit scripts. You lose that once you start porting your init scripts over to rc or whatever it is you replaced your init with, and now your system will break if you try to switch to something else.
- AnthonyMouse 7y agoYour init scripts would have to be replaced, that's to be expected, and when multiple ordinary init systems are popular, packages could easily support more than one since an init script is generally only a few lines. I'm talking about all the things that now break not related to the normal function of init at all.
- BiteCode_dev 7y agoI hear that a lot. It's not my experience. Setuping services has been a pain for 15 years for me. Then with systemd it was suddenly easy. Detractors keep repeating systemd sucks, but that this point, Debian and Fedora apparently decided it was good enough. And I never heard an end user complain. I've stopped to be open to convincing.
- nlgza 7y agoIt is even easier with daemontools, which is much more lightweight and follows the Unix philosophy. But somehow distro maintainers can never integrate djbware for political reasons. In the case of RedHat, also NIH syndrome.
- AsyncAwait 7y agoExactly, my biggest problem with the systemd complainers is that it's all kind of abstract things, like it's not the Unix philosophy, which in practice doesn't mean much and certainly doesn't automatically mean it's a bad thing. SysVinit was a horrible, inconsistent mess of bash scripts of per-individual-developer variations in quality and not insignificant variations between distros, or even individual services within the same disto, I wouldn't hold that as the cornerstone of the Unix philosophy, that is if one's intention is to paint it in a positive light.
- mprovost 7y agoThe article did a good job articulating that the "end users" of an init system aren't sysadmins, they're distro maintainers. If most distros have switched then it must make things better for them.
- AnthonyMouse 7y agoThe problem, of course, being that for most init systems the "end users" are the distro maintainers but for systemd, because it spreads itself into so many other things, it also impacts the sysadmins and the upstream third party package developers. Who then complain about it and are met with "it was good enough for Debian and Fedora" even though they aren't the ones complaining.
- api 7y agoThat's exactly my view. We had a chance to replace sysvinit with something better. Instead we managed to replace it with something that's horrible in an entirely different set of ways. The most devastating thing I can say about it is that it feels "enterprise." It's unnecessarily obtuse, complex, and clunky. It feels like a port from some other operating system that's been shoehorned into Linux, or like something designed to sell consulting services and training courses by wrapping very simple things in needless complexity. We could have ended up with something more straightforward, clean, and simple. It's a tragedy.
- rstuart4133 7y ago> What you are missing is that it is possible for two things to be true at once. Actually, it's possible for more than two things to be true at once. It could be: 1. It could be true sysv init sucked. (Actually, it wasn't that bad as a modular version of local.rc, but parallel and delayed startup was always just a kludge). 2. It could be true that systemd is a good init system. (For me it's a step change improvement sysv init for parallel and delayed start up. About my only complaint is you don't see failures on "systemctl start xxx"; you have to run journalctl.) 3. It could be that adding another 1M lines of code into the systemd source unrelated to starting stuff was an unsound engineering decision - almost inexplicable coming as it does from a senior engineer. Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud. 4. It could be that moving sending state changes via an RPC interface (dbus) so you have a million places you have to inspect to understand the state of the system and how it got there rather than writing the desired state of the system to disk with a journal where everyone can inspect it was a bad idea.
- rezonant 6y ago> Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud. I don't think this invalidates the thrust of your point, and I don't necessarily subscribe to the concept of monorepos, but Google is a monorepo, so senior engineers are definitely signing off on this sort of thing. Is it a ball of mud though? Actually does seem true more and more as time goes on. Probably the biggest difference is that well-disciplined engineers utilizing monorepo do sustain those well-defined interfaces. The example of systemd-logind having no interface guarantees (cited in the linked article) is a sign of trouble.
- gorgoiler 7y agoA lot of init problems you describe are very real, but it felt like by the late 00s they had mostly been solved with efforts like linting and standards, as well as LSB init shell functions making init a much more standardized process. PAM was and still is quite hard to understand but, again, it had enough power and flexibility with @ based includes to allow individual distributions to lay things out sensibly and allow new packages to follow a coherent standard that emerged rather than being forced. Concurrent init systems helped drive a lot of this, back when there was a sweet spot of Linux complexity increasing faster than processing power could handle. (Nowadays I feel like boot speed is less important, given how many more resources a modern machine has, though it’s still a concern for low resource platforms.) I guess I’ve just never seen revolution as a solution to hacks and untidiness. Linting, guidelines and policy, with technically compelling reasons to encourage people to follow them have seen much more success, where I’ve seen them deployed, than centralized authority.
- Conan_Kudo 7y ago> A lot of init problems you describe are very real, but it felt like by the late 00s they had mostly been solved with efforts like linting and standards, as well as LSB init shell functions making init a much more standardized process. These improvements weren't real. There was a long tail of scripts that didn't follow the standard and never would. Every distribution still had their own customizations on how sysvinit scripts worked (e.g. debian with start-stop-daemon, suse with insserv, etc.) And rc was different on almost every major distro family, which made life even more difficult for no particularly good reason. And there's a certain point where all these things you are doing are ultimately to paper over the "hacks and untidiness" of everything. The introduction of systemd had two major effects: 1. It broke all non-LSB init scripts (because LSB is all it supported) 2. It introduced a straightforward declarative syntax for system resources (units for services, timers, mounts, etc.) I would argue that the above two things have done more to standardize the Linux plumbing layer and help people take fuller advantage of it than literally all the linting, guidelines, and policywork we did for the decade prior.
- dTal 7y agoAnd if that's all it did, we wouldn't be having this discussion. Instead it is becoming some sort of self-appointed "userland middleware", which nobody asked for. People resent "better init" turning into "henceforth home directories will behave differently".
- jbverschoor 7y agoExactly. I was immensely frustrated with sysvinit for over 15 years. Jumped over to systemd as soon as it was available on Debian, never looked back, only one had a problem due to symlinking.
- kelnos 7y ago> Debian, Ubuntu, Arch all chose the systemd voluntarily. My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to if they wanted a reasonable default GNOME experience for their users. If they hadn't, GNOME would have been missing some features that would have been obvious and jarring. You can certainly argue that they made the wrong call there, and that they should have pushed back (though arguably by the time Debian got there, that ship had already sailed), but "chose systemd voluntarily" is an incorrect simplification.
- pseudalopex 7y agoAnd Ubuntu followed Debian. Mark Shuttleworth called it "losing graciously".[1] [1] https://www.markshuttleworth.com/archives/1316 https://www.markshuttleworth.com/archives/1316
- JoshTriplett 7y ago> My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to This (and much of the article) is incorrect. Debian went through an extensive evaluation process at the time, over the course of 4 months, considering several different init systems, and chose systemd based on this detailed technical evaluation. That evaluation also considered sysvinit, openrc, and upstart. All of that evaluation occurred publicly, and is archived on Debian's mailing lists and bug-tracking system.
- vezzy-fnord 7y agoAll of my views come straight from primary sources which I meticulously uncovered, including Debian mailing lists. I realize that you have been one of the more visible members of the pro-systemd side for years now, so I'd definitely like a more coherent rebuttal to both the historical and technical sections. Of course, you'd probably consider any kind of response to be undignified, and I understand.
- JoshTriplett 7y ago
- vezzy-fnord 7y agoThe problem with this straightforward thesis is twofold: firstly, distribution developers did not have any coherent position on the disadvantages of sysvinit or initscripts; secondly, the problem was mostly an iatrogenic one, as the very attempts to 'solve' or 'reform' it made things even less sustainable. When it comes to Upstart, it was almost never used except as a wrapper around initscripts. Effectively you booted from Upstart which greedily synthesized all of its 'events' on startup that in turn triggered initscripts, which brought some degree of parallelism and that was about it. The reverse-dependency 'start/stop on' model was too esoteric, and Upstart's state machine brittle. Hilariously enough, the only system to actually use Upstart to its fullest (still does) is ChromeOS. Prior to systemd and Upstart, as I discuss in chapter 1, distro developers either simply added various preprocessors for initscripts to introduce dependency headers, or they had these vague hunches of 'waiting for Godot' that we could either replace init entirely with D-Bus, put D-Bus interfaces in initscripts, or some other roundabout way that involved grafting D-Bus. Much of this was owing to the 'irrational exuberance' of Linux developers wanting to hotplug the world after 2003-4, but it always remains an on-and-off effort with the people behind it never certain what they want. Indeed, besides grunt work of optimizing initscripts, I point out that as late as 2009 distros like Debian were still very concerned about LSB-compliance, and that even when they were willing to adopt a new fancy event-driven init like Upstart, it was conditional on reusing tools like insserv to parse LSB headers, and on incorporating initscripts for service management. Also the entire bizarre diversion that the Debian group had of using a Perl script to generate initscripts. So far as I know, the daemontools approach was never influential. The available dichotomy was between putting lipstick on a pig, and utopian dreaming. The paradox remains: how can we trust that the same people who did the Wrong Thing so persistently then suddenly turned out and did the Right Thing like they hit jackpot? If anything, 'jackpot' is the right analogy -- it was a fluke. And the same integration failures that were had with sysvinit have now been shifted into wrestling systemd's job model. "Unification" was a folly, as the distro is still the place that gets its finger pointed at if upstream can't deliver, and upstream insists it's "just" offering a bunch of "modular tools," which ironically was supposed to be the problem to be addressed! Besides, there is still a whole world of Linux devices out there running Busybox, musl, OpenWrt, etc. that are outside the systemd bubble. Or think of all the container images running Alpine (which is also the basis of postmarketOS). Linux is still the same bazaar it always logically must be, but with a new class distinction added to the mix.
- Conan_Kudo 7y ago> Remember, even if you can claim for Fedora that someone "convinced the PHB" If you looked at the discussions that happened during that time (I was there as a Fedora contributor back then, and wow that was a long time ago!), you'll see that we mostly did the change out of weariness of trying to use Upstart features well and failing. And even with that, we delayed it by one release because it wasn't ready during the Fedora 14 development timeframe (it was supposed to be in Fedora 14, we actually launched it in Fedora 15).
- StreamBright 7y agoYou are describing the downside of bazaar-style development. Init had very little to do with it. The real problem was a non-standard way of writing init scripts. That is the _only_ part that systemd was supposed to solve.
- dvfjsdhgfv 7y ago> User A likes to leave their executables on their machine after logout, so we need a cleanup script. Of all examples you gave, this one is the least convincing reason to introduce this Behemoth on our systems.
- x3blah 7y agoSounds more like the problem is with the packages and a general lack of quality control with respect to the userland than with GNU init. Perhaps that is why BSD projects, which do have some quality control over userland, can still use BSD init.
- voldacar 7y agoThe parent comment is criticizing systemd, not upholding the virtues of sysvinit. There are more than two init systems out there, and it is possible for sysvinit and systemd to both be bad in their own ways