21 ms·

The importance of Devuan

129 points by telmich 9y ago

moreentropy 9y ago
I think most systemd criticism misses the point and revolves too much around usage semantics. Systemd allows actual resource management on a level that simply didn't exist with traditional init [1]. Yes we can argue about implementation, but the necessity of better resource management in more dense and shared environments can't just be ignored, especially when building a shared VM hosting platform like the author is talking about. [1] http://0pointer.de/blog/projects/resources.html http://0pointer.de/blog/projects/resources.html
- zAy0LfpBZLC8mAC 9y ago
  I don't think many people disagree that systemd implements some useful ideas. The problem is the overall architecture and the quality of the code.
  - geertj 9y ago
    Some examples of poor quality of code? This has not been my experience. I've gone through the code base a couple of times for my own education and to do some small patches, and I found it clean and easy to read.
    - zAy0LfpBZLC8mAC 9y ago
      The problem isn't so much "typical programming errors", but the appoach to solving problems. So, it's not that there are tons of buffer overflows in the code (well, not that I know of, though I didn't explicitly look for them either), but stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades, or the bug that caused .* to match .. and thus systems to be destroyed by the temp cleanup code, which is equally a potential pitfall that is well-documented and explicitly even standardized against in POSIX, but they still failed to implement this in a way that is both evidently sane and also otherwise generally accepted to be the correct way to do things ... and then they even pretended that it's somehow not actually a bug. Those are two somewhat recent examples I can think of, but there have been many more showing the same kind of attitude. They are ignoring hard-earned knowledge on how to do things securely, safely and reliably, they ignore the intentions of abstractions, and they pretend that obvious bugs aren't bugs because there is some weird way to reinterpret the bug into being correct behaviour (that noone expects and that causes harm).
      - deleted 9y ago
        [deleted]
      - someone12345 9y ago
        > stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades By domain knowledge how to do this securely in implementations tested oer time, do you mean something like https://access.redhat.com/articles/2161461 https://access.redhat.com/articles/2161461? It doesn't look like systemd's resolver fares worse in comparison... In addition it can be sandboxed (and is), an advantage over having a resolver part of libc.
        zAy0LfpBZLC8mAC 9y ago
        Thank you for the demonstration, that is indeed roughly what that lack of domain knowledge combined with arrogance looks like. Would you mind explaining how exactly sandboxing prevents cache poisoning?
  - moreentropy 9y ago
    I've had way more headache with badly written and plain faulty init scripts than with systemd, including systemd-networkd, systemd-timesyncd, systemd-resolved and systemd-nspawn, which I used to replace a lot of LXC, LXD and openvz setups.
    - yupyup 9y ago
      Please, could you share a little of your experience with systemd-nspawn vs LXD? I have used LXD (with ZFS storage) in production but have not played with systemd-nspawn... Thanks!
      - moreentropy 9y ago
        Well it just works and is extremely simple to use. I usually debootstrap into /var/lib/machines/something and do "machinectl enable something; machinectl start something", that's it. Then I attach to the machine using "machienctl shell something" and configure networking (host0 interface) inside the domain, that's it. For drop in configuration systemd-nspawn parses a config file /etc/systemd/nspawn/something.nspawn which usually just contains network configuration on my hosts: [Network] Bridge=br-int Systemd-nspawn enables and user namespacing by default and chowns the machines's root filesystem on first start. If that's not desired (Things like Samba fileservers don't work well with user namespacing) just disable it in the .nspawn file: [Exec] PrivateUsers=no Everything you need to know is in the manpages systemd-nspawn and systemd.nspawn. I usually install systemd from stretch-backports because running a fairly recent systemd version helps as it still gets new features, but I never had problems with stability.
        yupyup 9y ago
        Great, sounds quite simple. One thing I somewhat miss from what you are explaining is all the aditional things that LXD gets you (snapshots using ZFS, image publishing/sharing, migrating containers between LXD hosts...) But maybe some of those things are still doable (e.g. mounting a ZFS dataset as storage for /var/lib/machines/containerX)... Thanks for your answer!
- telmich 9y ago
  I think them implementation is, what sucks. Not the idea itself.
- pmontra 9y ago
  The conclusion of the post is: > the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis. They lament > servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration Using systemd was costing them too much. Moving back to the previous init system looks like a rational choice for them. My experience is different but I only manage a handful of virtual servers and not on a daily basis, plus my laptop. Systemd configuration files are not difficult to write and they restart the daemons if they crash. For complex stuff I make systemd run a bash script that eventually executes a daemon, kind of cheating. My laptop still runs well. Booting time is definitely not an issue, it went from fast once per month or so (kernel upgrades), to fast still once per month so. I didn't notice any difference after the change of the init system. A good thing but maybe it means that the return on investment was dubious for this use case. Binary log files are objectively worse than text ones. cat, less and tail were good enough and shorter to type than journalctl. Ok, I could alias it to a four characters word but it was still a lot of work that could have been invested on some other goal. Instead I've got servers with possibly compact binary logs made of very few lines and large text logs from web applications. I'm also puzzled by the philosophy of bundling more things together and tighten dependencies. It's somewhat disconcerting and I'd like a system where we could swap components out more freely, but this is an opinion and not facts.
  - someone12345 9y ago
    > They lament >> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration Which is interesting given Debian doesn't use systemd-resolved (unless manually configured). So they made up problems that don't exist by default?
  - gerbilly 9y ago
    > Systemd configuration files are not difficult to write and they restart the daemons if they crash. This I have a problem with, if it crashes, I'd prefer it stay crashed, instead of crashing over and over. At least when it crashes, a human can come and see why it's crashing, rather than doing the caveman thing and just restarting it over and over.
- benchaney 9y ago
  Actually, I think that you are missing the point. It’s true that systemd implements some good ideas that did not exist in other init systems at the time, but most of those feature are pretty niche, and more importantly, you can not evaluate software purely on how good it’s ideas are. You have to consider how good it actually is. Systemd has real flaws that cause real problems for real people. The sooner everyone understands this the better.
dvfjsdhgfv 9y ago
I spprt Devuan because I support choice. And I hope one day - be it 5 or 10 years - we arrive to the point where Devuan is no longer needed - you will be able to replace one init system with another in Debian. Currently it's not possible and the system I've been using since 1998 became something else that I can't control in a way that was possible until now. When I ask my fellow colleagues (mostly sysadmins), some of them like Systemd, some don't, but we all agree that in this particular case the freedom of choice is much needed.
- icebraining 9y ago
  Why can't you run Debian with another init system? Some things probably won't work, but wasn't that always the case? Sincerely asking, I never tried running a non-default init system.
  - rini17 9y ago
    Debian decided to not support using any other init system. If it was just "same breakage as always", devuan would not be created.
    - icebraining 9y ago
      Debian decided to not support using any other init system. Right, but did they support using any other init system (besides sysv) before?
    - bigon 9y ago
      That's actually not true at all... * https://packages.qa.debian.org/s/systemd.html * https://packages.qa.debian.org/u/upstart.html (it's gone from stretch that's true but still supported in jessie) * https://packages.qa.debian.org/o/openrc.html * https://packages.qa.debian.org/s/sysvinit.html
- agwa 9y ago
  It's really easy to replace systemd with sysvinit on Debian: https://news.ycombinator.com/item?id=14581820 https://news.ycombinator.com/item?id=14581820 All my Debian systems run sysvinit, and they work great - certainly better than the one system I co-administer which runs systemd. I frankly do not understand why Devuan exists.
badrabbit 9y ago
So, the whole thing with systemd is not that "it's bad" but that "I don't like it and I am stuck with it". Moreover, it leaves those of us who don't like it disenfranchised by distros that insist on using only systemd Opensource is about alternatives. Forks are a good thing,I wish devuan great success so people like myself won't have to whine about systemd all the time. If you like systemd and it's philosophy,I wish you the best of luck with it. Just keep in mind that others don't have to like it and different people or organizations have different needs.
- telmich 9y ago
  So true!
- buu700 9y ago
  My thoughts exactly. I personally have nothing against systemd and haven't been directly affected since I rarely need to touch init scripts, but I still appreciate Devuan and loosely follow its progress for this reason. (Kind of like ReactOS — 99% I'll never want or need it, but the fact that it exists is still really cool.)
  - cjsuk 9y ago
    I was a systemd proponent until I had a hosed box. At that point I learned that alternatives are important. It made my life very difficult due to how opaque it is. Whike problems are surmountable, rehearsing for failure is advised.
- binaryapparatus 9y ago
  It is also plain bad but even more important is lock in that you mention. Bugs can be fixed but wrong philosophy can't.
  - johnny22 9y ago
    We've been living with sysvinit style systems for years, and were able to transition away. We could easily do it again.
cosarara97 9y ago
You can run a traditional init in debian. Systemd is just the default.
- telmich 9y ago
  That might be true, but you cannot get rid of systemd in Debian
  - duckerude 9y ago
    You can just uninstall it. It looks like apt remove systemd would work on my system, at the cost of GNOME and NetworkManager. APT automatically fills the gap with sysvinit and consolekit. OpenRC is also available. Debian even has ports to kFreeBSD and HURD that don't support systemd at all. There were some rough edges last time I ran Debian without systemd, and it might have gotten worse since then, but it surprises me how Debian ended up at the center of the debate when it doesn't seem that bad as systemd-defaulting distros go.
    - greenhouse_gas 9y ago
      >kFreeBSD It seems dead.
telmich 9y ago
Btw, I wanted to thank everybody for upvoting this post! For me it is not only a Sunday morning essay, but also important that people understand, why the Devuan movement is so important to all of us.
majewsky 9y ago
> We tried to build Data Center Light on Debian and Ubuntu, but servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration made it too expensive to run Debian or Ubuntu. Can you expand on that? I work at a cloud provider, and I never heard of such issues before. Our default images are Ubuntu 16.04, RHEL 7 and SLES 12, so systemd all around.
- telmich 9y ago
  Sure. With huge storage arrays, fsck might take a long time. systemd by defaults aborts after some time and will not mount that array.
  - jamiesonbecker 9y ago
    So your file system check is killed and leaves your array in a possibly corrupt state? That is simply insane.
  - majewsky 9y ago
    Did you file that in the systemd issue tracker? I had a quick look and couldn't see a report there: https://github.com/systemd/systemd/issues?utf8=%E2%9C%93&q=is%3Aissue+fsck+timeout https://github.com/systemd/systemd/issues?utf8=%E2%9C%93&q=i... This could be an honest mistake. On my system, systemd-fsck-root.service and systemd-fsck@.service have `TimeoutSec = 0`, which I guess to mean "no timeout", but the systemd-fsck executable could have its own timeout. In any case, filing a bugreport sounds much easier than switching distributions.
- telmich 9y ago
  Regarding not rebooting: sometimes systemd hangs and does not reboot in our setups, but we were never able to sufficiently debug this, as all terminals are already gone and it seems that systemds is waiting in a loop for dependencies that don't solve.
  - moreentropy 9y ago
    Have you tried passing 'debug' instead of 'quiet splash' to the kernel cmdline and doing halt instead of poweroff/reboot? That way you should see what's going on during shutdown.
turbinerneiter 9y ago
Soooooo, I'm part of a CubeSat team and we use systemd. Extensively. We us their dbus library for our daemons to communicate, some of the ops is basically done in service files. I have to get our systems people to write a paper on it. My experiences with systemd are pretty much perpendicular to what some people claim.
- telmich 9y ago
  It's not about claiming here, it's about experiences. I guess we should open up a youtube channel for filming our most catastrophic experiences with systemd, because otherwise somebody will claim again that its brokeness is just a claim ...
  - chris_wot 9y ago
    Would have been good if the author did this in the original article.
  - lima 9y ago
    Let me tell you about my most catastrophic experiences with sysv... I've had quite a few. Anecdotal evidence is useless. What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd. I run large scale VM infrastructure on a systemd distro (RHEL, in particular) and I've yet to encounter a single issue that was caused by systemd or, for that matter, any issue with systemd that wasn't easily resolved.
    - fao_ 9y ago
      > What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd. That doesn't mean that there aren't any. Also, FUD (as I understand it) is people spreading unfounded rumours, not people documenting their own experiences and problems with a piece of software. The intent is incomparable. If you are actually interested in tangible structural flaws in systemd, look no further: https://suckless.org/sucks/systemd https://suckless.org/sucks/systemd https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awards/ https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awa... Also, see how Poettering handled this zero day (Which grants root to any user with a numeral): https://github.com/systemd/systemd/issues/6237 https://github.com/systemd/systemd/issues/6237 HINT: He doesn't think it's a bug... The fact that this even fucking exists in the first damn place: https://latesthackingnews.com/2017/06/29/a-systemd-vulnerability-allows-attackers-hack-linux-machines-via-malicious-dns-response/ https://latesthackingnews.com/2017/06/29/a-systemd-vulnerabi... See also: http://without-systemd.org/wiki/index.php/Arguments_against_systemd http://without-systemd.org/wiki/index.php/Arguments_against_...
jchw 9y ago
I don't like systemd, but it works pretty well nowadays in my experience and we use it for servers at work. Truth be told, I'm fine with monoliths. They just have to be good. The Linux kernel is a monolith. The real problem to me is how systemd makes the rest of the system more complex and integrated to systemd. Dbus is also not a great IPC layer, and yet it's the current state of the art. We need better APIs and foundations. Systemd doesn't provide those; it just lets everyone depend on systemd. Good for Redhat, bad for innovation.
raziel2p 9y ago
The fact that they weren't able to set up computers to boot using systemd is enough to warrant that I'd never want to buy or rent servers from this company, sorry to say.
- telmich 9y ago
  Question: did you every try to start a system with > 60TB storage attached with systemd?
  - lima 9y ago
    Maybe he didn't, but I did, and it works fine.
  - hpcjoe 9y ago
    I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical storage in a single box. Linked together via infiniband, 40/100 GbE, and others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS). Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well. The greatest pain was the networking. Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking. Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases. My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works. I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over. Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.
wicket 9y ago
> But first, let me put some warning out here: Dear Devuan friends, while I honor your work, I also have to be very honest with you: in theory, you should not have done this. Looking at creating Devuan, which means splitting of Debian, economically, you caused approximately infinite cost. I thought the same at first but after trying to maintain a Debian installation without systemd, I was convinced otherwise. The "infinite cost" was not caused by Devuan developers but by Debian developers. I felt the Debian transition to systemd was handled very badly and was rushed in order to get Jessie out of the door. Debian still provides the sysvinit-core package as if it were a supported init system however if you use it you'll find the operating system has many quirks and many things are broken. Before Jessie was released, I commented on one bug report related to a hard dependency on systemd for NetworkManager. Instead of fixing the regression, the package maintainer gave the excuse that they don't have the resources and would need more maintainers. I sympathise with the package maintainer but this is the sort of thing that led to this mess. I remember there being a lot of hostility in that period from Debian developers towards those who did not wish to use systemd. Based on my experience, I think the right decision was made to "fork" Debian to create Devuan. It's a shame that they had to do this. I still hold hope that maybe one day all of the work put into Devuan can be reintegrated into Debian and it can return to be the "Universal Operating System" that they still claim to be.
- icebraining 9y ago
  I'm pretty agnostic towards systemd, but I don't understand this complaint that Debian now doesn't support other init systems. Did they support other systems besides sysv before?
  - wicket 9y ago
    They also supported upstart but that was dropped in Stretch. We are left with systemd and System V init to provide /sbin/init. Debian still provides System V init which itself works fine but other packages, many of which previously worked, have come to depend systemd only. This type of regression would not be acceptable in other projects but Debian has done nothing about it. I think they should stop pretending that Debian is still usable System V init and just drop the package altogether.
zbentley 9y ago
Every time I read something complaining about systemd from a centralization/reducing-choices POV, I mentally replace the word "systemd" with "glibc" and check to see if the argument still holds water. Mind you, there are plenty of other complaints (stability, back-compatibility, etc.) that are useful/valid regarding this software, but the above litmus serves to eliminate rather a lot of FUD and hand-wringing in my experience.
- benchaney 9y ago
  What criticisms have you eliminated using this strategy? glibc does not do the things that people hate about systemd.
  - zbentley 9y ago
    It filters out: people who just want to complain about centralization; people who are annoyed about change and trying to rationalize it without trying or learning the new tool; and people who irrationally feel like they are no longer in control. Paradoxically, this improves the signal of discussions about systemd's flaws, and there are plenty. With the obviously irrational/haven't-used-it/axe-to-grind stuff filtered out, it's much easier to learn about real features, problems, fixes, benefits, and drawbacks of any project, systemd included.
- greenhouse_gas 9y ago
  >Every time I read something complaining about systemd from a centralization/reducing-choices POV, I mentally replace the word "systemd" with "glibc" and check to see if the argument still holds water. Actually, I replace systemd with *BSD. You know, the OS which has blessed core packages which are not replaceable?
  - petre 9y ago
    Actually the BSDs with their non-replaceable core packages are a sane alternative to Linux w/ systemd.
vgy7ujm 9y ago
There is also Slackware!
- Wildgoose 9y ago
  And Gentoo.
gargravarr 9y ago
I thoroughly support Devuan. I switched my home servers to it around the systemd introduction and kept their complete stability, and now I've been able to influence my company to run it instead of pure Debian. You have to make a few allowances for packages that assume they're running on systemd Debian, but overall the system works exactly as intended. And as commented in the article, it's stable and easy to understand. At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux. In my opinion, the Debian developers were wrong to fully embrace systemd; it goes quite against the Debian principle of being able to adapt your machine to do anything. Yes, Red Hat held a lot of sway in convincing the community to adopt it, but Debian is a big distro as well - just look at the install base of Ubuntu. They could have held out and pushed back, but instead caved and now we're in the systemd mess. It intrudes so massively into userspace that it's impossible to get away from, and we wind up with the Windows model as this article points out - a massively complex spaghetti model of processes, all interlinked, all hinging on everything playing nicely. Just like Windows, it's a neat house of cards that could topple if any one of those crashed. This isn't Linux. I'm not defending sysvinit, there's no denying it's showing its age and harks back to a much simpler time, but systemd is not the answer.
- lima 9y ago
  > At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux. That's far from the only advantage systemd has. Even then, boot times are extremely important for modern VM and container infrastructures.
arca_vorago 9y ago
I still think systemd was a NSA/Redhat backdoor in the midnight for Linux. I've forced myself to deal with it because its everywhere, but my Spidey-senses tingle everytime I find some new way its trying to takeover command of a core function.
lima 9y ago
What irks me more about this blog post than the systemd part - which has been discussed in-depth countless times in the past, with valid arguments for and against - is the decision making and risk assessment process. I've run large-scale infrastructure on both Debian and RHEL, with and without systemd. I've been involved in many distro decisions in the past. The init system was never a major consideration, and I've always been able to work around whatever issues I've had with both sysv and systemd. Here's some considerations which were important for my team the last time we had to make a choice: - Vendor stability and long term support If you're running a business, you'll want long term stability guarantees - both from a technical, and a business point of view. Running a small community distro with few - if any - commercial users is a huge liability. Yes, you always have the option to fork it or take it over, but unless you're in the business of building a Linux distro, it's almost certain to be a bad business decision since your competitors are spending their time on their products instead. Anyone who doesn't appreciate long term vendor stability hasn't yet been burned by the lack of it. Your Gentoo wizard left the company? Too bad, good luck maintaining your infra now (I've actually seen that exact scenario play out not once, but twice!). A few core maintainers left for another project and you're left building your own packages? D'oh. Having a mature and well-funded organization or a large community of commercial users (in the case of Debian) supporting your distro is extremely valuable, even if you aren't the ones paying for it. - Ecosystem The ecosystem is also really important. It's often the main motivation for using a particular distro. Development tools, third party package repos, 3rd party enterprise software, troubleshooting resources, documentation and much more depend on a healthy ecosystem. With Devuan, this isn't as much of an issue, but it's still sufficiently different from Debian in sometimes subtle ways that it will nullify some of the advantages of being in a common ecosystem. This also includes hiring - you'll have a harder time staffing your company if you're using exotic stuff and you'll unnecessarily spend your time training them. Ever wondered why companies like Google publish so many papers, talk about their infrastructure and even publish books about it[1]? It's because it means they can - by advancing the state of the industry - hire people who are already familiar with their architecture and concepts. This is a real problem for them - Google, for instance, is in some cases so far ahead of others that they have to spend a significant amount of resources to bring their new hires up to speed. I don't know about Amazon and Facebook, but I'm sure they have similar problems. [1]: https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html - Security Security follows the supply chain. Your security is only as strong as your distro's - no matter how good your security controls and processes are, if your distribution vendor is compromised, you won't stand a chance unless you have a world-class security team. Likewise, your customers fully rely on you - if you're compromised, they will be, too. Security is much more than signing your packages and publishing security advisories. In fact, those are merely the results of a properly implemented security management framework - risk assessment and mitigation needs to be an integral part of your vendor's (and your) processes. Where do they keep their PGP keys? Who has access to it? Is is stored on a HSM or just sitting on a server somewhere? Is there a process for revoking it? Is there a change review process? Does the build system verify source code integrity? Is there two factor authentication for production? Do the team members keep their SSH keys on a smart card? This list could go on for miles. - Quality Assurance Even with the main distros, QA quality varies wildly. In my personal experience, Red Hat has by far the best QA, followed by Debian and then (with some distance) Ubuntu. I've had particularly bad experiences with Ubuntu, a sentiment shared by many operations people I've talked to (the worst one being the grub timeout bug that made me spend a few days pressing the return key a lot - basically, they removed the default timeout in a minor update and all my machines sat there waiting for someone to manually continue the boot process). Attention to details, even if it's minor and seemingly insignificant, really helps productivity if you sum it up at the end of the day - each bug your vendor catches and each quirk they document is time you won't have to spend with debugging. With smaller distros, this is even worse - in some cases, they don't even have proper QA. Now, let's assume that the issues they had with systemd were as significant as they say they are (I'm doubting it, but unfortunately, they didn't talk about any particular ones), it still might very well have been a better business decision to stick with Debian and spend their time fixing the issues they had instead of migrating to Devuan, which is a HUGE liability compared to plain Debian.
crististm 9y ago
What makes the door responsible for the failures of engine ignition? This is a good analogy to remind Devuan why they rejected systemd in the first place.
bitwize 9y ago
Oh, suck it up. From a business perspective, the most important thing -- indeed, the only thing that truly matters -- isn't the software itself. It's support. When it comes to open source, you either use a supported configuration or you assume ALL the responsibility for supporting your systems yourself. Who would you rather handle the support on the off chance your machine goes tits up? You? Or Red Hat? No wait, scratch that. Who would YOUR BOSS rather handle the support? The supported init system for most flavors of Linux (that aren't in LTS) is systemd. Systemd makes the most sense from a business perspective, and "but muh Unix philosophy" doesn't hold when there's money on the line. May as well get used to it.
- gerbilly 9y ago
  >From a business perspective That's just it, some of us aren't interested in the 'business perspective' and have other uses for our systems.

129 points by telmich 9y ago

I think most systemd criticism misses the point and revolves too much around usage semantics. Systemd allows actual resource management on a level that simply didn't exist with traditional init [1]. Yes we can argue about implementation, but the necessity of better resource management in more dense and shared environments can't just be ignored, especially when building a shared VM hosting platform like the author is talking about. [1] http://0pointer.de/blog/projects/resources.html http://0pointer.de/blog/projects/resources.html

zAy0LfpBZLC8mAC 9y ago
I don't think many people disagree that systemd implements some useful ideas. The problem is the overall architecture and the quality of the code.
- geertj 9y ago
  Some examples of poor quality of code? This has not been my experience. I've gone through the code base a couple of times for my own education and to do some small patches, and I found it clean and easy to read.
  - zAy0LfpBZLC8mAC 9y ago
    The problem isn't so much "typical programming errors", but the appoach to solving problems. So, it's not that there are tons of buffer overflows in the code (well, not that I know of, though I didn't explicitly look for them either), but stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades, or the bug that caused .* to match .. and thus systems to be destroyed by the temp cleanup code, which is equally a potential pitfall that is well-documented and explicitly even standardized against in POSIX, but they still failed to implement this in a way that is both evidently sane and also otherwise generally accepted to be the correct way to do things ... and then they even pretended that it's somehow not actually a bug. Those are two somewhat recent examples I can think of, but there have been many more showing the same kind of attitude. They are ignoring hard-earned knowledge on how to do things securely, safely and reliably, they ignore the intentions of abstractions, and they pretend that obvious bugs aren't bugs because there is some weird way to reinterpret the bug into being correct behaviour (that noone expects and that causes harm).
    - deleted 9y ago
      [deleted]
    - someone12345 9y ago
      > stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades By domain knowledge how to do this securely in implementations tested oer time, do you mean something like https://access.redhat.com/articles/2161461 https://access.redhat.com/articles/2161461? It doesn't look like systemd's resolver fares worse in comparison... In addition it can be sandboxed (and is), an advantage over having a resolver part of libc.
      - zAy0LfpBZLC8mAC 9y ago
        Thank you for the demonstration, that is indeed roughly what that lack of domain knowledge combined with arrogance looks like. Would you mind explaining how exactly sandboxing prevents cache poisoning?
- moreentropy 9y ago
  I've had way more headache with badly written and plain faulty init scripts than with systemd, including systemd-networkd, systemd-timesyncd, systemd-resolved and systemd-nspawn, which I used to replace a lot of LXC, LXD and openvz setups.
  - yupyup 9y ago
    Please, could you share a little of your experience with systemd-nspawn vs LXD? I have used LXD (with ZFS storage) in production but have not played with systemd-nspawn... Thanks!
    - moreentropy 9y ago
      Well it just works and is extremely simple to use. I usually debootstrap into /var/lib/machines/something and do "machinectl enable something; machinectl start something", that's it. Then I attach to the machine using "machienctl shell something" and configure networking (host0 interface) inside the domain, that's it. For drop in configuration systemd-nspawn parses a config file /etc/systemd/nspawn/something.nspawn which usually just contains network configuration on my hosts: [Network] Bridge=br-int Systemd-nspawn enables and user namespacing by default and chowns the machines's root filesystem on first start. If that's not desired (Things like Samba fileservers don't work well with user namespacing) just disable it in the .nspawn file: [Exec] PrivateUsers=no Everything you need to know is in the manpages systemd-nspawn and systemd.nspawn. I usually install systemd from stretch-backports because running a fairly recent systemd version helps as it still gets new features, but I never had problems with stability.
      - yupyup 9y ago
        Great, sounds quite simple. One thing I somewhat miss from what you are explaining is all the aditional things that LXD gets you (snapshots using ZFS, image publishing/sharing, migrating containers between LXD hosts...) But maybe some of those things are still doable (e.g. mounting a ZFS dataset as storage for /var/lib/machines/containerX)... Thanks for your answer!
telmich 9y ago
I think them implementation is, what sucks. Not the idea itself.
pmontra 9y ago
The conclusion of the post is: > the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis. They lament > servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration Using systemd was costing them too much. Moving back to the previous init system looks like a rational choice for them. My experience is different but I only manage a handful of virtual servers and not on a daily basis, plus my laptop. Systemd configuration files are not difficult to write and they restart the daemons if they crash. For complex stuff I make systemd run a bash script that eventually executes a daemon, kind of cheating. My laptop still runs well. Booting time is definitely not an issue, it went from fast once per month or so (kernel upgrades), to fast still once per month so. I didn't notice any difference after the change of the init system. A good thing but maybe it means that the return on investment was dubious for this use case. Binary log files are objectively worse than text ones. cat, less and tail were good enough and shorter to type than journalctl. Ok, I could alias it to a four characters word but it was still a lot of work that could have been invested on some other goal. Instead I've got servers with possibly compact binary logs made of very few lines and large text logs from web applications. I'm also puzzled by the philosophy of bundling more things together and tighten dependencies. It's somewhat disconcerting and I'd like a system where we could swap components out more freely, but this is an opinion and not facts.
- someone12345 9y ago
  > They lament >> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration Which is interesting given Debian doesn't use systemd-resolved (unless manually configured). So they made up problems that don't exist by default?
- gerbilly 9y ago
  > Systemd configuration files are not difficult to write and they restart the daemons if they crash. This I have a problem with, if it crashes, I'd prefer it stay crashed, instead of crashing over and over. At least when it crashes, a human can come and see why it's crashing, rather than doing the caveman thing and just restarting it over and over.
benchaney 9y ago
Actually, I think that you are missing the point. It’s true that systemd implements some good ideas that did not exist in other init systems at the time, but most of those feature are pretty niche, and more importantly, you can not evaluate software purely on how good it’s ideas are. You have to consider how good it actually is. Systemd has real flaws that cause real problems for real people. The sooner everyone understands this the better.

dvfjsdhgfv 9y ago

I spprt Devuan because I support choice. And I hope one day - be it 5 or 10 years - we arrive to the point where Devuan is no longer needed - you will be able to replace one init system with another in Debian. Currently it's not possible and the system I've been using since 1998 became something else that I can't control in a way that was possible until now. When I ask my fellow colleagues (mostly sysadmins), some of them like Systemd, some don't, but we all agree that in this particular case the freedom of choice is much needed.

icebraining 9y ago
Why can't you run Debian with another init system? Some things probably won't work, but wasn't that always the case? Sincerely asking, I never tried running a non-default init system.
- rini17 9y ago
  Debian decided to not support using any other init system. If it was just "same breakage as always", devuan would not be created.
  - icebraining 9y ago
    Debian decided to not support using any other init system. Right, but did they support using any other init system (besides sysv) before?
  - bigon 9y ago
    That's actually not true at all... * https://packages.qa.debian.org/s/systemd.html * https://packages.qa.debian.org/u/upstart.html (it's gone from stretch that's true but still supported in jessie) * https://packages.qa.debian.org/o/openrc.html * https://packages.qa.debian.org/s/sysvinit.html
agwa 9y ago
It's really easy to replace systemd with sysvinit on Debian: https://news.ycombinator.com/item?id=14581820 https://news.ycombinator.com/item?id=14581820 All my Debian systems run sysvinit, and they work great - certainly better than the one system I co-administer which runs systemd. I frankly do not understand why Devuan exists.

badrabbit 9y ago

So, the whole thing with systemd is not that "it's bad" but that "I don't like it and I am stuck with it". Moreover, it leaves those of us who don't like it disenfranchised by distros that insist on using only systemd Opensource is about alternatives. Forks are a good thing,I wish devuan great success so people like myself won't have to whine about systemd all the time. If you like systemd and it's philosophy,I wish you the best of luck with it. Just keep in mind that others don't have to like it and different people or organizations have different needs.

telmich 9y ago
So true!
buu700 9y ago
My thoughts exactly. I personally have nothing against systemd and haven't been directly affected since I rarely need to touch init scripts, but I still appreciate Devuan and loosely follow its progress for this reason. (Kind of like ReactOS — 99% I'll never want or need it, but the fact that it exists is still really cool.)
- cjsuk 9y ago
  I was a systemd proponent until I had a hosed box. At that point I learned that alternatives are important. It made my life very difficult due to how opaque it is. Whike problems are surmountable, rehearsing for failure is advised.
binaryapparatus 9y ago
It is also plain bad but even more important is lock in that you mention. Bugs can be fixed but wrong philosophy can't.
- johnny22 9y ago
  We've been living with sysvinit style systems for years, and were able to transition away. We could easily do it again.

cosarara97 9y ago

You can run a traditional init in debian. Systemd is just the default.

telmich 9y ago
That might be true, but you cannot get rid of systemd in Debian
- duckerude 9y ago
  You can just uninstall it. It looks like apt remove systemd would work on my system, at the cost of GNOME and NetworkManager. APT automatically fills the gap with sysvinit and consolekit. OpenRC is also available. Debian even has ports to kFreeBSD and HURD that don't support systemd at all. There were some rough edges last time I ran Debian without systemd, and it might have gotten worse since then, but it surprises me how Debian ended up at the center of the debate when it doesn't seem that bad as systemd-defaulting distros go.
  - greenhouse_gas 9y ago
    >kFreeBSD It seems dead.

telmich 9y ago

Btw, I wanted to thank everybody for upvoting this post! For me it is not only a Sunday morning essay, but also important that people understand, why the Devuan movement is so important to all of us.

majewsky 9y ago

> We tried to build Data Center Light on Debian and Ubuntu, but servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration made it too expensive to run Debian or Ubuntu. Can you expand on that? I work at a cloud provider, and I never heard of such issues before. Our default images are Ubuntu 16.04, RHEL 7 and SLES 12, so systemd all around.

telmich 9y ago
Sure. With huge storage arrays, fsck might take a long time. systemd by defaults aborts after some time and will not mount that array.
- jamiesonbecker 9y ago
  So your file system check is killed and leaves your array in a possibly corrupt state? That is simply insane.
- majewsky 9y ago
  Did you file that in the systemd issue tracker? I had a quick look and couldn't see a report there: https://github.com/systemd/systemd/issues?utf8=%E2%9C%93&q=is%3Aissue+fsck+timeout https://github.com/systemd/systemd/issues?utf8=%E2%9C%93&q=i... This could be an honest mistake. On my system, systemd-fsck-root.service and systemd-fsck@.service have `TimeoutSec = 0`, which I guess to mean "no timeout", but the systemd-fsck executable could have its own timeout. In any case, filing a bugreport sounds much easier than switching distributions.
telmich 9y ago
Regarding not rebooting: sometimes systemd hangs and does not reboot in our setups, but we were never able to sufficiently debug this, as all terminals are already gone and it seems that systemds is waiting in a loop for dependencies that don't solve.
- moreentropy 9y ago
  Have you tried passing 'debug' instead of 'quiet splash' to the kernel cmdline and doing halt instead of poweroff/reboot? That way you should see what's going on during shutdown.

turbinerneiter 9y ago

Soooooo, I'm part of a CubeSat team and we use systemd. Extensively. We us their dbus library for our daemons to communicate, some of the ops is basically done in service files. I have to get our systems people to write a paper on it. My experiences with systemd are pretty much perpendicular to what some people claim.

telmich 9y ago
It's not about claiming here, it's about experiences. I guess we should open up a youtube channel for filming our most catastrophic experiences with systemd, because otherwise somebody will claim again that its brokeness is just a claim ...
- chris_wot 9y ago
  Would have been good if the author did this in the original article.
- lima 9y ago
  Let me tell you about my most catastrophic experiences with sysv... I've had quite a few. Anecdotal evidence is useless. What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd. I run large scale VM infrastructure on a systemd distro (RHEL, in particular) and I've yet to encounter a single issue that was caused by systemd or, for that matter, any issue with systemd that wasn't easily resolved.
  - fao_ 9y ago
    > What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd. That doesn't mean that there aren't any. Also, FUD (as I understand it) is people spreading unfounded rumours, not people documenting their own experiences and problems with a piece of software. The intent is incomparable. If you are actually interested in tangible structural flaws in systemd, look no further: https://suckless.org/sucks/systemd https://suckless.org/sucks/systemd https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awards/ https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awa... Also, see how Poettering handled this zero day (Which grants root to any user with a numeral): https://github.com/systemd/systemd/issues/6237 https://github.com/systemd/systemd/issues/6237 HINT: He doesn't think it's a bug... The fact that this even fucking exists in the first damn place: https://latesthackingnews.com/2017/06/29/a-systemd-vulnerability-allows-attackers-hack-linux-machines-via-malicious-dns-response/ https://latesthackingnews.com/2017/06/29/a-systemd-vulnerabi... See also: http://without-systemd.org/wiki/index.php/Arguments_against_systemd http://without-systemd.org/wiki/index.php/Arguments_against_...

jchw 9y ago

I don't like systemd, but it works pretty well nowadays in my experience and we use it for servers at work. Truth be told, I'm fine with monoliths. They just have to be good. The Linux kernel is a monolith. The real problem to me is how systemd makes the rest of the system more complex and integrated to systemd. Dbus is also not a great IPC layer, and yet it's the current state of the art. We need better APIs and foundations. Systemd doesn't provide those; it just lets everyone depend on systemd. Good for Redhat, bad for innovation.

raziel2p 9y ago

The fact that they weren't able to set up computers to boot using systemd is enough to warrant that I'd never want to buy or rent servers from this company, sorry to say.

telmich 9y ago
Question: did you every try to start a system with > 60TB storage attached with systemd?
- lima 9y ago
  Maybe he didn't, but I did, and it works fine.
- hpcjoe 9y ago
  I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical storage in a single box. Linked together via infiniband, 40/100 GbE, and others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS). Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well. The greatest pain was the networking. Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking. Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases. My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works. I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over. Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.

wicket 9y ago

> But first, let me put some warning out here: Dear Devuan friends, while I honor your work, I also have to be very honest with you: in theory, you should not have done this. Looking at creating Devuan, which means splitting of Debian, economically, you caused approximately infinite cost. I thought the same at first but after trying to maintain a Debian installation without systemd, I was convinced otherwise. The "infinite cost" was not caused by Devuan developers but by Debian developers. I felt the Debian transition to systemd was handled very badly and was rushed in order to get Jessie out of the door. Debian still provides the sysvinit-core package as if it were a supported init system however if you use it you'll find the operating system has many quirks and many things are broken. Before Jessie was released, I commented on one bug report related to a hard dependency on systemd for NetworkManager. Instead of fixing the regression, the package maintainer gave the excuse that they don't have the resources and would need more maintainers. I sympathise with the package maintainer but this is the sort of thing that led to this mess. I remember there being a lot of hostility in that period from Debian developers towards those who did not wish to use systemd. Based on my experience, I think the right decision was made to "fork" Debian to create Devuan. It's a shame that they had to do this. I still hold hope that maybe one day all of the work put into Devuan can be reintegrated into Debian and it can return to be the "Universal Operating System" that they still claim to be.

icebraining 9y ago
I'm pretty agnostic towards systemd, but I don't understand this complaint that Debian now doesn't support other init systems. Did they support other systems besides sysv before?
- wicket 9y ago
  They also supported upstart but that was dropped in Stretch. We are left with systemd and System V init to provide /sbin/init. Debian still provides System V init which itself works fine but other packages, many of which previously worked, have come to depend systemd only. This type of regression would not be acceptable in other projects but Debian has done nothing about it. I think they should stop pretending that Debian is still usable System V init and just drop the package altogether.

zbentley 9y ago

Every time I read something complaining about systemd from a centralization/reducing-choices POV, I mentally replace the word "systemd" with "glibc" and check to see if the argument still holds water. Mind you, there are plenty of other complaints (stability, back-compatibility, etc.) that are useful/valid regarding this software, but the above litmus serves to eliminate rather a lot of FUD and hand-wringing in my experience.

benchaney 9y ago
What criticisms have you eliminated using this strategy? glibc does not do the things that people hate about systemd.
- zbentley 9y ago
  It filters out: people who just want to complain about centralization; people who are annoyed about change and trying to rationalize it without trying or learning the new tool; and people who irrationally feel like they are no longer in control. Paradoxically, this improves the signal of discussions about systemd's flaws, and there are plenty. With the obviously irrational/haven't-used-it/axe-to-grind stuff filtered out, it's much easier to learn about real features, problems, fixes, benefits, and drawbacks of any project, systemd included.
greenhouse_gas 9y ago
>Every time I read something complaining about systemd from a centralization/reducing-choices POV, I mentally replace the word "systemd" with "glibc" and check to see if the argument still holds water. Actually, I replace systemd with *BSD. You know, the OS which has blessed core packages which are not replaceable?
- petre 9y ago
  Actually the BSDs with their non-replaceable core packages are a sane alternative to Linux w/ systemd.

vgy7ujm 9y ago

There is also Slackware!

Wildgoose 9y ago
And Gentoo.

gargravarr 9y ago

I thoroughly support Devuan. I switched my home servers to it around the systemd introduction and kept their complete stability, and now I've been able to influence my company to run it instead of pure Debian. You have to make a few allowances for packages that assume they're running on systemd Debian, but overall the system works exactly as intended. And as commented in the article, it's stable and easy to understand. At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux. In my opinion, the Debian developers were wrong to fully embrace systemd; it goes quite against the Debian principle of being able to adapt your machine to do anything. Yes, Red Hat held a lot of sway in convincing the community to adopt it, but Debian is a big distro as well - just look at the install base of Ubuntu. They could have held out and pushed back, but instead caved and now we're in the systemd mess. It intrudes so massively into userspace that it's impossible to get away from, and we wind up with the Windows model as this article points out - a massively complex spaghetti model of processes, all interlinked, all hinging on everything playing nicely. Just like Windows, it's a neat house of cards that could topple if any one of those crashed. This isn't Linux. I'm not defending sysvinit, there's no denying it's showing its age and harks back to a much simpler time, but systemd is not the answer.

lima 9y ago
> At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux. That's far from the only advantage systemd has. Even then, boot times are extremely important for modern VM and container infrastructures.

arca_vorago 9y ago

I still think systemd was a NSA/Redhat backdoor in the midnight for Linux. I've forced myself to deal with it because its everywhere, but my Spidey-senses tingle everytime I find some new way its trying to takeover command of a core function.

lima 9y ago

What irks me more about this blog post than the systemd part - which has been discussed in-depth countless times in the past, with valid arguments for and against - is the decision making and risk assessment process. I've run large-scale infrastructure on both Debian and RHEL, with and without systemd. I've been involved in many distro decisions in the past. The init system was never a major consideration, and I've always been able to work around whatever issues I've had with both sysv and systemd. Here's some considerations which were important for my team the last time we had to make a choice: - Vendor stability and long term support If you're running a business, you'll want long term stability guarantees - both from a technical, and a business point of view. Running a small community distro with few - if any - commercial users is a huge liability. Yes, you always have the option to fork it or take it over, but unless you're in the business of building a Linux distro, it's almost certain to be a bad business decision since your competitors are spending their time on their products instead. Anyone who doesn't appreciate long term vendor stability hasn't yet been burned by the lack of it. Your Gentoo wizard left the company? Too bad, good luck maintaining your infra now (I've actually seen that exact scenario play out not once, but twice!). A few core maintainers left for another project and you're left building your own packages? D'oh. Having a mature and well-funded organization or a large community of commercial users (in the case of Debian) supporting your distro is extremely valuable, even if you aren't the ones paying for it. - Ecosystem The ecosystem is also really important. It's often the main motivation for using a particular distro. Development tools, third party package repos, 3rd party enterprise software, troubleshooting resources, documentation and much more depend on a healthy ecosystem. With Devuan, this isn't as much of an issue, but it's still sufficiently different from Debian in sometimes subtle ways that it will nullify some of the advantages of being in a common ecosystem. This also includes hiring - you'll have a harder time staffing your company if you're using exotic stuff and you'll unnecessarily spend your time training them. Ever wondered why companies like Google publish so many papers, talk about their infrastructure and even publish books about it[1]? It's because it means they can - by advancing the state of the industry - hire people who are already familiar with their architecture and concepts. This is a real problem for them - Google, for instance, is in some cases so far ahead of others that they have to spend a significant amount of resources to bring their new hires up to speed. I don't know about Amazon and Facebook, but I'm sure they have similar problems. [1]: https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html - Security Security follows the supply chain. Your security is only as strong as your distro's - no matter how good your security controls and processes are, if your distribution vendor is compromised, you won't stand a chance unless you have a world-class security team. Likewise, your customers fully rely on you - if you're compromised, they will be, too. Security is much more than signing your packages and publishing security advisories. In fact, those are merely the results of a properly implemented security management framework - risk assessment and mitigation needs to be an integral part of your vendor's (and your) processes. Where do they keep their PGP keys? Who has access to it? Is is stored on a HSM or just sitting on a server somewhere? Is there a process for revoking it? Is there a change review process? Does the build system verify source code integrity? Is there two factor authentication for production? Do the team members keep their SSH keys on a smart card? This list could go on for miles. - Quality Assurance Even with the main distros, QA quality varies wildly. In my personal experience, Red Hat has by far the best QA, followed by Debian and then (with some distance) Ubuntu. I've had particularly bad experiences with Ubuntu, a sentiment shared by many operations people I've talked to (the worst one being the grub timeout bug that made me spend a few days pressing the return key a lot - basically, they removed the default timeout in a minor update and all my machines sat there waiting for someone to manually continue the boot process). Attention to details, even if it's minor and seemingly insignificant, really helps productivity if you sum it up at the end of the day - each bug your vendor catches and each quirk they document is time you won't have to spend with debugging. With smaller distros, this is even worse - in some cases, they don't even have proper QA. Now, let's assume that the issues they had with systemd were as significant as they say they are (I'm doubting it, but unfortunately, they didn't talk about any particular ones), it still might very well have been a better business decision to stick with Debian and spend their time fixing the issues they had instead of migrating to Devuan, which is a HUGE liability compared to plain Debian.

crististm 9y ago

What makes the door responsible for the failures of engine ignition? This is a good analogy to remind Devuan why they rejected systemd in the first place.

bitwize 9y ago

Oh, suck it up. From a business perspective, the most important thing -- indeed, the only thing that truly matters -- isn't the software itself. It's support. When it comes to open source, you either use a supported configuration or you assume ALL the responsibility for supporting your systems yourself. Who would you rather handle the support on the off chance your machine goes tits up? You? Or Red Hat? No wait, scratch that. Who would YOUR BOSS rather handle the support? The supported init system for most flavors of Linux (that aren't in LTS) is systemd. Systemd makes the most sense from a business perspective, and "but muh Unix philosophy" doesn't hold when there's money on the line. May as well get used to it.

gerbilly 9y ago
>From a business perspective That's just it, some of us aren't interested in the 'business perspective' and have other uses for our systems.