24 ms·

Systemd: The Good Parts

364 points by pcr910303 5y ago

ape4 5y ago
1. You don't have to clone then modify a 100 line bash script for a normal daemon's /etc/rc.d/init.d script.
- fiddlerwoaroof 5y ago
  Instead you have to write a couple-hundred-line bash script and then modify an under-documented ini-file to reference it
  - merb 5y ago
    you never used systemd?
    - tgv 5y ago
      There's a bit of truth in that. The file format is underdocumented. You've got to look everywhere to find snippets of information, and then there are these weird incompatibilities between versions (Service.Type bit me once, and either After or Wants simply didn't do what the scant information suggested). And it often starts a script or a dedicated tool, e.g. for backing up, or watching directories. However, it's easier to deploy and keep track of your own services, sudo and chdir are built in, there's a watch dog, easy journaling. That's nothing to sneeze at.
      - fiddlerwoaroof 5y ago
        I’ve written a handful of systemd units now and, while the simple cases are nice, anything more complicated is really hard to figure out by reading manpages or other official sources of documentation. And, additionally, I’ve found it’s often easier to write a shell script to be invoked by systemd than to handle certain issues in the unit file itself. As far as the watchdog but things go there’s plenty of good enough solutions that don’t really on systemd: daemonize, DJB’s daemontools, etc.
        thayne 5y ago
        I've written many systemd services, and other units. Some of which were quite complicated. I found the man pages extremely useful. Moreso than the manpages for sysvinit or upstart, when I used those systems.
        kaba0 5y ago
        > I’ve found it’s often easier to write a shell script to be invoked by systemd than to handle certain issues in the unit file itself. But that’s a completely fine solution, service files are not meant for declaring arbitrary logic.
- jude- 5y ago
  The OpenBSD init system and OpenRC init system haven't had this problem for years -- even before systemd became a thing.
  - CameronNemo 5y ago
    Similarly runit, s6, and Upstart have never had that issue. It is the oddest complaint to come up with when considering life without systemd.
    - jraph 5y ago
      Upstart was good and probably paved the way to systemd.
      - agumonkey 5y ago
        I wonder if systemd is influencing a new init right now
      - cbmuser 5y ago
        No, Upstart’s design was so bad that even Scott James Remnant (the creator of Upstart) admitted it’s design was inherently flawed. The biggest problem with Upstart was that it put the dependency chain upside down and just started a service whenever it’s dependencies were fulfilled, no matter whether it actually made sense to start the service.
        ahartmetz 5y ago
        I remember reading about it and thought it seemed wrong, but presumably the author did his research and knew what he was doing. Oh well. His attitude and the readability of Upstart code were very good, though. The Upstart design was probably easier to implement. With the systemd design, the dependency tree needs to be traversed bottom up (figure out what needs to be started to start the targeted set of services), then top down (start dependees when dependencies are ready). With Upstart, top down was enough. Or so it seems.
        5y ago
- deleted 5y ago
  [deleted]
- iso1210 5y ago
  I have 90 scripts in my /etc/init.d directory. 51 are less than 100 lines, the most recent one I created is actually a perl script which is 29 lines long.
  - fiddlerwoaroof 5y ago
    Yeah, with the #!-lines, sysv init scripts can be written in just about anything and, as long as you implement the expected command line interface, the init system will handle it correctly.
    - MarkSweep 5y ago
      Hmmm, I wonder if anyone has made program for this purpose that will read the rest of the file a declarative definition of a service? Like imagine a SYSTEMd unit file with a shebang at the top .
- Arch-TK 5y ago
  I've not seen even sysvinit based systems with this problem. Every service manager I know of solves this problem. This is as much an argument in favour of systemd as "don't have to clean up horse excrement" is an argument in favour of owning a fiat 500.
  - viraptor 5y ago
    Most init scripts are in that ballpark. See openssh for example (88 lines) https://github.com/openssh/openssh-portable/blob/master/opensshd.init.in https://github.com/openssh/openssh-portable/blob/master/open... And yet, it still tries to use pidfiles (which systemd has a better alternative for), and notes "We really should check if the service is already going" which they don't do because it's an actual non-trivial issue if you're writing a standalone init script.
    - Arch-TK 5y ago
      Does anyone use these init scripts? Every popular sysvinit based init system uses short declarative service scripts for most services. Every non-sysvinit init system seems to mostly manage with 2 line scripts. And yes pidfiles are crap and I don't like them either (a genuine criticism of sysvinit and advantage of systemd) but systemd isn't the first init system to do away with pidfiles so it's hardly a unique strength.
      - viraptor 5y ago
        Could you show an example of a declarative sysv-init script? To be honest, I'm not aware of it being possible and ddg doesn't bring up anything. > Does anyone use these init scripts? By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them)
        Arch-TK 5y ago
        This is OpenBSD's sshd init script: https://github.com/openbsd/src/blob/master/etc/rc.d/sshd https://github.com/openbsd/src/blob/master/etc/rc.d/sshd On an archlinux system running systemd I see that the equivalent of the rc_reload function is "ExecReload=/bin/kill -HUP $MAINPID" (which is actually less safe...) I'm not sure if you think that's declarative but to me it's basically as declarative as you can make a shell script. > By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them) I mean is anyone CURRENTLY using them? Because as far as I'm aware the linux sysvinit situation was a gigantic mess which nobody wanted to bother fixing despite the fact that OpenBSD and other BSDs had already fixed it by that point. I think most linux distros can be taken as an example of how not to do things if you want to make your life easier. Nowadays BSDs don't use these awful scripts. OpenRC based systems don't use these awful scripts. Who is using them?
A4ET8a8uTh0 5y ago
"I hope Alpine ends up with something similar!" Direct quote from the website. Eh, no. There is already a short list of viable non-systemd candidates. We.. well.. I don't need less. It already feels like I am transgressing some law by typing this on PopOS.
- darthrupert 5y ago
  To each their own, but I won't touch Alpine (outside of docker use) until they get systemd or something fairly similar.
  - lixtra 5y ago
    The same design goals that make them a good docker distribution may have made them to stay away from systemd. Meanwhile others claim that systemd makes docker redundant anyway[1]. [1] https://news.ycombinator.com/item?id=26473581 https://news.ycombinator.com/item?id=26473581
  - efiecho 5y ago
    > To each their own, but I won't touch Alpine (outside of docker use) until they get systemd.... Luckily that won't happen as systemd doesn't support musl which Alpine uses for libc.
    - 2pEXgD0fZ5cF 5y ago
      I'm very glad for that too.
- rzzzt 5y ago
  There is some work going on around replacing OpenRC in Alpine, this might be the thing the author is alluding to: https://skarnet.com/projects/service-manager.html https://skarnet.com/projects/service-manager.html
  - A4ET8a8uTh0 5y ago
    Thank you for sharing this. I was not aware of it.
memco 5y ago
> If you need to reboot the server right now for some reason, will that service come back up on reboot? I didn’t see this discussed more after it was asked, but I could use some pointers on how to answer this question since I have a service I wrote that I think should start on reboot, but has to be started manually. It would be nice to know when I have fixed it without having to reboot after each change to just to test. Anyone know how I can do that?
- rav 5y ago
  In the systemctl status output, it says "Loaded: loaded (...; enabled; ...)" - "enabled" means that the service starts on boot. Use "systemctl enable" to "enable" a service. It creates a symlink in /etc/systemd/system/multi-user.target.wants assuming the service's unit file says to enable it through the "multi-user" target. By listing that directory you can see what services load on boot.
  - CameronNemo 5y ago
    Not exactly. Socket units can pull in services that are not otherwise enabled. You also have to search through all of your udev rules to check if there are SYSTEMD_WANTS directives pulling services in.
    - Arnavion 5y ago
      rav (correctly) said that enabled services are started automatically on boot. They didn't say that non-enabled services are not started automatically on boot.
      - nemetroid 5y ago
        They also said: > By listing that directory you can see what services load on boot.
        Arnavion 5y ago
        Yes, and I didn't read that sentence as disallowing other ways to autostart services either.
ratsmack 5y ago
>systemd is a set of building blocks that you can use to make a linux system. A more accurate description would be "Systemd is a set of building blocks that you can use to make Linux more like Windows".
- dale_glass 5y ago
  systemd is incredibly Linux centric. In fact it intentionally doesn't even try to be remotely portable, because an explicit design aim was to expose all the cool stuff that Linux has, but wasn't getting enough use.
  - trasz 5y ago
    Sure, but design-wise it follows Windows principles: it’s tightly coupled, and not particularly tweakable.
    - deleted 5y ago
      [deleted]
    - dale_glass 5y ago
      These days I see it as a bonus. I don't want to mess around with the boot process. I don't want to write some sort of glue layer in between loosely coupled parts. And I certainly don't want somebody else to do it either, because that too often tends to result in some kind of surprise that one has to deal with at 3 AM. What I want is a solid system that does what it's supposed to -- start stuff -- in a maximally boring fashion.
      - trasz 5y ago
        Sure, I understand that. But this way your getting another Windows, except with source code.
        dale_glass 5y ago
        I'm not seeing the problem. I use computers to get stuff done, not because of some sort of religious ideas about how things out to work. While I prefer Linux in general, Windows does have some good ideas in some places, and I don't see any issue with using them.
Decabytes 5y ago
I just learned about Vtubers this week. I would have never thought this would have crossed into the Linux world but here we are!
mwvr 5y ago
A furry named Xe promoting systemd, just another day in the Linux twilight zone.
AshamedCaptain 5y ago
I have a system where systemd decides, about half the times I boot it, to immediately unmount all of the filesystems listed in fstab literally right after having finished mounted them as part of the regular boot process. All of this with other daemons starting in parallel. It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login. No idea how to even start debugging it.
- jraph 5y ago
  I would run systemd-analyze critical-chain and then systemd-analyze blame to see if there is a service that could do this. And then I would look at the detailed boot logs. If nothing can be found, I'd look for some way to increase the log level. If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.
  - AshamedCaptain 5y ago
    The logs are mostly useless and random. There is nothing that consistently appears between the point where it finishes mounting (lots of "Mouting...", then "Reached target Local File Systems") and where it starts unmounting ("Stopped target Local File Systems", then lots of "Unmouting ..."). Sometimes these two lines are less than one second apart. But I haven't found any message which is consistently between these two points that would point to a suspect. And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already). It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...
    - jraph 5y ago
      Most certainly you can only analyze things on boots where the problem appears, unless logs are persisted across boots. Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run. If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.
bfrog 5y ago
Yeah it's all pretty darn good. People forget the cluster of bash scripts, various cronz, inetd, log, and other supervisor type daemons all to do what systemd does better with a simpler config. Now ask me about pulseaudio and there's a project with nothing but issues, looking forward to the new pipewire overlords.
- tediousdemise 5y ago
  > Now ask me about pulseaudio and there's a project with nothing but issues Don’t forget Bluetooth, WiFi, and graphics (cough Nvidia). Or any peripheral, for that matter. Until vendors stop forcing us to play games with their broken blobs, Linux will forever be just a grand experiment for frustrated hobbyists and unfortunate professionals around the globe. A digital tragedy of the commons.
  - techrat 5y ago
    >Linux will forever be just a grand experiment for frustrated hobbyists and unfortunate professionals around the globe. Linux is the most widely used kernel in the entire world. If you consider 'Linux' to be the OS, it is #1. If you consider all distributions that use the Linux kernel, it is also #1, collectively and individually (Android.) If you look at the ranked list of Supercomputers, Linux has the entire top 500. What the hell are you talking about?
    - OhSoHumble 5y ago
      Probably just referring to desktop usage.
    - kllrnohj 5y ago
      > and individually (Android.) Which, worth noting, does its own thing for init & services ( https://cs.android.com/android/platform/superproject/+/master:system/core/init/ https://cs.android.com/android/platform/superproject/+/maste... ). I think it's fair to assume they were talking about Linux as a desktop OS, which for some reason absolutely refuses to learn lessons from the other 99.9% of the market. And is also infatuated with internal fragmentation despite its tiny usage. That makes it really fun as a hobby for sure, and powerfully flexible for datacenter usages (from the mundane to the supercomputers). But also pretty terrible as a "I just want my laptop to work reliably & well" OS.
jph 5y ago
> By not having something competitive Alpine is less and less attractive for newer production deployments. Alpine uses OpenRC (https://wiki.gentoo.org/wiki/OpenRC https://wiki.gentoo.org/wiki/OpenRC). IMHO Alpine and OpenRC tend to make production deployments more predictable and reliable, because of deliberately simple init ordering, as well as a smaller surface area as compared to systemd.
- detaro 5y ago
  There is some work on a replacement that does more while fitting the requirements and philosophy of alpine: https://skarnet.com/projects/service-manager.html https://skarnet.com/projects/service-manager.html
  - mendelmaleh 5y ago
    Good read, thank you!
  - GekkePrutser 5y ago
    Interesting, I have a lot more faith in the alpine team implementing something like this than in RedHat. Even when RedHat made consumer OSes (before spinning them off to Fedora) I didn't like them. What they're doing looks like a great alternative. I also hope they will not have any reliance on dbus as systemd does. Reliance on dbus is one of the main issues I have with systemd (besides its size and complexity).
    - zxzax 5y ago
      I've seen a lot of these comments to this effect but I'm still not quite sure: what is the issue with dbus? Is there another replacement that meets all the use cases?
      - detaro 5y ago
        Question is if an init system needs all the use cases dbus can support. Dbus always feels unwieldy to work with, especially once you leave specific desktop stacks, and for me personally, the biggest failures of systemd-based systems have been issues with dbus not working properly for whatever reason.
xena 5y ago
Author of the talk here. If you have questions for me please feel free to reply to this comment with them.
- jen20 5y ago
  I did a talk at HashiConf EU a few years ago with the same title - interesting to see where our ideas of "the good parts" overlap!
  - xena 5y ago
    Wow, I think that was completely by coincidence. I've never seen your talk before. Where did we overlap?
    - jen20 5y ago
      I think we definitely agree on the most of the benefits - there's a transcript at [1]. I'm happy to see more people discuss the benefits and practical uses rather than the sea of hate that systemd usually gets (especially in these parts). [1]: https://www.hashicorp.com/resources/systemd-the-good-parts https://www.hashicorp.com/resources/systemd-the-good-parts
- Hackbraten 5y ago
  No question, just thank you for an excellent article. I’ve found it difficult to come across articles that focus on the good parts of systemd.
- Traster 5y ago
  Hey, thanks for the great talk, I often type something like "journalctl -xfe" and just get completely lost, you mentioned that's out of scope for this talk but do you have a recommendation for some hints about how to use journalctl effectively?
  - dale_glass 5y ago
    A quick guide: * Want logs since boot? journalctl -b. The first boot is -b 0, the previous boot is -b -1. * Want logs for smartd? journalctl -u smartd * Want warnings and above? journalctl -p warning * Want logs since yesterday? journalctl -S yesterday * Want logs on April 22 between 10 and 11? journalctl -S "2021-04-22 10:00" -U "2021-04-22 11:00" * Want logs with microsecond timestamps? journalctl -o short-precise * Or in JSON for parsing? journalctl -o json * Want tail -f? journalctl -f
fghorow 5y ago
"This page intentionally left blank." (Yeah, I'm a dinosaur who remembers dead tree IBM manuals. Sue me.)
exabrial 5y ago
Things that are terrible: logging. Why did we have to screw this up? Just writing stuff to a file was fine. Things that are good: simple syntax, easy to use utils Hidden gems: cgroups, file systems containment, etc. Systemd could completely eliminate the need for stuff like docker by totally isolating a process.
- porsager 5y ago
  That's interesting. Do you have a link to more info about isolating a process like that?
  - comboy 5y ago
    Arch wiki is usually a good place to learn something in a succinct way (I've never even used Arch): https://wiki.archlinux.org/title/cgroups https://wiki.archlinux.org/title/cgroups
- moshmosh 5y ago
  The nice thing about docker is that you can use the same tooling on lots of operating systems, not just that it wraps some Linux functionality. Sure, it may involve using a VM, depending on the system, but I still get to just run "docker" and "docker-compose" and such, as if it were native. A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. Really, though, if I care about isolation and don't want to use docker, I'm gonna reach for FreeBSD + jails anyway, and I don't think systemd's gonna change that for me any time soon. If I want containers with better tooling outside the target OS, I'll use docker. Systemd falls in an awkward middle that I don't have much interest in.
  - vially 5y ago
    > A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. But that's exactly what happens with docker anyway (e.g.: Docker is still running in the Linux VM when running it from a different operating system). And considering the widespread systemd adoption throughout the Linux ecosystem, it's far more likely for systemd to be already installed in the VM than docker anyway.
ohazi 5y ago
I actually like using systemd. I took the time to learn how to use it, and as a desktop Linux user, it's honestly quite nice. But the one thing that still really pisses me off about systemd-the-project was the fact that they ate udev-the-project. In my view, that decision was unnecessary and was done for purely anti-competitive reasons. If you're not familiar, udev is basically the user-facing device manager for Linux. It's what allows you to easily configure rules and permissions for all the USB devices that you plug into your computer. These devices still need kernel drivers, but udev is how you tell your system "Please let user ohazi use this device, and also give it a convenient name like /dev/someDevice0" By devouring the udev project, the systemd maintainers have guaranteed that dealing with USB devices on non-systemd systems was going to be a giant pain. Then came the forks -- Gentoo maintains eudev, which is a systemd-free fork of udev. But really, this shouldn't be a fork. Udev should be independent and available on all systems. If systemd wants to do something special with devices, they should use udev APIs like everybody else. In my view, this is what finally allowed systemd to win the init war, despite all the protest. Ideological arguments about how best to start daemons is one thing, but you simply can't use a modern system without a sane approach to USB. Edit: It seems that some of my concerns were overblown, e.g. you apparently can run udev without systemd as pid 1. Udev has a build-time dependency on systemd, but not a run-time dependency. The eudev fork removes that build-time dependency, and (maybe?) also papers over some other inconsistencies. I'm pleased that this is the case. I'm annoyed that the discourse around systemd/udev has been muddy enough to lead me to incorrect conclusions (and I'm aware that the original version of this comment likely added fuel to this particular fire). Oh well... live and learn.
- edoceo 5y ago
  I love Gentoo and really like they chose to make eudev and a elogind so I can continue to use OpenRC, which has been working great for me since forever.
  - ohazi 5y ago
    Don't get me wrong -- I too am extremely grateful to the Gentoo devs for maintaining eudev. It's also the obvious choice for highly memory constrained embedded Linux systems that can't fit systemd. I just think it's absolutely bonkers that it has to be a fork... eudev should really just be udev.
dijit 5y ago
Copying my reply from Lobste.rs because there’s more people here who can pick me apart: _FD: I’m not a large supporter of SystemD after using it when it was first introduced in Fedora and watching it take over the ecosystem, I do use it and as a sysadmin/SRE type I’m heavily versed in it and how it functions, so, everything with a pinch of salt._ I gave the video a good shot, half the time is spent going over some systemd-ecosystem basics; The author then makes the case that Alpine is a less attractive development/production-deployment platform due to lack of this ecosystem (though they make effort to avoid directly saying that systemd would be a good solution itself.) I couldn’t disagree more honestly. I love the concept of Alpine: as small as possible and no smaller. The attention to minimise dependencies makes it the distro of choice in containers; I’ve even considered (strongly) running it on some machines where systemd would have been a lot of needless complexity. I think deviating from this path for alpine is a grave error, there will always be a place for a niche system that is small; especially when it’s commonly paired with some other log managment/service orchestrator like Alpine is. Additionally, it gives people like me an option to run without init systems that could be considered heavy and intrusive with heavy dependency chains. So, kindly, Alpine: please don’t.
ChuckMcM 5y ago
Okay, I'll admit I was hoping for a blank web page :-). Not a fan. I always ask ""What problem is this trying to solve?" and then "Is this a good solution for that?" I appreciate the "startup dependency problem" and when I was in the Systems Group at Sun we worked with AT&T on their solution which was the whole "run level" thing. And I appreciate the whole "shell scripts can do anything" problem that just using a naming scheme on shell scripts to run can lead to unexpected behaviors. From an architectural taste perspective, I prefer systems which are more comprehensible in parts and thus don't try to abstract out semantics to the point of incomprehension. So at the end of the day, something with a better model, better visibility into its operation and configuration, with a way to validate and debug dependencies would be good. But I haven't been impressed with systemd.
- dale_glass 5y ago
  It solves a whole lot of problems to me. * It gets rid of a system cobbled together from init, inittab (which people rarely remember exists), monit, cron, and inetd/xinetd. * It removes a bunch of horrible hacks in init scripts such as calls to sleep. * It removes the need for pid files. * It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts. * It allows precisely tracking what belongs to each service. I can easily find out what a random process belongs to. * It captures all the log output of each process * It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again * As a developer it means I don't need to write a slightly different script for every distro * It makes logging a whole lot pleasant * It allows having user services without having to hack around it with cron * It allows a whole bunch of settings to be applied to any random service, without having to edit a shell script and figure out how to stick it in there. * It means services run identically when started by hand, without the confusion of a service inheriting my local environment. * It makes my VM servers boot faster. And if you're going to pipe up with "I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand, when they're needed. They adapt to the load.
  - ChuckMcM 5y ago
    I had to chime in here and say I really like this post, thank you for taking the time to write it out. I would give it extra up votes if I could. It is a nicely concise summary of the problems. I'd disagree with the characterizations of the previous system as "cobbled" or "horrible hacks" but there are solid things that need to be solved well by any system initialization architecture/service.
viseztrance 5y ago
I ignored systemd because I thought it didn't affect me, as I only run linux (fedora) on my desktop, and hardly had any issues at that. But recently, when I changed my fstab file to mount a drive at boot, I made a typo. My system wouldn't boot at all and it wouldn't even drop to a shell so I can chroot the file system like you would with init.d. I thought that there's some arcane command I don't know about. But there wasn't. You're locked off, and expected to boot off something else to fix your system. Horrible. Having this said, I honestly expected way more on the section about benefits to users.
- zaarn 5y ago
  Atleast on Arch, Systemd should drop you into the Emergency shell if it cannot mount a device that isn't defined as optional. IIRC it might require a root password to be set, however. Otherwise, edit the kernel commandline to add "systemd.unit=emergency.target" to the end, which triggers systemd to straight boot into this console without trying anything that it doesn't need to bring up the console.
  - otterlicious 5y ago
    Or if you're blocked by the root password prompt add `init=/bin/bash` and make your filesystem writable if necessary with `mount -oremount,rw /`
    - zaarn 5y ago
      This will block systemd however. The nice thing about the emergency shell is that you can use systemd things like trying to mount filesystems. And you can later isolate multi-user to continue the boot when you finished your fix. /bin/bash would be atleast useful to try to get a root password set.
  - viseztrance 5y ago
    It tried when the mounting failed, saying it would drop me to a shell and then said "it fail to open up a shell".
efiecho 5y ago
Poetteringware seems to be highly praised by mostly young people who would like Linux to be a Windows clone. People who understands Unix concepts tends to dislike it.
- StreamBright 5y ago
  Well yes and no. I like the good parts of SystemD (not the parts this video is talking about). For me there are 3 things that SystemD gets right: - no shell scripts for service configurations - no forking services (no need for daemon mode anymore) - logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great) And the rest is just pure madness. - DNS (but whyt?) - NTP (but why????) - system limits (seriously, da f???) - pid 1 should be a very simple service (More: https://ewontfix.com/14/ https://ewontfix.com/14/) I have spent 25 years on Unix, so I guess I understand the concepts.
  - Arch-TK 5y ago
    > - no shell scripts for service configurations Other than "sysvinit based init had horrific shell scripts with 100 lines of repeated nonsense", given that almost every alternative to systemd which still uses shell scripts needs around 2 lines of shell per service (including the shebang) what exactly is the issue with shell scripts (if all they do is run a wrapper which sets up the environment of a program or the program itself)? > - no forking services When I used archlinux these were extremely common and I used archlinux as recently as last year. > - logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great) So runit handles spawning a log daemon before the service is started to ensure all log entries are kept and comes with a really simple yet extremely powerful log daemon (but obviously you can use almost anything else including cat if you want). It manages to integrate logging tightly into service supervision while being maximally flexible and decoupled. What do you think about this?
    - 1vuio0pswjnm7 5y ago
      I use daemontools instead of runit. What am I missing. multilog is the simplest (smallest) and most versatile log rotating daemon I have found.
deleted 5y ago
[deleted]
cerved 5y ago
I can't weigh in on the pros/cons of systemd because that's out of my depth. But I am somewhat fascinated by the hate it seems to endure. Like this https://suckless.org/sucks/systemd/ https://suckless.org/sucks/systemd/ I guess it's a bit of a watershed
- Arch-TK 5y ago
  For something which the authors seem intent on forcing down everyone's throat whether they want it or not (genuinely you can find quotes of Lennart claiming he wants to make it more difficult to not use systemd or that he wants to "gently push" people towards it) it's not really surprising. If their only goal was to gain adoption based on the merits of their software while not intentionally making themselves intrusive nobody would really care.
  - bkor 5y ago
    > forcing down everyone's throat whether they want it or not You're being highly offensive towards these developers for no good reason. All kinds of grand claims, just because things are done on a way that you don't agree with. You've stated that systemd developers are wanting to force people to eat things. And their action is to actually force it into someone's mouth, likely requiring the person to be put into restraints to make this possible. You're being highly offensive. Systemd has existed for loads of years, yet this accusation continuous to be repeated. Aren't you aware of German libel laws? If I'd accuse you of such behaviour, wouldn't you get upset? Why say such things, things that suggest criminal behaviour, so easily? These things are repeated so often that it isn't even moderated upon. It's odd that this behaviour is tolerated.
    - cerved 5y ago
      Forcing something down everyone's throat is an hyperbole. They are not slandering systemd developers, nobody's suggesting people have to eat systemd. This is not offensive and it's not a crime, it's an opinion
yuuta 5y ago
I'm using split horizon DNS with WireGuard on my systems (I mean using special DNS servers for some domain names and the public ones for the rest. Correct me if I'm wrong). I used to try systemd-networkd and systemd-resolved because they seem to fit together. I found systemd-resolved is extremely buggy with systemd-networkd managed WireGuard. My WireGuard endpoints use domain names and systemd-networkd fails to resolve them after a manual restart. I couldn't figure out why. I also could not find a way to NOT combine split horizon DNS with network interfaces (like most documents online only include scenarios like resolving corporate domain names using a VPN interface but that wasn't my case). I have multiple VPN connections to the same network (runs BGP basically) and I did not want the split DNS to bind to some specified interfaces. Moreover, systemd-resolved seems not able to do split DNS without systemd-networkd. It needs systemd-networkd managed interfaces to do that. What? I just want separate DNS servers for separate domains. My WireGuard interfaces are managed by wg-quick. Correct me if it is possible to use systemd-resolved standalone in this case. The hard systemd-resolved dependency of systemd-networkd also had issues. Sometimes, I just want systemd-networkd to configure resolv.conf using DHCP DNS but it looks like I HAD TO START systemd-resolved AND LINK /run/systemd/stub-resolv.conf to /etc/resolv.conf in order to achieve that (correct me if there's a way to not use resolved completely). Excuse me? systemd-networkd itself was buggy, too. I once encountered https://github.com/systemd/systemd/issues/18108 https://github.com/systemd/systemd/issues/18108 and my friends reported they faced this issue as well. I don't mean that software should not have bugs but encounter them is really inconvenient. In conclusion, I mean, systemd components seem to fit together and help you do a lot of things, but they did't do the things well, compared to separate programs for their specified tasks. Moreover, you have to deal with various components of systemd and figure out their relationships in order to achieve some specified tasks. For split DNS, I'm using dnsmasq now and it's working pretty well.
kbenson 5y ago
I think systemd brought a lot of good stuff to the table, but I also think it's a sprawling mess with poorly named and documented (as in there are behaviors implied by statements in different config directive sections that make it hard to reason about how stuff works). I'm looking forward to the successor of systemd, or at a minimum, them rolling out config syntax 2.0 which breaks compatibility in an effort to make stuff more sanely named and documented. I know that would cause huge churn. I know it will never happen. But I can dream, because I don't think anyone can look at how systemd configs define dependencies and say that they both understand the nuances and that those directives are are named sanely in a way that normal people can understand, and that's indicative of a lot of stuff with systemd. Even though it was probably designed by some of the most knowledgeable people WRT init systems, it grew organically as they discovered new things they needed, and it needed a lot (which I think just goes to show how little was really understood by anyone about how to tightly couple these things).
deleted 5y ago
[deleted]
GekkePrutser 5y ago
I don't agree with her sentiment that alpine should adopt something similar. The whole reason I use alpine is because it doesn't use systemd. I like alpine because it uses a simpler, lighter approach. It's not for everyone, and I'm pretty sure alpine maintainers don't intend it to be for everyone. It serves its niche really well as it is. For those who like systemd there's many distro choices already. Edit: As detaro mentioned in another comment ( https://news.ycombinator.com/item?id=27176391 https://news.ycombinator.com/item?id=27176391 ) they are actually working on something, but they aim to solve the various issues around alpine that don't fit well with alpine's philosophy. It sounds like a good story and I'm open to this idea. The main reasons I don't like systemd are its complexity and its dbus reliance, basically bringing too many desktop paradigms into all Linux systems, even servers and containers. So about my above point: I'm open to it adopting a service manager but not too similar to systemd :)
- deleted 5y ago
  [deleted]
theteapot 5y ago
Cool .---------. .---------. | SystemD |------>| Service | '---------' '---------' Not Cool .---------. .---------. | SystemD |<------| Service | '---------' '---------'
- dale_glass 5y ago
  I don't get it?
  - vgel 5y ago
    Using SystemD to manage your services easier if that's what you want is nice (aka, if your SystemD setup depends on a service, that's fine). It's not great if the service depends on SystemD, because then it can't be used with other init systems (without patching / forking, at least).
tlamponi 5y ago
The declarative format of systemd unit files is just much easier to maintain, grasp and allows doing things that just cannot be done else, for example, a generic way to parallel starting service at boot, as all dependencies and before/after ordering is known. And the great things is, that declarative format means that systemd can actually be replaced by something else, as one just needs to be able to parse the ini-style configs and execute the ExecStart and ExecPreStart in order. One really cannot do that to the other init-systems, AFAICT, they are far too scripted and rely a lot on side effects to do anything like that, besides packager choosing different solutions for the same problem. Some KI-paper could possibly take a shot at parsing out some sensible information reusable for another tool ;-) I can get the dissentient about systemd incorporating so much stuff, but just from POV of it as init-system it is really nicer to work with, speaking as user and as someone which job includes packaging software for a Debian based distro.
- froh 5y ago
  in system V init scripts on Linux they had standardized dependency declaration comments. when you added services you ran a dependency checker which would translate the dag into a total ordering, laid out using 01 02 03 ... numeric file name (symlink) prefixes. it was robust, simple and straightforward. however it was built around bash. which was large and slow to start up,compared to e.g. dash as scripting shell. and dash was not widely around yet. so each script had the cost of bash startup, the milliseconds of which added up. and at some point the tipping point was reached.
  - tlamponi 5y ago
    > in system V init scripts on Linux they had standardized dependency declaration comments. when you added services you ran a dependency checker which would translate the dag into a total ordering, laid out using 01 02 03 ... numeric file name (symlink) prefixes. Yeah, I know, but in practice this did not really work out and was a mess. In fact, every update was scary to do, in that time I frequently just re-installed the distro on my private workstation, simpler than to deal with the rc stuff. A plain setup may have worked, but anything slightly complex was just bonkers to do. > it was robust, simple and straightforward. Do you package for a relevant distro or is this just your experience as user? Asking because I just personally do not know any maintainers of software with a slightly complexer system interaction which would agree with that. There surely are some, but FWICT on some ML's anti/pro systemd flames it's <1% of maintainers - and from them it's actually almost never the new unit file format what is the issue, but the way systemd eats up basic system services. IMO the systemd unit files are exactly what you say "robust, simple and straightforward", the implementation may not be, I never even touched that, but the unit file schema is just better, as it's not bash/dash with some slight optional possibility to use some hacks like comments (urgh) as schema, but actually provides a standardized format for doing that and more. As someone maintaining dozens of packages and as common user, I'm happy that I do not need to mess around and fight the init-system, and I'm not the only one thinking so[0]. [0]: https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_archlinux_embrace_systemd/d3rhxlc/ https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_a... Thanks for the constructive downvote though ;-)
arid_eden 5y ago
The post was on 'the good parts' of systemd, so complaining about balance seems unfair. I will call out one aspect that the author deemed a 'good part' as in my experience its a bad part: journald. Usability: journald requires administrators to learn a new command to access their journals - everytime I have to use it I need to look up the syntax again. Compare with logs in /var/log and you can use all the standard unix tooling (text editors, tail, grep etc). Reliability: in my use journald has made my servers more unreliable. Two CentOS 7 VMs, setup at the same time. On one journald works just fine, on the other approximately once a month journald would stop logging. It wouldn't start logging again until you noticed and rebooted the server. The real issue this exposes is that on a systemd machine, journald doesn't just log for itself, it also supplies the logs to syslog. So on this server when journald broke, there were no logs whatsoever. This issue was apparently common on the version of systemd shipped with CentOS 7. The fix was to disable log compression in journald. What it highlighted to me was the inherent issues in an all encompassing system controller like systemd, in that if there is a bug somewhere in there you lose not only the added bells-and-whistles its intended to provide (journald in this case) but also the old previously reliable functionality (syslog). In my mind the fix for this is a redesign of systemd to make it an optional layer on top of the reliable functionality rather than a low level system component that everything else needs to depend on. In the case of logs journald should consume logs from syslog, not provide them to syslog.
- kaba0 5y ago
  Aren’t you assuming way too much on the architecture of systemd with only a surface level knowledge?
iso1210 5y ago
> So, pretend that this somewhat realistic scenario is happening to you: it's 4:00 am you just got a panicked call from someone at your company that the website is down. You log into a server and you want to see if the website is actually down or if it's just dns. You probably want to know the answers to these basic questions Those questions the page raises are based on what systemd tells you, not what I'd be askiny myself. I wouldn't want to know most of those, nor do they actually answer if "DNS is down" or not (what does it even mean). From my computer: dig www.mysite.com -- checks DNS is available externally. Check it at a few sites (dig www.mysite.com @1.1, @8.8.8.8, @9.9.9.9 etc). My monitoring checks my website every 10 minutes anyway from multiple locations across the world and tells me if something is wrong, so it won't be that. If I think the problem is that DNS lookups are failing on the server (so it can't see db1.mysite.com or whatever), something I might suspect if it takes a couple of seconds to log in as the reverse dns times out, I'd just log in and type $ dig www.google.com -- checks DNS lookups are working internally If it's not working, I'd check what's configured $ cat /etc/resolv.conf But oh dear, systemd has put its ugly fingers into DNS, and resolv.conf is pointing to 127.0.0.53, so I have to remember some convoluted command to run to try to find out what DNS it's using. The easiest way in a systemd world to do that is $ tcpdump -i any udp port 53 -nn One of the many reasons systemd is so broken. In systemd that's no longer the case, another daemon is running on my machine, another daemon to go wrong, another daemon to hide what's actually happening, another daemon to wake me up at 4am because it's not working. I type systemd-resolve --status and get 82 lines of nonsense (on my desktop), with perhaps 1 line perhaps telling my the 'current' server configured, and a few lines telling me other servers. But then there's different servers on different links, so it's all a bit of a guess, hence tcpdump is far more reliable to see where lookups are actually going. I have a small monitoring PC which runs a bunch of VLC sessions and webpages. It broke a couple of months ago because of systemd hijacking DNS. That means spending time looking in to why it's broken, which is a waste of time, or just having a script to fix it on bootup. The script is an ugly hack, but systemd is draining. It gets in my way, I never asked for it, but practically every distro forces its use in various (and constantly changing) ways. That's why some people get pissed off by it, it doesn't just add features you can use, it fundamentally changes the core of your machine without even buying you dinner first.
cyberpunk 5y ago
I really miss SMF :(

364 points by pcr910303 5y ago

ape4 5y ago

1. You don't have to clone then modify a 100 line bash script for a normal daemon's /etc/rc.d/init.d script.

fiddlerwoaroof 5y ago
Instead you have to write a couple-hundred-line bash script and then modify an under-documented ini-file to reference it
- merb 5y ago
  you never used systemd?
  - tgv 5y ago
    There's a bit of truth in that. The file format is underdocumented. You've got to look everywhere to find snippets of information, and then there are these weird incompatibilities between versions (Service.Type bit me once, and either After or Wants simply didn't do what the scant information suggested). And it often starts a script or a dedicated tool, e.g. for backing up, or watching directories. However, it's easier to deploy and keep track of your own services, sudo and chdir are built in, there's a watch dog, easy journaling. That's nothing to sneeze at.
    - fiddlerwoaroof 5y ago
      I’ve written a handful of systemd units now and, while the simple cases are nice, anything more complicated is really hard to figure out by reading manpages or other official sources of documentation. And, additionally, I’ve found it’s often easier to write a shell script to be invoked by systemd than to handle certain issues in the unit file itself. As far as the watchdog but things go there’s plenty of good enough solutions that don’t really on systemd: daemonize, DJB’s daemontools, etc.
      - thayne 5y ago
        I've written many systemd services, and other units. Some of which were quite complicated. I found the man pages extremely useful. Moreso than the manpages for sysvinit or upstart, when I used those systems.
      - kaba0 5y ago
        > I’ve found it’s often easier to write a shell script to be invoked by systemd than to handle certain issues in the unit file itself. But that’s a completely fine solution, service files are not meant for declaring arbitrary logic.
jude- 5y ago
The OpenBSD init system and OpenRC init system haven't had this problem for years -- even before systemd became a thing.
- CameronNemo 5y ago
  Similarly runit, s6, and Upstart have never had that issue. It is the oddest complaint to come up with when considering life without systemd.
  - jraph 5y ago
    Upstart was good and probably paved the way to systemd.
    - agumonkey 5y ago
      I wonder if systemd is influencing a new init right now
    - cbmuser 5y ago
      No, Upstart’s design was so bad that even Scott James Remnant (the creator of Upstart) admitted it’s design was inherently flawed. The biggest problem with Upstart was that it put the dependency chain upside down and just started a service whenever it’s dependencies were fulfilled, no matter whether it actually made sense to start the service.
      - ahartmetz 5y ago
        I remember reading about it and thought it seemed wrong, but presumably the author did his research and knew what he was doing. Oh well. His attitude and the readability of Upstart code were very good, though. The Upstart design was probably easier to implement. With the systemd design, the dependency tree needs to be traversed bottom up (figure out what needs to be started to start the targeted set of services), then top down (start dependees when dependencies are ready). With Upstart, top down was enough. Or so it seems.
      - 5y ago
deleted 5y ago
[deleted]
iso1210 5y ago
I have 90 scripts in my /etc/init.d directory. 51 are less than 100 lines, the most recent one I created is actually a perl script which is 29 lines long.
- fiddlerwoaroof 5y ago
  Yeah, with the #!-lines, sysv init scripts can be written in just about anything and, as long as you implement the expected command line interface, the init system will handle it correctly.
  - MarkSweep 5y ago
    Hmmm, I wonder if anyone has made program for this purpose that will read the rest of the file a declarative definition of a service? Like imagine a SYSTEMd unit file with a shebang at the top .
Arch-TK 5y ago
I've not seen even sysvinit based systems with this problem. Every service manager I know of solves this problem. This is as much an argument in favour of systemd as "don't have to clean up horse excrement" is an argument in favour of owning a fiat 500.
- viraptor 5y ago
  Most init scripts are in that ballpark. See openssh for example (88 lines) https://github.com/openssh/openssh-portable/blob/master/opensshd.init.in https://github.com/openssh/openssh-portable/blob/master/open... And yet, it still tries to use pidfiles (which systemd has a better alternative for), and notes "We really should check if the service is already going" which they don't do because it's an actual non-trivial issue if you're writing a standalone init script.
  - Arch-TK 5y ago
    Does anyone use these init scripts? Every popular sysvinit based init system uses short declarative service scripts for most services. Every non-sysvinit init system seems to mostly manage with 2 line scripts. And yes pidfiles are crap and I don't like them either (a genuine criticism of sysvinit and advantage of systemd) but systemd isn't the first init system to do away with pidfiles so it's hardly a unique strength.
    - viraptor 5y ago
      Could you show an example of a declarative sysv-init script? To be honest, I'm not aware of it being possible and ddg doesn't bring up anything. > Does anyone use these init scripts? By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them)
      - Arch-TK 5y ago
        This is OpenBSD's sshd init script: https://github.com/openbsd/src/blob/master/etc/rc.d/sshd https://github.com/openbsd/src/blob/master/etc/rc.d/sshd On an archlinux system running systemd I see that the equivalent of the rc_reload function is "ExecReload=/bin/kill -HUP $MAINPID" (which is actually less safe...) I'm not sure if you think that's declarative but to me it's basically as declarative as you can make a shell script. > By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them) I mean is anyone CURRENTLY using them? Because as far as I'm aware the linux sysvinit situation was a gigantic mess which nobody wanted to bother fixing despite the fact that OpenBSD and other BSDs had already fixed it by that point. I think most linux distros can be taken as an example of how not to do things if you want to make your life easier. Nowadays BSDs don't use these awful scripts. OpenRC based systems don't use these awful scripts. Who is using them?

A4ET8a8uTh0 5y ago

"I hope Alpine ends up with something similar!" Direct quote from the website. Eh, no. There is already a short list of viable non-systemd candidates. We.. well.. I don't need less. It already feels like I am transgressing some law by typing this on PopOS.

darthrupert 5y ago
To each their own, but I won't touch Alpine (outside of docker use) until they get systemd or something fairly similar.
- lixtra 5y ago
  The same design goals that make them a good docker distribution may have made them to stay away from systemd. Meanwhile others claim that systemd makes docker redundant anyway[1]. [1] https://news.ycombinator.com/item?id=26473581 https://news.ycombinator.com/item?id=26473581
- efiecho 5y ago
  > To each their own, but I won't touch Alpine (outside of docker use) until they get systemd.... Luckily that won't happen as systemd doesn't support musl which Alpine uses for libc.
  - 2pEXgD0fZ5cF 5y ago
    I'm very glad for that too.
rzzzt 5y ago
There is some work going on around replacing OpenRC in Alpine, this might be the thing the author is alluding to: https://skarnet.com/projects/service-manager.html https://skarnet.com/projects/service-manager.html
- A4ET8a8uTh0 5y ago
  Thank you for sharing this. I was not aware of it.

memco 5y ago

> If you need to reboot the server right now for some reason, will that service come back up on reboot? I didn’t see this discussed more after it was asked, but I could use some pointers on how to answer this question since I have a service I wrote that I think should start on reboot, but has to be started manually. It would be nice to know when I have fixed it without having to reboot after each change to just to test. Anyone know how I can do that?

rav 5y ago
In the systemctl status output, it says "Loaded: loaded (...; enabled; ...)" - "enabled" means that the service starts on boot. Use "systemctl enable" to "enable" a service. It creates a symlink in /etc/systemd/system/multi-user.target.wants assuming the service's unit file says to enable it through the "multi-user" target. By listing that directory you can see what services load on boot.
- CameronNemo 5y ago
  Not exactly. Socket units can pull in services that are not otherwise enabled. You also have to search through all of your udev rules to check if there are SYSTEMD_WANTS directives pulling services in.
  - Arnavion 5y ago
    rav (correctly) said that enabled services are started automatically on boot. They didn't say that non-enabled services are not started automatically on boot.
    - nemetroid 5y ago
      They also said: > By listing that directory you can see what services load on boot.
      - Arnavion 5y ago
        Yes, and I didn't read that sentence as disallowing other ways to autostart services either.

ratsmack 5y ago

>systemd is a set of building blocks that you can use to make a linux system. A more accurate description would be "Systemd is a set of building blocks that you can use to make Linux more like Windows".

dale_glass 5y ago
systemd is incredibly Linux centric. In fact it intentionally doesn't even try to be remotely portable, because an explicit design aim was to expose all the cool stuff that Linux has, but wasn't getting enough use.
- trasz 5y ago
  Sure, but design-wise it follows Windows principles: it’s tightly coupled, and not particularly tweakable.
  - deleted 5y ago
    [deleted]
  - dale_glass 5y ago
    These days I see it as a bonus. I don't want to mess around with the boot process. I don't want to write some sort of glue layer in between loosely coupled parts. And I certainly don't want somebody else to do it either, because that too often tends to result in some kind of surprise that one has to deal with at 3 AM. What I want is a solid system that does what it's supposed to -- start stuff -- in a maximally boring fashion.
    - trasz 5y ago
      Sure, I understand that. But this way your getting another Windows, except with source code.
      - dale_glass 5y ago
        I'm not seeing the problem. I use computers to get stuff done, not because of some sort of religious ideas about how things out to work. While I prefer Linux in general, Windows does have some good ideas in some places, and I don't see any issue with using them.

Decabytes 5y ago

I just learned about Vtubers this week. I would have never thought this would have crossed into the Linux world but here we are!

mwvr 5y ago

A furry named Xe promoting systemd, just another day in the Linux twilight zone.

AshamedCaptain 5y ago

I have a system where systemd decides, about half the times I boot it, to immediately unmount all of the filesystems listed in fstab literally right after having finished mounted them as part of the regular boot process. All of this with other daemons starting in parallel. It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login. No idea how to even start debugging it.

jraph 5y ago
I would run systemd-analyze critical-chain and then systemd-analyze blame to see if there is a service that could do this. And then I would look at the detailed boot logs. If nothing can be found, I'd look for some way to increase the log level. If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.
- AshamedCaptain 5y ago
  The logs are mostly useless and random. There is nothing that consistently appears between the point where it finishes mounting (lots of "Mouting...", then "Reached target Local File Systems") and where it starts unmounting ("Stopped target Local File Systems", then lots of "Unmouting ..."). Sometimes these two lines are less than one second apart. But I haven't found any message which is consistently between these two points that would point to a suspect. And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already). It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...
  - jraph 5y ago
    Most certainly you can only analyze things on boots where the problem appears, unless logs are persisted across boots. Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run. If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.

bfrog 5y ago

Yeah it's all pretty darn good. People forget the cluster of bash scripts, various cronz, inetd, log, and other supervisor type daemons all to do what systemd does better with a simpler config. Now ask me about pulseaudio and there's a project with nothing but issues, looking forward to the new pipewire overlords.

tediousdemise 5y ago
> Now ask me about pulseaudio and there's a project with nothing but issues Don’t forget Bluetooth, WiFi, and graphics (cough Nvidia). Or any peripheral, for that matter. Until vendors stop forcing us to play games with their broken blobs, Linux will forever be just a grand experiment for frustrated hobbyists and unfortunate professionals around the globe. A digital tragedy of the commons.
- techrat 5y ago
  >Linux will forever be just a grand experiment for frustrated hobbyists and unfortunate professionals around the globe. Linux is the most widely used kernel in the entire world. If you consider 'Linux' to be the OS, it is #1. If you consider all distributions that use the Linux kernel, it is also #1, collectively and individually (Android.) If you look at the ranked list of Supercomputers, Linux has the entire top 500. What the hell are you talking about?
  - OhSoHumble 5y ago
    Probably just referring to desktop usage.
  - kllrnohj 5y ago
    > and individually (Android.) Which, worth noting, does its own thing for init & services ( https://cs.android.com/android/platform/superproject/+/master:system/core/init/ https://cs.android.com/android/platform/superproject/+/maste... ). I think it's fair to assume they were talking about Linux as a desktop OS, which for some reason absolutely refuses to learn lessons from the other 99.9% of the market. And is also infatuated with internal fragmentation despite its tiny usage. That makes it really fun as a hobby for sure, and powerfully flexible for datacenter usages (from the mundane to the supercomputers). But also pretty terrible as a "I just want my laptop to work reliably & well" OS.

jph 5y ago

> By not having something competitive Alpine is less and less attractive for newer production deployments. Alpine uses OpenRC (https://wiki.gentoo.org/wiki/OpenRC https://wiki.gentoo.org/wiki/OpenRC). IMHO Alpine and OpenRC tend to make production deployments more predictable and reliable, because of deliberately simple init ordering, as well as a smaller surface area as compared to systemd.

detaro 5y ago
There is some work on a replacement that does more while fitting the requirements and philosophy of alpine: https://skarnet.com/projects/service-manager.html https://skarnet.com/projects/service-manager.html
- mendelmaleh 5y ago
  Good read, thank you!
- GekkePrutser 5y ago
  Interesting, I have a lot more faith in the alpine team implementing something like this than in RedHat. Even when RedHat made consumer OSes (before spinning them off to Fedora) I didn't like them. What they're doing looks like a great alternative. I also hope they will not have any reliance on dbus as systemd does. Reliance on dbus is one of the main issues I have with systemd (besides its size and complexity).
  - zxzax 5y ago
    I've seen a lot of these comments to this effect but I'm still not quite sure: what is the issue with dbus? Is there another replacement that meets all the use cases?
    - detaro 5y ago
      Question is if an init system needs all the use cases dbus can support. Dbus always feels unwieldy to work with, especially once you leave specific desktop stacks, and for me personally, the biggest failures of systemd-based systems have been issues with dbus not working properly for whatever reason.

xena 5y ago

Author of the talk here. If you have questions for me please feel free to reply to this comment with them.

jen20 5y ago
I did a talk at HashiConf EU a few years ago with the same title - interesting to see where our ideas of "the good parts" overlap!
- xena 5y ago
  Wow, I think that was completely by coincidence. I've never seen your talk before. Where did we overlap?
  - jen20 5y ago
    I think we definitely agree on the most of the benefits - there's a transcript at [1]. I'm happy to see more people discuss the benefits and practical uses rather than the sea of hate that systemd usually gets (especially in these parts). [1]: https://www.hashicorp.com/resources/systemd-the-good-parts https://www.hashicorp.com/resources/systemd-the-good-parts
Hackbraten 5y ago
No question, just thank you for an excellent article. I’ve found it difficult to come across articles that focus on the good parts of systemd.
Traster 5y ago
Hey, thanks for the great talk, I often type something like "journalctl -xfe" and just get completely lost, you mentioned that's out of scope for this talk but do you have a recommendation for some hints about how to use journalctl effectively?
- dale_glass 5y ago
  A quick guide: * Want logs since boot? journalctl -b. The first boot is -b 0, the previous boot is -b -1. * Want logs for smartd? journalctl -u smartd * Want warnings and above? journalctl -p warning * Want logs since yesterday? journalctl -S yesterday * Want logs on April 22 between 10 and 11? journalctl -S "2021-04-22 10:00" -U "2021-04-22 11:00" * Want logs with microsecond timestamps? journalctl -o short-precise * Or in JSON for parsing? journalctl -o json * Want tail -f? journalctl -f

fghorow 5y ago

"This page intentionally left blank." (Yeah, I'm a dinosaur who remembers dead tree IBM manuals. Sue me.)

exabrial 5y ago

Things that are terrible: logging. Why did we have to screw this up? Just writing stuff to a file was fine. Things that are good: simple syntax, easy to use utils Hidden gems: cgroups, file systems containment, etc. Systemd could completely eliminate the need for stuff like docker by totally isolating a process.

porsager 5y ago
That's interesting. Do you have a link to more info about isolating a process like that?
- comboy 5y ago
  Arch wiki is usually a good place to learn something in a succinct way (I've never even used Arch): https://wiki.archlinux.org/title/cgroups https://wiki.archlinux.org/title/cgroups
moshmosh 5y ago
The nice thing about docker is that you can use the same tooling on lots of operating systems, not just that it wraps some Linux functionality. Sure, it may involve using a VM, depending on the system, but I still get to just run "docker" and "docker-compose" and such, as if it were native. A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. Really, though, if I care about isolation and don't want to use docker, I'm gonna reach for FreeBSD + jails anyway, and I don't think systemd's gonna change that for me any time soon. If I want containers with better tooling outside the target OS, I'll use docker. Systemd falls in an awkward middle that I don't have much interest in.
- vially 5y ago
  > A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases. But that's exactly what happens with docker anyway (e.g.: Docker is still running in the Linux VM when running it from a different operating system). And considering the widespread systemd adoption throughout the Linux ecosystem, it's far more likely for systemd to be already installed in the VM than docker anyway.

ohazi 5y ago

I actually like using systemd. I took the time to learn how to use it, and as a desktop Linux user, it's honestly quite nice. But the one thing that still really pisses me off about systemd-the-project was the fact that they ate udev-the-project. In my view, that decision was unnecessary and was done for purely anti-competitive reasons. If you're not familiar, udev is basically the user-facing device manager for Linux. It's what allows you to easily configure rules and permissions for all the USB devices that you plug into your computer. These devices still need kernel drivers, but udev is how you tell your system "Please let user ohazi use this device, and also give it a convenient name like /dev/someDevice0" By devouring the udev project, the systemd maintainers have guaranteed that dealing with USB devices on non-systemd systems was going to be a giant pain. Then came the forks -- Gentoo maintains eudev, which is a systemd-free fork of udev. But really, this shouldn't be a fork. Udev should be independent and available on all systems. If systemd wants to do something special with devices, they should use udev APIs like everybody else. In my view, this is what finally allowed systemd to win the init war, despite all the protest. Ideological arguments about how best to start daemons is one thing, but you simply can't use a modern system without a sane approach to USB. Edit: It seems that some of my concerns were overblown, e.g. you apparently can run udev without systemd as pid 1. Udev has a build-time dependency on systemd, but not a run-time dependency. The eudev fork removes that build-time dependency, and (maybe?) also papers over some other inconsistencies. I'm pleased that this is the case. I'm annoyed that the discourse around systemd/udev has been muddy enough to lead me to incorrect conclusions (and I'm aware that the original version of this comment likely added fuel to this particular fire). Oh well... live and learn.

edoceo 5y ago
I love Gentoo and really like they chose to make eudev and a elogind so I can continue to use OpenRC, which has been working great for me since forever.
- ohazi 5y ago
  Don't get me wrong -- I too am extremely grateful to the Gentoo devs for maintaining eudev. It's also the obvious choice for highly memory constrained embedded Linux systems that can't fit systemd. I just think it's absolutely bonkers that it has to be a fork... eudev should really just be udev.

dijit 5y ago

Copying my reply from Lobste.rs because there’s more people here who can pick me apart: _FD: I’m not a large supporter of SystemD after using it when it was first introduced in Fedora and watching it take over the ecosystem, I do use it and as a sysadmin/SRE type I’m heavily versed in it and how it functions, so, everything with a pinch of salt._ I gave the video a good shot, half the time is spent going over some systemd-ecosystem basics; The author then makes the case that Alpine is a less attractive development/production-deployment platform due to lack of this ecosystem (though they make effort to avoid directly saying that systemd would be a good solution itself.) I couldn’t disagree more honestly. I love the concept of Alpine: as small as possible and no smaller. The attention to minimise dependencies makes it the distro of choice in containers; I’ve even considered (strongly) running it on some machines where systemd would have been a lot of needless complexity. I think deviating from this path for alpine is a grave error, there will always be a place for a niche system that is small; especially when it’s commonly paired with some other log managment/service orchestrator like Alpine is. Additionally, it gives people like me an option to run without init systems that could be considered heavy and intrusive with heavy dependency chains. So, kindly, Alpine: please don’t.

ChuckMcM 5y ago

Okay, I'll admit I was hoping for a blank web page :-). Not a fan. I always ask ""What problem is this trying to solve?" and then "Is this a good solution for that?" I appreciate the "startup dependency problem" and when I was in the Systems Group at Sun we worked with AT&T on their solution which was the whole "run level" thing. And I appreciate the whole "shell scripts can do anything" problem that just using a naming scheme on shell scripts to run can lead to unexpected behaviors. From an architectural taste perspective, I prefer systems which are more comprehensible in parts and thus don't try to abstract out semantics to the point of incomprehension. So at the end of the day, something with a better model, better visibility into its operation and configuration, with a way to validate and debug dependencies would be good. But I haven't been impressed with systemd.

dale_glass 5y ago
It solves a whole lot of problems to me. * It gets rid of a system cobbled together from init, inittab (which people rarely remember exists), monit, cron, and inetd/xinetd. * It removes a bunch of horrible hacks in init scripts such as calls to sleep. * It removes the need for pid files. * It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts. * It allows precisely tracking what belongs to each service. I can easily find out what a random process belongs to. * It captures all the log output of each process * It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again * As a developer it means I don't need to write a slightly different script for every distro * It makes logging a whole lot pleasant * It allows having user services without having to hack around it with cron * It allows a whole bunch of settings to be applied to any random service, without having to edit a shell script and figure out how to stick it in there. * It means services run identically when started by hand, without the confusion of a service inheriting my local environment. * It makes my VM servers boot faster. And if you're going to pipe up with "I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand, when they're needed. They adapt to the load.
- ChuckMcM 5y ago
  I had to chime in here and say I really like this post, thank you for taking the time to write it out. I would give it extra up votes if I could. It is a nicely concise summary of the problems. I'd disagree with the characterizations of the previous system as "cobbled" or "horrible hacks" but there are solid things that need to be solved well by any system initialization architecture/service.

viseztrance 5y ago

I ignored systemd because I thought it didn't affect me, as I only run linux (fedora) on my desktop, and hardly had any issues at that. But recently, when I changed my fstab file to mount a drive at boot, I made a typo. My system wouldn't boot at all and it wouldn't even drop to a shell so I can chroot the file system like you would with init.d. I thought that there's some arcane command I don't know about. But there wasn't. You're locked off, and expected to boot off something else to fix your system. Horrible. Having this said, I honestly expected way more on the section about benefits to users.

zaarn 5y ago
Atleast on Arch, Systemd should drop you into the Emergency shell if it cannot mount a device that isn't defined as optional. IIRC it might require a root password to be set, however. Otherwise, edit the kernel commandline to add "systemd.unit=emergency.target" to the end, which triggers systemd to straight boot into this console without trying anything that it doesn't need to bring up the console.
- otterlicious 5y ago
  Or if you're blocked by the root password prompt add `init=/bin/bash` and make your filesystem writable if necessary with `mount -oremount,rw /`
  - zaarn 5y ago
    This will block systemd however. The nice thing about the emergency shell is that you can use systemd things like trying to mount filesystems. And you can later isolate multi-user to continue the boot when you finished your fix. /bin/bash would be atleast useful to try to get a root password set.
- viseztrance 5y ago
  It tried when the mounting failed, saying it would drop me to a shell and then said "it fail to open up a shell".

efiecho 5y ago

Poetteringware seems to be highly praised by mostly young people who would like Linux to be a Windows clone. People who understands Unix concepts tends to dislike it.

StreamBright 5y ago
Well yes and no. I like the good parts of SystemD (not the parts this video is talking about). For me there are 3 things that SystemD gets right: - no shell scripts for service configurations - no forking services (no need for daemon mode anymore) - logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great) And the rest is just pure madness. - DNS (but whyt?) - NTP (but why????) - system limits (seriously, da f???) - pid 1 should be a very simple service (More: https://ewontfix.com/14/ https://ewontfix.com/14/) I have spent 25 years on Unix, so I guess I understand the concepts.
- Arch-TK 5y ago
  > - no shell scripts for service configurations Other than "sysvinit based init had horrific shell scripts with 100 lines of repeated nonsense", given that almost every alternative to systemd which still uses shell scripts needs around 2 lines of shell per service (including the shebang) what exactly is the issue with shell scripts (if all they do is run a wrapper which sets up the environment of a program or the program itself)? > - no forking services When I used archlinux these were extremely common and I used archlinux as recently as last year. > - logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great) So runit handles spawning a log daemon before the service is started to ensure all log entries are kept and comes with a really simple yet extremely powerful log daemon (but obviously you can use almost anything else including cat if you want). It manages to integrate logging tightly into service supervision while being maximally flexible and decoupled. What do you think about this?
  - 1vuio0pswjnm7 5y ago
    I use daemontools instead of runit. What am I missing. multilog is the simplest (smallest) and most versatile log rotating daemon I have found.

deleted 5y ago

[deleted]

cerved 5y ago

I can't weigh in on the pros/cons of systemd because that's out of my depth. But I am somewhat fascinated by the hate it seems to endure. Like this https://suckless.org/sucks/systemd/ https://suckless.org/sucks/systemd/ I guess it's a bit of a watershed

Arch-TK 5y ago
For something which the authors seem intent on forcing down everyone's throat whether they want it or not (genuinely you can find quotes of Lennart claiming he wants to make it more difficult to not use systemd or that he wants to "gently push" people towards it) it's not really surprising. If their only goal was to gain adoption based on the merits of their software while not intentionally making themselves intrusive nobody would really care.
- bkor 5y ago
  > forcing down everyone's throat whether they want it or not You're being highly offensive towards these developers for no good reason. All kinds of grand claims, just because things are done on a way that you don't agree with. You've stated that systemd developers are wanting to force people to eat things. And their action is to actually force it into someone's mouth, likely requiring the person to be put into restraints to make this possible. You're being highly offensive. Systemd has existed for loads of years, yet this accusation continuous to be repeated. Aren't you aware of German libel laws? If I'd accuse you of such behaviour, wouldn't you get upset? Why say such things, things that suggest criminal behaviour, so easily? These things are repeated so often that it isn't even moderated upon. It's odd that this behaviour is tolerated.
  - cerved 5y ago
    Forcing something down everyone's throat is an hyperbole. They are not slandering systemd developers, nobody's suggesting people have to eat systemd. This is not offensive and it's not a crime, it's an opinion

yuuta 5y ago

I'm using split horizon DNS with WireGuard on my systems (I mean using special DNS servers for some domain names and the public ones for the rest. Correct me if I'm wrong). I used to try systemd-networkd and systemd-resolved because they seem to fit together. I found systemd-resolved is extremely buggy with systemd-networkd managed WireGuard. My WireGuard endpoints use domain names and systemd-networkd fails to resolve them after a manual restart. I couldn't figure out why. I also could not find a way to NOT combine split horizon DNS with network interfaces (like most documents online only include scenarios like resolving corporate domain names using a VPN interface but that wasn't my case). I have multiple VPN connections to the same network (runs BGP basically) and I did not want the split DNS to bind to some specified interfaces. Moreover, systemd-resolved seems not able to do split DNS without systemd-networkd. It needs systemd-networkd managed interfaces to do that. What? I just want separate DNS servers for separate domains. My WireGuard interfaces are managed by wg-quick. Correct me if it is possible to use systemd-resolved standalone in this case. The hard systemd-resolved dependency of systemd-networkd also had issues. Sometimes, I just want systemd-networkd to configure resolv.conf using DHCP DNS but it looks like I HAD TO START systemd-resolved AND LINK /run/systemd/stub-resolv.conf to /etc/resolv.conf in order to achieve that (correct me if there's a way to not use resolved completely). Excuse me? systemd-networkd itself was buggy, too. I once encountered https://github.com/systemd/systemd/issues/18108 https://github.com/systemd/systemd/issues/18108 and my friends reported they faced this issue as well. I don't mean that software should not have bugs but encounter them is really inconvenient. In conclusion, I mean, systemd components seem to fit together and help you do a lot of things, but they did't do the things well, compared to separate programs for their specified tasks. Moreover, you have to deal with various components of systemd and figure out their relationships in order to achieve some specified tasks. For split DNS, I'm using dnsmasq now and it's working pretty well.

kbenson 5y ago

I think systemd brought a lot of good stuff to the table, but I also think it's a sprawling mess with poorly named and documented (as in there are behaviors implied by statements in different config directive sections that make it hard to reason about how stuff works). I'm looking forward to the successor of systemd, or at a minimum, them rolling out config syntax 2.0 which breaks compatibility in an effort to make stuff more sanely named and documented. I know that would cause huge churn. I know it will never happen. But I can dream, because I don't think anyone can look at how systemd configs define dependencies and say that they both understand the nuances and that those directives are are named sanely in a way that normal people can understand, and that's indicative of a lot of stuff with systemd. Even though it was probably designed by some of the most knowledgeable people WRT init systems, it grew organically as they discovered new things they needed, and it needed a lot (which I think just goes to show how little was really understood by anyone about how to tightly couple these things).

deleted 5y ago

[deleted]

GekkePrutser 5y ago

I don't agree with her sentiment that alpine should adopt something similar. The whole reason I use alpine is because it doesn't use systemd. I like alpine because it uses a simpler, lighter approach. It's not for everyone, and I'm pretty sure alpine maintainers don't intend it to be for everyone. It serves its niche really well as it is. For those who like systemd there's many distro choices already. Edit: As detaro mentioned in another comment ( https://news.ycombinator.com/item?id=27176391 https://news.ycombinator.com/item?id=27176391 ) they are actually working on something, but they aim to solve the various issues around alpine that don't fit well with alpine's philosophy. It sounds like a good story and I'm open to this idea. The main reasons I don't like systemd are its complexity and its dbus reliance, basically bringing too many desktop paradigms into all Linux systems, even servers and containers. So about my above point: I'm open to it adopting a service manager but not too similar to systemd :)

deleted 5y ago
[deleted]

theteapot 5y ago

Cool .---------. .---------. | SystemD |------>| Service | '---------' '---------' Not Cool .---------. .---------. | SystemD |<------| Service | '---------' '---------'

dale_glass 5y ago
I don't get it?
- vgel 5y ago
  Using SystemD to manage your services easier if that's what you want is nice (aka, if your SystemD setup depends on a service, that's fine). It's not great if the service depends on SystemD, because then it can't be used with other init systems (without patching / forking, at least).

tlamponi 5y ago

The declarative format of systemd unit files is just much easier to maintain, grasp and allows doing things that just cannot be done else, for example, a generic way to parallel starting service at boot, as all dependencies and before/after ordering is known. And the great things is, that declarative format means that systemd can actually be replaced by something else, as one just needs to be able to parse the ini-style configs and execute the ExecStart and ExecPreStart in order. One really cannot do that to the other init-systems, AFAICT, they are far too scripted and rely a lot on side effects to do anything like that, besides packager choosing different solutions for the same problem. Some KI-paper could possibly take a shot at parsing out some sensible information reusable for another tool ;-) I can get the dissentient about systemd incorporating so much stuff, but just from POV of it as init-system it is really nicer to work with, speaking as user and as someone which job includes packaging software for a Debian based distro.

froh 5y ago
in system V init scripts on Linux they had standardized dependency declaration comments. when you added services you ran a dependency checker which would translate the dag into a total ordering, laid out using 01 02 03 ... numeric file name (symlink) prefixes. it was robust, simple and straightforward. however it was built around bash. which was large and slow to start up,compared to e.g. dash as scripting shell. and dash was not widely around yet. so each script had the cost of bash startup, the milliseconds of which added up. and at some point the tipping point was reached.
- tlamponi 5y ago
  > in system V init scripts on Linux they had standardized dependency declaration comments. when you added services you ran a dependency checker which would translate the dag into a total ordering, laid out using 01 02 03 ... numeric file name (symlink) prefixes. Yeah, I know, but in practice this did not really work out and was a mess. In fact, every update was scary to do, in that time I frequently just re-installed the distro on my private workstation, simpler than to deal with the rc stuff. A plain setup may have worked, but anything slightly complex was just bonkers to do. > it was robust, simple and straightforward. Do you package for a relevant distro or is this just your experience as user? Asking because I just personally do not know any maintainers of software with a slightly complexer system interaction which would agree with that. There surely are some, but FWICT on some ML's anti/pro systemd flames it's <1% of maintainers - and from them it's actually almost never the new unit file format what is the issue, but the way systemd eats up basic system services. IMO the systemd unit files are exactly what you say "robust, simple and straightforward", the implementation may not be, I never even touched that, but the unit file schema is just better, as it's not bash/dash with some slight optional possibility to use some hacks like comments (urgh) as schema, but actually provides a standardized format for doing that and more. As someone maintaining dozens of packages and as common user, I'm happy that I do not need to mess around and fight the init-system, and I'm not the only one thinking so[0]. [0]: https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_archlinux_embrace_systemd/d3rhxlc/ https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_a... Thanks for the constructive downvote though ;-)

arid_eden 5y ago

The post was on 'the good parts' of systemd, so complaining about balance seems unfair. I will call out one aspect that the author deemed a 'good part' as in my experience its a bad part: journald. Usability: journald requires administrators to learn a new command to access their journals - everytime I have to use it I need to look up the syntax again. Compare with logs in /var/log and you can use all the standard unix tooling (text editors, tail, grep etc). Reliability: in my use journald has made my servers more unreliable. Two CentOS 7 VMs, setup at the same time. On one journald works just fine, on the other approximately once a month journald would stop logging. It wouldn't start logging again until you noticed and rebooted the server. The real issue this exposes is that on a systemd machine, journald doesn't just log for itself, it also supplies the logs to syslog. So on this server when journald broke, there were no logs whatsoever. This issue was apparently common on the version of systemd shipped with CentOS 7. The fix was to disable log compression in journald. What it highlighted to me was the inherent issues in an all encompassing system controller like systemd, in that if there is a bug somewhere in there you lose not only the added bells-and-whistles its intended to provide (journald in this case) but also the old previously reliable functionality (syslog). In my mind the fix for this is a redesign of systemd to make it an optional layer on top of the reliable functionality rather than a low level system component that everything else needs to depend on. In the case of logs journald should consume logs from syslog, not provide them to syslog.

kaba0 5y ago
Aren’t you assuming way too much on the architecture of systemd with only a surface level knowledge?

iso1210 5y ago

> So, pretend that this somewhat realistic scenario is happening to you: it's 4:00 am you just got a panicked call from someone at your company that the website is down. You log into a server and you want to see if the website is actually down or if it's just dns. You probably want to know the answers to these basic questions Those questions the page raises are based on what systemd tells you, not what I'd be askiny myself. I wouldn't want to know most of those, nor do they actually answer if "DNS is down" or not (what does it even mean). From my computer: dig www.mysite.com -- checks DNS is available externally. Check it at a few sites (dig www.mysite.com @1.1, @8.8.8.8, @9.9.9.9 etc). My monitoring checks my website every 10 minutes anyway from multiple locations across the world and tells me if something is wrong, so it won't be that. If I think the problem is that DNS lookups are failing on the server (so it can't see db1.mysite.com or whatever), something I might suspect if it takes a couple of seconds to log in as the reverse dns times out, I'd just log in and type $ dig www.google.com -- checks DNS lookups are working internally If it's not working, I'd check what's configured $ cat /etc/resolv.conf But oh dear, systemd has put its ugly fingers into DNS, and resolv.conf is pointing to 127.0.0.53, so I have to remember some convoluted command to run to try to find out what DNS it's using. The easiest way in a systemd world to do that is $ tcpdump -i any udp port 53 -nn One of the many reasons systemd is so broken. In systemd that's no longer the case, another daemon is running on my machine, another daemon to go wrong, another daemon to hide what's actually happening, another daemon to wake me up at 4am because it's not working. I type systemd-resolve --status and get 82 lines of nonsense (on my desktop), with perhaps 1 line perhaps telling my the 'current' server configured, and a few lines telling me other servers. But then there's different servers on different links, so it's all a bit of a guess, hence tcpdump is far more reliable to see where lookups are actually going. I have a small monitoring PC which runs a bunch of VLC sessions and webpages. It broke a couple of months ago because of systemd hijacking DNS. That means spending time looking in to why it's broken, which is a waste of time, or just having a script to fix it on bootup. The script is an ugly hack, but systemd is draining. It gets in my way, I never asked for it, but practically every distro forces its use in various (and constantly changing) ways. That's why some people get pissed off by it, it doesn't just add features you can use, it fundamentally changes the core of your machine without even buying you dinner first.

cyberpunk 5y ago

I really miss SMF :(