13 ms·

Systemd has been a complete, utter, unmitigated success

106 points by Mond_ 1y ago

dontlaugh 1y ago
These were pretty much my feelings back when I envied launchd and was frustrated with some upstart behaviour. Systemd turned out better than both.
spicyusername 1y ago
I remember when I first upgraded a system to a version using it. This was probably a RHEL 6 -> 7 update back in 2014 or so. It made so many common things I had to do so as a sysadmin super straightforward.
- freedomben 1y ago
  Same, that was my first experience as well. It was a breath of fresh air in many ways, and after my initial learning of the new syntax, I quickly became a convert. That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
  - SAI_Peregrinus 1y ago
    I think the biggest problem with systemd is that they named the main init service manager systemd and also named the organization systemd! It'd be like if GNU named their initrd "GNU". Their other tools just have systemd prepended, e.g. systemd-journald, much like GNU Make or GNU libc (glibc) or similar. The "scope creep" issue doesn't get complained about for GNU because it's clear they're an organization making a bunch of tools for different purposes. Systemd-the-organization is the same, but the naming makes it unclear that many of these tools are not depended on by systemd-the-init-system.
- Propelloni 1y ago
  And some things got super hard to do, e.g. sequence. I guess it is sorted now, but either way, I wouldn't want to go back.
jmogly 1y ago
Agree wholeheartedly.
up2isomorphism 1y ago
There are numerous distributions not using systemd. So adding “complete, utter, unmitigated “ in front of success seems far-fetched.
- freedomben 1y ago
  The whole post does have a somewhat hyperbolic tone to it. Personally, I agree with the content and analysis, but I could see the language could feel a bit off-putting, especially given the past flame wars over systemd
  - jhickok 1y ago
    It's a lighthearted post, so probably not worth tone policing.
    - freedomben 1y ago
      Agreed, I didn't mean to tone police, just acknowledging it might be off-putting to some people who got wrapped up in the battles back in the day. I wouldn't suggest the author make any changes
    - up2isomorphism 1y ago
      It has nothing to do with the tone, it is just a fact. So there is nothing to police about.
rootnod3 1y ago
I wouldn’t call it a success it all. Almost managed to do was moving or forcing other software to get locked in. GDM, Podman, Pipewire…just to name a few.
- fsflover 1y ago
  As another comment says, it's a success in the same way Windows has been a success.
- NewJazz 1y ago
  Gnome is pretty guilty of adding hard dependencies on systemd, but podman and pipewire have worked fine for me without systemd... Not aware of any tight integration there that is mandatory (obviously quadlets, but you don't have to use that feature)
iambvk 1y ago
I am not a sysadmin -- so I keep forgetting the command to see logs cause I only need it once every 3 months. So, systemd change is a pain for me.
- slashdev 1y ago
  AI is so helpful for doing those kinds of tasks in the terminal that you only do a couple times a year and can never remember the exact incantation
  - hundchenkatze 1y ago
    You only need to look those things up once, put it in a snippets.txt file, and then search in the file when you need it.
    - slashdev 1y ago
      I keep a per project file like that with common commands and related things.
- hearsathought 1y ago
  Keep a file with a list of commands used semi-regularly. Or have a script dump your command history into a file. Better to spend a minute searching for a command in a file than to waste an hour recreating it.
- akdev1l 1y ago
  use a shell function/alia? show-logs() { journalctl -u "$1" } show-logs nginx (Not that I think journalctl -u even needs an alias tbh but if you want it you can have it)
haiku2077 1y ago
I've worked as a sysadmin/devops for a double digit number of years. Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
- thyristan 1y ago
  And made a number of new ones appear. E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd. Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations. Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse. Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons. I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
  - idoubtit 1y ago
    The grand-parent post was about the general consensus: sysadmins generally like systemd. So I'm not sure answering individual concerns is useful, but I'll take the bait... > now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd. No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng. And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files. > Boot and shutdown has previously been deterministic, now things just randomly hang. That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either. > Network device naming is also a fun one. The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.
bcoates 1y ago
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example). You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
- inetknght 1y ago
  For debugging my desktop: journalctl --follow --tail --no-trunc -b 0 For anything else: export to a syslog server, which basically any tool that matters will support in some fashion.
  - LargoLasskhyfv 1y ago
    journalctl: unrecognized option '--tail'
- NewJazz 1y ago
  Cloudwatch fucking sucks. Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector) Even ChromeOS uses it, even on devices that still use Upstart.
  - bigstrat2003 1y ago
    You're being downvoted but you're absolutely right. The fact that Cloudwatch doesn't support journald is a major, major fail on AWS' part. It's not like this is new or obscure software.
    - NewJazz 1y ago
      I'm just surprised anyone wants to use cloudwatch when they don't need to. It is expensive, and far from the best observability platform.
      - dontlaugh 1y ago
        It's always there by default. That's the only reason I've ever seen it used.
shrubble 1y ago
My view is pragmatic: if you are managing the server and I am a user, then use systemd if you want to. If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
- hungryhobbit 1y ago
  FreeBSD is many great things, but ... a pragmatic choice?
  - linksnapzz 1y ago
    Yes; it does work nicely in many circumstances.
  - shrubble 1y ago
    It works for many, including Netflix. The jails are better integrated as a lightweight container system than Docker is on Linux, IMHO.
  - burnt-resistor 1y ago
    And powers zillions of unsung, critical infrastructure tasks.
pram 1y ago
I personally love the init/service/unit-file portion of systemd and have few complaints. It has a lot of powerful features for security, cron, etc that are very simple to use. I also really like journald. The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
- thyristan 1y ago
  And those ancillary managers are usually half-baked or outright broken and dangerous, in that they only implement half of the stuff they are trying to replace. E.g. https://github.com/systemd/systemd/issues/25676 https://github.com/systemd/systemd/issues/25676
- bryanlarsen 1y ago
  Hard disagree. Name resolution feels like a solved problem, so why fix it if it isn't broken? But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better. I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved. Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
  - msgodel 1y ago
    I think even for new people the systemd configs are much more complex (and the APIs are certainly much more complex) than the libc ones which worked well (often better.) The problem is that almost nothing in Systemd shares theory with the rest of the OS. It's entirely its own island of new ideas, commands, and configuration language that you have to learn and memorize.
  - const_cast 1y ago
    The older init systems were incredibly frail and often did not work. The reason is because Bash, as much as we all love it, is actually super mega ass. It's very bug-prone and actually one of the least portable languages ever invented, in practice. Because you can't, or shouldn't, call to any other binary from within Bash, otherwise you risk your portability. Now your script won't work the same on other platforms, or even the same platform with slightly different configuration. But um... yeah... calling to other binaries is the entire ethos of Bash. It's a shell, that's supposed to be all it does. Even getting a "true" value should be a call to another binary. Unit files, despite being a bespoke format for one singular program, are actually more portable. Like, WAY more portable.
artooro 1y ago
I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.
- haiku2077 1y ago
  +1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
  - SEJeff 1y ago
    Or if the service didn't support pam_limits because it was legacy trash, you had to hack something into the initscript like `ulimit -n XYZ` and restart it. Now things like this are trivial and easy to solve. Using systemd makes large scale Linux systems administration much easier. Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in. Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.
tylerjl 1y ago
Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!
- rootnod3 1y ago
  This is a nit-pick, but if you picked the title that way to BE hyperbolic, I am not sure if that is the best thing to do for a technical discussion. It sets an internal opinion before even clicking on it. It subconsciously makes people go into the article with an already preconceived notion.
pessimizer 1y ago
No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product. They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now. Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
- Analemma_ 1y ago
  > Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product. Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
  - jmclnx 1y ago
    >Who exactly was going to write the systemd alternative? And that quote proves 100% what is wrong with Linux these days. In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more. That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch. Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then. Seems these days we are dealing with some corporate behemoth drowning in red tape.
  - 1y ago
fsflover 1y ago
> The unix philosophy cries out: is this the end of Linux (or, as many are calling it, GNU plus Linux)? I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence: https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd https://news.ycombinator.com/item?id=42918448 https://news.ycombinator.com/item?id=42918448 https://news.ycombinator.com/item?id=42889792 https://news.ycombinator.com/item?id=42889792 (I reposted my comment from previous submission)
scraptor 1y ago
I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext. Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file. I'm all in favour of storing logs in an indexed structured format but journald ain't it.
- mrweasel 1y ago
  Journald is an odd one. I don't think it being a binary log/database makes sense. If you have a tiny operation, with a single server, then the binary database doesn't really make sense, having plain text is just easier and faster. If you're a bigger operation, you'll have a central logging solution, in which case you need journald to store the longs as plain text as well, before you can do log shipping. The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
  - akdev1l 1y ago
    afaik journald can just forward logs via rsyslog directly to a remote server Why would it need to store plaintext locally?
    - mrweasel 1y ago
      Doesn't that still involve a conversion? I believe that rsyslog can read the journald database, but you're typically not querying syslog data directly, so there's a conversion between rsyslog and logstash, Splunk, Datalog, whatever.
    - NewJazz 1y ago
      Some folks like having some local storage to act as a buffer in case the remote syslog server is down. Not sure if journald can do that on its own.
- bombela 1y ago
  Even doing zcat | grep is faster than journald. I now turn off journald and use rotated pain text log files. It's more efficient in all metrics.
jmclnx 1y ago
systemd has been a success for 2 groups, Windows admins and Fortune 500 Companies. For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though. For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.
- rootnod3 1y ago
  DRM is almost there. Try using Netflix on a *BSD, you can't. The DRM Netflix uses is not supported on BSDs.
plantain 1y ago
journald is such a disaster. How is it possible to be so much slower than even gzipped plain text.
amoe_ 1y ago
I was pro-systemd at the time of the controversial discussions. I still think it's a net positive relative to what was there before. But personally speaking, it's only the core of the software (service management) that improved things for me. The other things (timers, journald) I either ignore or don't like, but perhaps they're useful for distributions.
- msgodel 1y ago
  I think if it was just service management most people wouldn't mind it nearly as much.
- ahartmetz 1y ago
  Yes, systemd's weirdly bundled system services largely suck. The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) and resolvd has or used to have basic bugs that other systems don't have.
  - Ferret7446 1y ago
    > The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) That sounds like an unequal comparison. For the same amount of log data and the exact same filter operation, journald should be strictly faster or equal to text logs. Are you sure this isn't because in your journald test you're going through a lot more logs? Because journald makes it easier/feasible to collect more logs.
    - ahartmetz 1y ago
      No, journald shouldn't be strictly faster. It signs/encrypts/whatever the logs and stores them in some kind of structured format that needs to be converted to plaintext before processing it as such. That is not faster than just reading the plaintext. Grep and less search etc are very fast. And no, I don't want to use the structured features. For error diagnosis, I want every occurrence of the search string in every field over a long time for comparison purposes.
mindslight 1y ago
> i used to think that systemd was made the default and adopted by most distros because of its ease of use and the fact it supplied a whole bunch of things in one suite and i see where the appeal is in that but after switching to artix openrc, im just lost on why they decided to use systemd when openrc is objectively better when it comes to being an init system and for managing services, and all the other components of systemd suite can just be replaced, like why would they do this? I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on. It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns. ... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it. Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different. I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems. (Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)
christophilus 1y ago
I like systemd, but do agree that it's clunky in a lot of places. I'd be interested in hearing from anyone who has used Obarun or its 66suite.
- LargoLasskhyfv 1y ago
  You can have S66 with AntiX too. Just use the 'Init Diversity Edition' (Respin), and pick it from several other options. AFAIK the author of S66 directly collaborated with Antix, to make it perfect.
guenthert 1y ago
https://www.cvedetails.com/vendor/15978/Systemd-Project.html https://www.cvedetails.com/vendor/15978/Systemd-Project.html https://www.theregister.com/2017/07/28/black_hat_pwnie_awards/ https://www.theregister.com/2017/07/28/black_hat_pwnie_award...
johnea 1y ago
I'm sure it's a success for someone 8-/ Mostly IBM. It's certainly been a successful takeover of all of the userspace runtime by one giant integrated set of executables. Pwned off as an init replacement, then feature creeping it's way into every nook of userspace. For me, it's still a disaster for distro and system s/w independence and the "do one thing well" philosophy. While many welcome our new system overlords, the s/w is still not liked by many many people... Move into the void! The DJB way and runit await...
pragmatic 1y ago
Loonix users: We hate windows Loonix developers: Now Loonix is windows Loonix users: booo … several hours later Loonix users: yaaaay /s I remember running Linux servers for the first time and couldn’t believe there wasn’t a straight forward way to get something to run on startup. Ask glue and duct tape and non portable. Don’t get me started on the Python situation. I assume it’s much better now. With the clown services U hadn’t had to run bare Linux in a long time.
suspended_state 1y ago
I don't know what to think about Systemd. I think however that a lot of hate comes from: - a very opaque structure: it's actually not that opaque, since configuration files are still plain text, and better yet with more structure, but it first appears as inscrutable, since the programs aren't shell scripts; - a set of new tools to learn: Systemd doesn't make use of the existing Unix tool-set (sed, awk, etc), ie. the vocabulary most sysadmins are familiar with in this ecosystem. It seems on the outset that Systemd is trying to get away from the traditional Unix "one program should do one and only one thing well" (which is colloquial phrasing for separation of concerns). Still, one idea that occurred to me is that in the early days of Unix, system programs might have been quite simple, with very few options (that man pages were probably very short), but that's clearly not the case nowadays. So haven't we left the "one program should do one and only one thing well" paradigm a long time ago anyway? Actually, I think that one crucial issue with systemd is its specialization: can it be used for anything else than service management? At first, this seems to clash with the Unix principle, but even awk and sed are meant to do one thing, yet their field of action is very general, not just confined to handle one style of files or perform one style of transformation.

106 points by Mond_ 1y ago

These were pretty much my feelings back when I envied launchd and was frustrated with some upstart behaviour. Systemd turned out better than both.

spicyusername 1y ago

I remember when I first upgraded a system to a version using it. This was probably a RHEL 6 -> 7 update back in 2014 or so. It made so many common things I had to do so as a sysadmin super straightforward.

freedomben 1y ago
Same, that was my first experience as well. It was a breath of fresh air in many ways, and after my initial learning of the new syntax, I quickly became a convert. That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
- SAI_Peregrinus 1y ago
  I think the biggest problem with systemd is that they named the main init service manager systemd and also named the organization systemd! It'd be like if GNU named their initrd "GNU". Their other tools just have systemd prepended, e.g. systemd-journald, much like GNU Make or GNU libc (glibc) or similar. The "scope creep" issue doesn't get complained about for GNU because it's clear they're an organization making a bunch of tools for different purposes. Systemd-the-organization is the same, but the naming makes it unclear that many of these tools are not depended on by systemd-the-init-system.
Propelloni 1y ago
And some things got super hard to do, e.g. sequence. I guess it is sorted now, but either way, I wouldn't want to go back.

jmogly 1y ago

Agree wholeheartedly.

up2isomorphism 1y ago

There are numerous distributions not using systemd. So adding “complete, utter, unmitigated “ in front of success seems far-fetched.

freedomben 1y ago
The whole post does have a somewhat hyperbolic tone to it. Personally, I agree with the content and analysis, but I could see the language could feel a bit off-putting, especially given the past flame wars over systemd
- jhickok 1y ago
  It's a lighthearted post, so probably not worth tone policing.
  - freedomben 1y ago
    Agreed, I didn't mean to tone police, just acknowledging it might be off-putting to some people who got wrapped up in the battles back in the day. I wouldn't suggest the author make any changes
  - up2isomorphism 1y ago
    It has nothing to do with the tone, it is just a fact. So there is nothing to police about.

rootnod3 1y ago

I wouldn’t call it a success it all. Almost managed to do was moving or forcing other software to get locked in. GDM, Podman, Pipewire…just to name a few.

fsflover 1y ago
As another comment says, it's a success in the same way Windows has been a success.
NewJazz 1y ago
Gnome is pretty guilty of adding hard dependencies on systemd, but podman and pipewire have worked fine for me without systemd... Not aware of any tight integration there that is mandatory (obviously quadlets, but you don't have to use that feature)

iambvk 1y ago

I am not a sysadmin -- so I keep forgetting the command to see logs cause I only need it once every 3 months. So, systemd change is a pain for me.

slashdev 1y ago
AI is so helpful for doing those kinds of tasks in the terminal that you only do a couple times a year and can never remember the exact incantation
- hundchenkatze 1y ago
  You only need to look those things up once, put it in a snippets.txt file, and then search in the file when you need it.
  - slashdev 1y ago
    I keep a per project file like that with common commands and related things.
hearsathought 1y ago
Keep a file with a list of commands used semi-regularly. Or have a script dump your command history into a file. Better to spend a minute searching for a command in a file than to waste an hour recreating it.
akdev1l 1y ago
use a shell function/alia? show-logs() { journalctl -u "$1" } show-logs nginx (Not that I think journalctl -u even needs an alias tbh but if you want it you can have it)

haiku2077 1y ago

I've worked as a sysadmin/devops for a double digit number of years. Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.

thyristan 1y ago
And made a number of new ones appear. E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd. Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations. Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse. Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons. I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
- idoubtit 1y ago
  The grand-parent post was about the general consensus: sysadmins generally like systemd. So I'm not sure answering individual concerns is useful, but I'll take the bait... > now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd. No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng. And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files. > Boot and shutdown has previously been deterministic, now things just randomly hang. That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either. > Network device naming is also a fun one. The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.

bcoates 1y ago

Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example). You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.

inetknght 1y ago
For debugging my desktop: journalctl --follow --tail --no-trunc -b 0 For anything else: export to a syslog server, which basically any tool that matters will support in some fashion.
- LargoLasskhyfv 1y ago
  journalctl: unrecognized option '--tail'
NewJazz 1y ago
Cloudwatch fucking sucks. Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector) Even ChromeOS uses it, even on devices that still use Upstart.
- bigstrat2003 1y ago
  You're being downvoted but you're absolutely right. The fact that Cloudwatch doesn't support journald is a major, major fail on AWS' part. It's not like this is new or obscure software.
  - NewJazz 1y ago
    I'm just surprised anyone wants to use cloudwatch when they don't need to. It is expensive, and far from the best observability platform.
    - dontlaugh 1y ago
      It's always there by default. That's the only reason I've ever seen it used.

shrubble 1y ago

My view is pragmatic: if you are managing the server and I am a user, then use systemd if you want to. If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.

hungryhobbit 1y ago
FreeBSD is many great things, but ... a pragmatic choice?
- linksnapzz 1y ago
  Yes; it does work nicely in many circumstances.
- shrubble 1y ago
  It works for many, including Netflix. The jails are better integrated as a lightweight container system than Docker is on Linux, IMHO.
- burnt-resistor 1y ago
  And powers zillions of unsung, critical infrastructure tasks.

pram 1y ago

I personally love the init/service/unit-file portion of systemd and have few complaints. It has a lot of powerful features for security, cron, etc that are very simple to use. I also really like journald. The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.

thyristan 1y ago
And those ancillary managers are usually half-baked or outright broken and dangerous, in that they only implement half of the stuff they are trying to replace. E.g. https://github.com/systemd/systemd/issues/25676 https://github.com/systemd/systemd/issues/25676
bryanlarsen 1y ago
Hard disagree. Name resolution feels like a solved problem, so why fix it if it isn't broken? But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better. I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved. Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
- msgodel 1y ago
  I think even for new people the systemd configs are much more complex (and the APIs are certainly much more complex) than the libc ones which worked well (often better.) The problem is that almost nothing in Systemd shares theory with the rest of the OS. It's entirely its own island of new ideas, commands, and configuration language that you have to learn and memorize.
- const_cast 1y ago
  The older init systems were incredibly frail and often did not work. The reason is because Bash, as much as we all love it, is actually super mega ass. It's very bug-prone and actually one of the least portable languages ever invented, in practice. Because you can't, or shouldn't, call to any other binary from within Bash, otherwise you risk your portability. Now your script won't work the same on other platforms, or even the same platform with slightly different configuration. But um... yeah... calling to other binaries is the entire ethos of Bash. It's a shell, that's supposed to be all it does. Even getting a "true" value should be a call to another binary. Unit files, despite being a bespoke format for one singular program, are actually more portable. Like, WAY more portable.

artooro 1y ago

I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.

haiku2077 1y ago
+1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
- SEJeff 1y ago
  Or if the service didn't support pam_limits because it was legacy trash, you had to hack something into the initscript like `ulimit -n XYZ` and restart it. Now things like this are trivial and easy to solve. Using systemd makes large scale Linux systems administration much easier. Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in. Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.

tylerjl 1y ago

Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!

rootnod3 1y ago
This is a nit-pick, but if you picked the title that way to BE hyperbolic, I am not sure if that is the best thing to do for a technical discussion. It sets an internal opinion before even clicking on it. It subconsciously makes people go into the article with an already preconceived notion.

pessimizer 1y ago

No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product. They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now. Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.

Analemma_ 1y ago
> Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product. Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
- jmclnx 1y ago
  >Who exactly was going to write the systemd alternative? And that quote proves 100% what is wrong with Linux these days. In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more. That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch. Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then. Seems these days we are dealing with some corporate behemoth drowning in red tape.
- 1y ago

fsflover 1y ago

> The unix philosophy cries out: is this the end of Linux (or, as many are calling it, GNU plus Linux)? I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence: https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd https://news.ycombinator.com/item?id=42918448 https://news.ycombinator.com/item?id=42918448 https://news.ycombinator.com/item?id=42889792 https://news.ycombinator.com/item?id=42889792 (I reposted my comment from previous submission)

scraptor 1y ago

I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext. Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file. I'm all in favour of storing logs in an indexed structured format but journald ain't it.

mrweasel 1y ago
Journald is an odd one. I don't think it being a binary log/database makes sense. If you have a tiny operation, with a single server, then the binary database doesn't really make sense, having plain text is just easier and faster. If you're a bigger operation, you'll have a central logging solution, in which case you need journald to store the longs as plain text as well, before you can do log shipping. The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
- akdev1l 1y ago
  afaik journald can just forward logs via rsyslog directly to a remote server Why would it need to store plaintext locally?
  - mrweasel 1y ago
    Doesn't that still involve a conversion? I believe that rsyslog can read the journald database, but you're typically not querying syslog data directly, so there's a conversion between rsyslog and logstash, Splunk, Datalog, whatever.
  - NewJazz 1y ago
    Some folks like having some local storage to act as a buffer in case the remote syslog server is down. Not sure if journald can do that on its own.
bombela 1y ago
Even doing zcat | grep is faster than journald. I now turn off journald and use rotated pain text log files. It's more efficient in all metrics.

jmclnx 1y ago

systemd has been a success for 2 groups, Windows admins and Fortune 500 Companies. For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though. For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.

rootnod3 1y ago
DRM is almost there. Try using Netflix on a *BSD, you can't. The DRM Netflix uses is not supported on BSDs.

plantain 1y ago

journald is such a disaster. How is it possible to be so much slower than even gzipped plain text.

amoe_ 1y ago

I was pro-systemd at the time of the controversial discussions. I still think it's a net positive relative to what was there before. But personally speaking, it's only the core of the software (service management) that improved things for me. The other things (timers, journald) I either ignore or don't like, but perhaps they're useful for distributions.

msgodel 1y ago
I think if it was just service management most people wouldn't mind it nearly as much.
ahartmetz 1y ago
Yes, systemd's weirdly bundled system services largely suck. The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) and resolvd has or used to have basic bugs that other systems don't have.
- Ferret7446 1y ago
  > The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) That sounds like an unequal comparison. For the same amount of log data and the exact same filter operation, journald should be strictly faster or equal to text logs. Are you sure this isn't because in your journald test you're going through a lot more logs? Because journald makes it easier/feasible to collect more logs.
  - ahartmetz 1y ago
    No, journald shouldn't be strictly faster. It signs/encrypts/whatever the logs and stores them in some kind of structured format that needs to be converted to plaintext before processing it as such. That is not faster than just reading the plaintext. Grep and less search etc are very fast. And no, I don't want to use the structured features. For error diagnosis, I want every occurrence of the search string in every field over a long time for comparison purposes.

mindslight 1y ago

> i used to think that systemd was made the default and adopted by most distros because of its ease of use and the fact it supplied a whole bunch of things in one suite and i see where the appeal is in that but after switching to artix openrc, im just lost on why they decided to use systemd when openrc is objectively better when it comes to being an init system and for managing services, and all the other components of systemd suite can just be replaced, like why would they do this? I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on. It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns. ... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it. Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different. I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems. (Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)

christophilus 1y ago

I like systemd, but do agree that it's clunky in a lot of places. I'd be interested in hearing from anyone who has used Obarun or its 66suite.

LargoLasskhyfv 1y ago
You can have S66 with AntiX too. Just use the 'Init Diversity Edition' (Respin), and pick it from several other options. AFAIK the author of S66 directly collaborated with Antix, to make it perfect.

guenthert 1y ago

https://www.cvedetails.com/vendor/15978/Systemd-Project.html https://www.cvedetails.com/vendor/15978/Systemd-Project.html https://www.theregister.com/2017/07/28/black_hat_pwnie_awards/ https://www.theregister.com/2017/07/28/black_hat_pwnie_award...

johnea 1y ago

I'm sure it's a success for someone 8-/ Mostly IBM. It's certainly been a successful takeover of all of the userspace runtime by one giant integrated set of executables. Pwned off as an init replacement, then feature creeping it's way into every nook of userspace. For me, it's still a disaster for distro and system s/w independence and the "do one thing well" philosophy. While many welcome our new system overlords, the s/w is still not liked by many many people... Move into the void! The DJB way and runit await...

pragmatic 1y ago

Loonix users: We hate windows Loonix developers: Now Loonix is windows Loonix users: booo … several hours later Loonix users: yaaaay /s I remember running Linux servers for the first time and couldn’t believe there wasn’t a straight forward way to get something to run on startup. Ask glue and duct tape and non portable. Don’t get me started on the Python situation. I assume it’s much better now. With the clown services U hadn’t had to run bare Linux in a long time.

suspended_state 1y ago

I don't know what to think about Systemd. I think however that a lot of hate comes from: - a very opaque structure: it's actually not that opaque, since configuration files are still plain text, and better yet with more structure, but it first appears as inscrutable, since the programs aren't shell scripts; - a set of new tools to learn: Systemd doesn't make use of the existing Unix tool-set (sed, awk, etc), ie. the vocabulary most sysadmins are familiar with in this ecosystem. It seems on the outset that Systemd is trying to get away from the traditional Unix "one program should do one and only one thing well" (which is colloquial phrasing for separation of concerns). Still, one idea that occurred to me is that in the early days of Unix, system programs might have been quite simple, with very few options (that man pages were probably very short), but that's clearly not the case nowadays. So haven't we left the "one program should do one and only one thing well" paradigm a long time ago anyway? Actually, I think that one crucial issue with systemd is its specialization: can it be used for anything else than service management? At first, this seems to clash with the Unix principle, but even awk and sed are meant to do one thing, yet their field of action is very general, not just confined to handle one style of files or perform one style of transformation.