7 ms·

For people who had well working low maintenence environments, systemd came in and changed everything - breaking things, requiring changes to get things working

For people who had well working low maintenence environments, systemd came in and changed everything - breaking things, requiring changes to get things working again. Its not just breaking init.d scripts, it’s ntp, dns, syslog. Systems throughout the OS fail to things that were no longer short commands with muscle memeory, there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf Even when you remember and type that in, you don’t get a simple list of nameserver and host, you get 100 lines of text you have to spend effort parsing to work out what’s going on. When it’s less mental effort to run tcpdump port 53 to see where your DNS is going, there’s a problem. For decades it was /etc/init.d/myserice restart Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer. Or the restart fails it doesn’t tell you why, it gives you two locations to look for log files about why it might have broken. Init.d scripts didn’t do that. Even if there was something really wrong that log files don’t reveal, running init.d with bash -x allowed easy debugging Systemd came in and changed working processes and systems and gave very little benefit to people with working processes and systems from a operator point of view.

bonzini 5y ago
> Now is it systemctl restart myservice or systemctl myservice restart Actually long before that it was service restart myservice, which still works.
- morelisp 5y ago
  Unfortunately, it was `service myservice restart`, which is the wrong order if you want to be able to do multiple things at once (or even just align with any other `cmd subcmd args` program).
  - iso1210 5y ago
    service myserver restart /etc/init.d/myserver restart Seems the traditional order to me $ service myserver status $ service myserver reload $ service myserver restart makes perfect sense when tackling a single service
    - morelisp 5y ago
      The traditional order is more like # /etc/init.d/myserver reload unknown command reload # /etc/init.d/myserver restart unknown command restart # /etc/init.d/myserver stop myserver: warning: frobulator did not fully unfurl # /etc/init.d/myserver start myserver: still running on pid 1234 # kill -TERM 1234 kill: (1234) - No such process # grep "running on pid" /etc/init.d/myserver echo "$0: still running on pid `cat /var/run/myserver.pid`" # rm -f /var/run/myserver.pid # /etc/init.d/myserver start What you're calling "traditional" is the minor, purely UI-facade-level consistency we got after 10 years trying to clean that mess up. But actually we eventually gave up, and went on to invent something better.
irl_ 5y ago
I think you're missing a few things here: * Services started with sysvinit would put logs where they want, which is fine if you know where they are but per-service you might be guessing. Having everything always in the same place is handy. * sysvinit wasn't giving you any of these security benefits. * If your system really was working fine before, why did you need to upgrade it to a newer distribution with systemd?
- turminal 5y ago
  > * If your system really was working fine before, why did you need to upgrade it to a newer distribution with systemd? Because the only two alternatives are 1) running mainstream distros from before 2014 or 2) running obscure distros that still don't use systemd.
  - jbverschoor 5y ago
    Or you could've actually improved init.d and add all the features to is, and make easy to use/create.
    - iso1210 5y ago
      I suspect many sysadmins, certainly the ones who complain about syslog, were quite happy with init.d Traditionally developers would write code and packagers would package them into a distro specific rpm/deb -- including the init scripts (which may be pushed back upstream). Developers wanted to bypass this slow process, and there were far more developers than people willing to package the software. Personally I've never had a problem writing an init.d script.
      - lanstin 5y ago
        All I ever could do was add a line to rc.local that started a supervisor process to run my stuff. But I have many service unit’s as part of user-data.sh or cloud init and they work and restart on failure and are visible to other admins I haven’t talked to etc. it was surprisingly easy.
kbumsik 5y ago
I sometimes feel a bit weird when I see complaints like this. Because I have never experienced broken by systemd. For logging, can't you write a systemd service in bash with -x flag?
- zeeZ 5y ago
  Recently a systemd update that came with a flatcar linux update added/enabled a systemd-resolved stub that hogged port 53, preventing the actual DNS server on the machine from starting up. Does that count as broken by systemd?
  - Tom4hawk 5y ago
    No. Sounds like broken distribution.
jbverschoor 5y ago
You do know that resolving something involves other things than a simple udp query to port 53?
- 5e92cb50239222b 5y ago
  Here's a decent overview if anyone is interested in details. https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-part-i/ https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...
deleted 5y ago
[deleted]
nmg 5y ago
> Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer. I felt so alone in the world until this moment
- theranger 5y ago
  Well, by the time the systemd gets as old as sysvinit currently is, everybody will have the correct one in their muscle memory.
  - GordonS 5y ago
    I agree with the GP in that I can never remember the order of args for systemctl either. But I also accept that systemd is here to stay (and I like it, especially for writing services, and across distros!), and I'll get used to it eventually. For example, I managed to get over `ifconfig -> ip addr` fairly quicker, even if it was really annoying at the start!
idoubtit 5y ago
I doubt answering rants is useful, but I'll try to give factual counter-arguments. > there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf systemd-resolved is not enabled by default in Debian and many distributions, and it is not needed in any way by systemd. If you don't like it, don't use it! Your rant does not sound very serious. Did you really have "ntp" or "syslog" in your muscle memory? That's strange, because most syslog daemon did not have a `syslog` command. Anyway, systemd-resolved was created because it has uses. And for systems that used a dns cache (dnsmasq, etc), rejoice, because the config is now simpler than it was. > For decades it was /etc/init.d/myserice restart > Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer. Before systemd, at least on Debian, for a few years the recommended way was NOT calling `/etc/init.d/something`, but instead `service apache restart`. Since sysv was unsuitable for many uses, several alternatives emerged, like "runit", or "upstart" for Ubuntu. So, before systemd, the recommended way changed with the distribution. Thanks to systemd, most linux installs now use `systemctl restart service1 service2`. Note that you can now act on multiple services at the same time. You can use this feature as a mnemonic. > Or the restart fails it doesn’t tell you why, ... Init.d scripts didn’t do that. In many cases, init.d scripts told you nothing when they failed. Each service has its own procedure. Nowdays you can always see what happened with the command systemctl prints on failure. And `systemctl cat s1` display starting instructions that are rarely longer than a dozen of lines. I remember init.d scripts that were hundreds of lines long, and awfully hard to understand.
- loudmax 5y ago
  I tend to agree with most of the points you're making but I do want to point out that "If you don't like it, don't use it" isn't helpful advice for system administrators who weren't given a choice in the matter. It's fine if you're designing a system from the ground up, but most of the time, most of us have to work with what we're given. I think most of the resentment against systemd is that it felt like it was forced down our throats. Systemd may be an improvement over the mess of shell scripts we had before but it isn't perfect. It's one thing to choose to give up decades of experience so one can voluntarily switch to a new system. It feels very difference when the choice isn't one's own.
darthrupert 5y ago
I used Linux about a decade before systemd came along, and I welcome it. I think it made almost everything better, and not by a small margin.
pjmlp 5y ago
I only used Linux mostly during university, while at CERN and only after most UNIXes gave up fighting against it. From what I remember, everyone else on UNIX world was already following up on systemd like systems before it got adopted on Linux world, and on Red-Hat/SuSE distributions daemon scripts existed for quite a long time as well, so no it wasn't * /etc/init.d/myserice restart* for decades.
viraptor 5y ago
> Or the restart fails it doesn’t tell you why, it gives you two locations to look for log files about why it might have broken. Init.d scripts didn’t do that. Some things are a matter of preference, but this bit is just wrong. Init.d was hilariously worse. Some services had their own configuration locations, some had those exposed via /etc/defaults, some used syslog, some redirected stdout/stderr, some redirected one and discarded the other. You're right that there are two places now - logs are either in the journal or in app-specific log location. And stdout/err go to the journal. Those 2 places mean fewer places to look through than we had before.
rossmohax 5y ago
> For decades it was /etc/init.d/myserice restart Thing is, it was never this command, it was always https://linux.die.net/man/8/service https://linux.die.net/man/8/service , but your command also worked in 99% of the times, until it didn't and restarted service misbehave. Systemd streamlined whole experience.
pmoriarty 5y ago
"Systems throughout the OS fail to things that were no longer short commands with muscle memeory, there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf" As a sysadmin, for me things like that were very, very minor issues. The main problem was that systemd had awful documentation, written by people who'd clearly never had to use systemd in anger and just assumed that everything would work swimmingly (and please don't say read the man pages.. those are barely adequate). When things broke there were no simple and obvious ways to fix it, you had to dive in to its labyrinthine spaghetti architecture and hope and prayed you somehow got the Rube Goldberg machine to work. Hopefully that's improved by now, and there's some canonical documentation that really shows you how it all fits together and how to fix it when it falls apart.
- yawaramin 5y ago
  The documentation (man pages) is still pretty bad. It's always a guessing game for me where to look up the config options: systemd.exec, or systemd.service, or systemd.limits, etc.
rep_movsd 5y ago
If you are using UNIX like environments, you are forced to remember arcane commands - after almost 25 years of using Linux, I still have to open the man page for something as ubiquitous as grep - because there's so many options and you are always finding new use cases. The best solution is to maintain infinite bash history and stop trying to remember arcane stuff. Don't even get me started on tools like jq - so convoluted that I use python instead
- lanstin 5y ago
  It took like six attempts but I finally like jq. Very handy for one liners or quick bash scripts that read data from AWS cli output. It yeah in general prefer to remember the search terms for the working examples. I Google “Awk add total” at least one a month.
cesarb 5y ago
> For decades it was /etc/init.d/myserice restart You should always use "service myservice start" instead of "/etc/init.d/myservice start". Running "/etc/init.d/myservice start" directly means the service ends up accidentally inheriting parts of your shell's state (environment variables, limits, current directory), which is a different environment from when the service is started at boot. The "service" command carefully cleans up most of the environment before running the script, making it much more similar to what will happen when it starts automatically on next boot. And if you were used to "service myservice start", it now automatically forwards to "systemctl start myservice" when the /etc/init.d/myservice script does not exist, so it keeps working nearly the same after the transition to systemd.
knorker 5y ago
Oh, you expect the error message when you restart? Oh, sweet summer child. You won't get that error message until someone connects to the unix socket and the daemon lazily gets started, and crashes because you had a typo in your config file. You didn't know that, because you were under the understanding that the "restart" succeeded, since it said it did and returned EXIT_SUCCESS.
sqqqqrly 5y ago
Tab tab...
lanstin 5y ago
The key Unix skill isn’t typing cat /etc/resolv.conf but strace -f -ff -o /tmp/1 -p 33333 to figure out how your DNS resolution is occurring. Maybe it is the traditional way, maybe some code is configured to use a network placed resolver. You can find out.

bonzini 5y ago

> Now is it systemctl restart myservice or systemctl myservice restart Actually long before that it was service restart myservice, which still works.

morelisp 5y ago
Unfortunately, it was `service myservice restart`, which is the wrong order if you want to be able to do multiple things at once (or even just align with any other `cmd subcmd args` program).
- iso1210 5y ago
  service myserver restart /etc/init.d/myserver restart Seems the traditional order to me $ service myserver status $ service myserver reload $ service myserver restart makes perfect sense when tackling a single service
  - morelisp 5y ago
    The traditional order is more like # /etc/init.d/myserver reload unknown command reload # /etc/init.d/myserver restart unknown command restart # /etc/init.d/myserver stop myserver: warning: frobulator did not fully unfurl # /etc/init.d/myserver start myserver: still running on pid 1234 # kill -TERM 1234 kill: (1234) - No such process # grep "running on pid" /etc/init.d/myserver echo "$0: still running on pid `cat /var/run/myserver.pid`" # rm -f /var/run/myserver.pid # /etc/init.d/myserver start What you're calling "traditional" is the minor, purely UI-facade-level consistency we got after 10 years trying to clean that mess up. But actually we eventually gave up, and went on to invent something better.

irl_ 5y ago

I think you're missing a few things here: * Services started with sysvinit would put logs where they want, which is fine if you know where they are but per-service you might be guessing. Having everything always in the same place is handy. * sysvinit wasn't giving you any of these security benefits. * If your system really was working fine before, why did you need to upgrade it to a newer distribution with systemd?

turminal 5y ago
> * If your system really was working fine before, why did you need to upgrade it to a newer distribution with systemd? Because the only two alternatives are 1) running mainstream distros from before 2014 or 2) running obscure distros that still don't use systemd.
- jbverschoor 5y ago
  Or you could've actually improved init.d and add all the features to is, and make easy to use/create.
  - iso1210 5y ago
    I suspect many sysadmins, certainly the ones who complain about syslog, were quite happy with init.d Traditionally developers would write code and packagers would package them into a distro specific rpm/deb -- including the init scripts (which may be pushed back upstream). Developers wanted to bypass this slow process, and there were far more developers than people willing to package the software. Personally I've never had a problem writing an init.d script.
    - lanstin 5y ago
      All I ever could do was add a line to rc.local that started a supervisor process to run my stuff. But I have many service unit’s as part of user-data.sh or cloud init and they work and restart on failure and are visible to other admins I haven’t talked to etc. it was surprisingly easy.

kbumsik 5y ago

I sometimes feel a bit weird when I see complaints like this. Because I have never experienced broken by systemd. For logging, can't you write a systemd service in bash with -x flag?

zeeZ 5y ago
Recently a systemd update that came with a flatcar linux update added/enabled a systemd-resolved stub that hogged port 53, preventing the actual DNS server on the machine from starting up. Does that count as broken by systemd?
- Tom4hawk 5y ago
  No. Sounds like broken distribution.

jbverschoor 5y ago

You do know that resolving something involves other things than a simple udp query to port 53?

5e92cb50239222b 5y ago
Here's a decent overview if anyone is interested in details. https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-part-i/ https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...

deleted 5y ago

[deleted]

nmg 5y ago

> Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer. I felt so alone in the world until this moment

theranger 5y ago
Well, by the time the systemd gets as old as sysvinit currently is, everybody will have the correct one in their muscle memory.
- GordonS 5y ago
  I agree with the GP in that I can never remember the order of args for systemctl either. But I also accept that systemd is here to stay (and I like it, especially for writing services, and across distros!), and I'll get used to it eventually. For example, I managed to get over `ifconfig -> ip addr` fairly quicker, even if it was really annoying at the start!

idoubtit 5y ago

I doubt answering rants is useful, but I'll try to give factual counter-arguments. > there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf systemd-resolved is not enabled by default in Debian and many distributions, and it is not needed in any way by systemd. If you don't like it, don't use it! Your rant does not sound very serious. Did you really have "ntp" or "syslog" in your muscle memory? That's strange, because most syslog daemon did not have a `syslog` command. Anyway, systemd-resolved was created because it has uses. And for systems that used a dns cache (dnsmasq, etc), rejoice, because the config is now simpler than it was. > For decades it was /etc/init.d/myserice restart > Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer. Before systemd, at least on Debian, for a few years the recommended way was NOT calling `/etc/init.d/something`, but instead `service apache restart`. Since sysv was unsuitable for many uses, several alternatives emerged, like "runit", or "upstart" for Ubuntu. So, before systemd, the recommended way changed with the distribution. Thanks to systemd, most linux installs now use `systemctl restart service1 service2`. Note that you can now act on multiple services at the same time. You can use this feature as a mnemonic. > Or the restart fails it doesn’t tell you why, ... Init.d scripts didn’t do that. In many cases, init.d scripts told you nothing when they failed. Each service has its own procedure. Nowdays you can always see what happened with the command systemctl prints on failure. And `systemctl cat s1` display starting instructions that are rarely longer than a dozen of lines. I remember init.d scripts that were hundreds of lines long, and awfully hard to understand.

loudmax 5y ago
I tend to agree with most of the points you're making but I do want to point out that "If you don't like it, don't use it" isn't helpful advice for system administrators who weren't given a choice in the matter. It's fine if you're designing a system from the ground up, but most of the time, most of us have to work with what we're given. I think most of the resentment against systemd is that it felt like it was forced down our throats. Systemd may be an improvement over the mess of shell scripts we had before but it isn't perfect. It's one thing to choose to give up decades of experience so one can voluntarily switch to a new system. It feels very difference when the choice isn't one's own.

darthrupert 5y ago

I used Linux about a decade before systemd came along, and I welcome it. I think it made almost everything better, and not by a small margin.

pjmlp 5y ago

I only used Linux mostly during university, while at CERN and only after most UNIXes gave up fighting against it. From what I remember, everyone else on UNIX world was already following up on systemd like systems before it got adopted on Linux world, and on Red-Hat/SuSE distributions daemon scripts existed for quite a long time as well, so no it wasn't * /etc/init.d/myserice restart* for decades.

viraptor 5y ago

> Or the restart fails it doesn’t tell you why, it gives you two locations to look for log files about why it might have broken. Init.d scripts didn’t do that. Some things are a matter of preference, but this bit is just wrong. Init.d was hilariously worse. Some services had their own configuration locations, some had those exposed via /etc/defaults, some used syslog, some redirected stdout/stderr, some redirected one and discarded the other. You're right that there are two places now - logs are either in the journal or in app-specific log location. And stdout/err go to the journal. Those 2 places mean fewer places to look through than we had before.

rossmohax 5y ago

> For decades it was /etc/init.d/myserice restart Thing is, it was never this command, it was always https://linux.die.net/man/8/service https://linux.die.net/man/8/service , but your command also worked in 99% of the times, until it didn't and restarted service misbehave. Systemd streamlined whole experience.

pmoriarty 5y ago

"Systems throughout the OS fail to things that were no longer short commands with muscle memeory, there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf" As a sysadmin, for me things like that were very, very minor issues. The main problem was that systemd had awful documentation, written by people who'd clearly never had to use systemd in anger and just assumed that everything would work swimmingly (and please don't say read the man pages.. those are barely adequate). When things broke there were no simple and obvious ways to fix it, you had to dive in to its labyrinthine spaghetti architecture and hope and prayed you somehow got the Rube Goldberg machine to work. Hopefully that's improved by now, and there's some canonical documentation that really shows you how it all fits together and how to fix it when it falls apart.

yawaramin 5y ago
The documentation (man pages) is still pretty bad. It's always a guessing game for me where to look up the config options: systemd.exec, or systemd.service, or systemd.limits, etc.

rep_movsd 5y ago

If you are using UNIX like environments, you are forced to remember arcane commands - after almost 25 years of using Linux, I still have to open the man page for something as ubiquitous as grep - because there's so many options and you are always finding new use cases. The best solution is to maintain infinite bash history and stop trying to remember arcane stuff. Don't even get me started on tools like jq - so convoluted that I use python instead

lanstin 5y ago
It took like six attempts but I finally like jq. Very handy for one liners or quick bash scripts that read data from AWS cli output. It yeah in general prefer to remember the search terms for the working examples. I Google “Awk add total” at least one a month.

cesarb 5y ago

> For decades it was /etc/init.d/myserice restart You should always use "service myservice start" instead of "/etc/init.d/myservice start". Running "/etc/init.d/myservice start" directly means the service ends up accidentally inheriting parts of your shell's state (environment variables, limits, current directory), which is a different environment from when the service is started at boot. The "service" command carefully cleans up most of the environment before running the script, making it much more similar to what will happen when it starts automatically on next boot. And if you were used to "service myservice start", it now automatically forwards to "systemctl start myservice" when the /etc/init.d/myservice script does not exist, so it keeps working nearly the same after the transition to systemd.

knorker 5y ago

Oh, you expect the error message when you restart? Oh, sweet summer child. You won't get that error message until someone connects to the unix socket and the daemon lazily gets started, and crashes because you had a typo in your config file. You didn't know that, because you were under the understanding that the "restart" succeeded, since it said it did and returned EXIT_SUCCESS.

sqqqqrly 5y ago

Tab tab...

lanstin 5y ago

The key Unix skill isn’t typing cat /etc/resolv.conf but strace -f -ff -o /tmp/1 -p 33333 to figure out how your DNS resolution is occurring. Maybe it is the traditional way, maybe some code is configured to use a network placed resolver. You can find out.