6 ms·

Ah yes, the good old days some people are mysteriously nostalgic about. I've seen quite a few people insist that SysV was perfectly fine and nothing about it n

Ah yes, the good old days some people are mysteriously nostalgic about. I've seen quite a few people insist that SysV was perfectly fine and nothing about it needed fixing. But this article is a good example of how that kind of thing actually went: a myriad gotchas, bugs and undesirable characteristics eventually became so entrenched and familiar that knowing about all of this just became part of the sysadmin job. Having an elaborate dance around system upgrades because it can break ssh was a perfectly normal thing. You're just supposed to know that. And init scripts are fine as they are, you're just supposed to know processes inherit stuff, and take that into account. And stray pid files sometimes stop a service from starting, that's also basic knowledge. And rebooting? Nah, we'll keep the machine running for 2 years without a kernel upgrade, because who knows if all the services will come back after a reboot. Better not risk it. And I kinda think I know why some people actually liked that. Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard. The problem is that the competition didn't sit still and eventually Windows and Mac got good enough, and all this stuff became even more of a clunky impediment to getting stuff done. I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation.

TacticalCoder 5y ago
> I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation. I, for one, I'm glad there are still options and that even Debian let the door open to, one day, switching to something else than systemd. I'm running Devuan (Chimaera) now and it's fine (for my needs). It's not all rosy: there's something to be said about the insane kitchen sink that systemd is. Having the PID1 so gigantic, with its tentacles everywhere in the system, ain't something the Linux community should be proud of. I'm very happy that, so far, Linus is resisting and nothing in the kernel is (yet?) tied to systemd. Linux is about choices too. The day there ain't no choice anymore, I'm switching to a BSD. I'm running Linus' Linux. Not IBM/Red Hat/Poettering's Linux.
- dale_glass 5y ago
  PID 1 isn't particularly gigantic under systemd. Most actual work is delegated to other binaries. Eg, journald isn't PID 1. Personally, I take the opposite stance. I have very little ideology besides things being open source. To me a machine is a tool to do a job, and if systemd does the job better, then that's the right tool, regardless of any violation of principles that it might involve. There's definitely good ideas in the Unix model, but if the model ceases to be useful, it's time to throw the bad parts out. I also have no issues whatsoever with running Red Hat's Linux (I use Fedora). It does the job, it provides the tools I like, that's what matters.
  - trasz 5y ago
    It doesn’t matter if PID 1 isn’t gigantic if it depends on stuff that is.
    - dale_glass 5y ago
      There are legitimate concerns for PID 1 being small and stable, since if that doesn't work you have big issues. The rest doesn't really matter. They're separate processes and many are task specific and not used if you don't need that particular bit of functionality.
sjansen 5y ago
> Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard. You're right. It was an awesome intellectual high each time. But only the first time. Eventually seeing the same stupid problem come back got tedious. Thank goodness our systems continue to evolve. It's so long since I've seen a leaky environment at service startup cause problems that I'd forgotten it was a thing. I don't understand anyone that seems to think UNIX was perfected in 1983 with the release of System V. We've learned so much since then by standing on the shoulders of giants.
- pjmlp 5y ago
  Somehow there is this mysticism that UNIX System V was the best thing since sliced bread, only outperformed by Plan 9. More to the point is how Linux community tends to re-invent poorly or be against stuff that had been already available for ages on commercial UNIXes, the ones that I had more fun using actually.
mindslight 5y ago
The problem with systemd isn't the modernization, it's the utterly ungrokkable bloat caused by not making a clean break from many old customs. Countless .ini files splayed across /lib/systemd and /etc/systemd with symlinks setting additional properties? It (ab)uses the filesystem to create the illusion of being able to grok it with standard file manipulation tools, despite that being impossible. The general concept needs little more than a single dependency graph, and yet they've created like 3+. Services are enabled/disabled by a separate graph of Wants/WantedBy, plus a whole bunch of ad-hoc look-them-up-every-time PartOf/ReloadPropagatedFrom/etc. It's like they did the brainstorming design phase, but never did the design optimization bit where you try to simplify the model. Much of the pain of systemd goes away under NixOS, with the functional/exit (as contrasted with mutable/voice) thing of hiding all the tedious stateful complexity behind persistent objects, and leaving you with just an overly ugly config format. But then you've got to ask why the systemd designers couldn't have created that interface in the first place! One file per service config and one (conceptual) top level file for the dependency graph! Then the traditional distros could manage the complexity of needing to reference every possible service package with their usual include .d schemes, and anybody breaking from that could just augment or fork the toplevel config as usual.
- dale_glass 5y ago
  I'm not familiar with NixOS, so can't argue there. However: /lib/systemd and /etc/systemd make perfect sense to me -- /lib/systemd is what the package installs. /etc/systemd is the admin overriding what's in /lib. That's a perfectly fine solution in that the admin's modifications apply cleanly on top of whatever the package installs, and works nicely with package updates. Installing files into /etc is troublesome. Stuff like PropagatesReloadTo exists because there exist things that are built from multiple cooperating units that need such measures. The vast majority of things don't need anything more than a 5 line service file. One file for the dependency graph? How would that work, and why would that be a good idea? In systemd, the service declares what it needs. Why would I want to manually go mess with a file after installing some third party service to tell the system what that service needs?
  - mindslight 5y ago
icedchai 5y ago
I did some contract work for a company that hadn't rebooted a couple of servers in almost 5 years because they were afraid of them not coming back up. Apparently the only employee who knew anything about that system left. This was an organization bringing in 100's of millions in revenue, not a tiny company.
- bravetraveler 5y ago
  This is a fun self-realizing thing, and I see it all over the place even today! Well, I haven't changed jobs in about 5 years - but there are plenty of indicators of the same problem around. Less the, 'we never reboot' thing but more... 'that person is the only one who understands this'; leaving the problem ripe for creation on any given day. Also a very large business/conglomerate - it's a household name. I'm part of the newer generation in a sense - I picked up Linux and the like in the mid-early 2000s. I was a glutton for punishment. I wanted to see everything break however it could. Eventually I learned quite a bit of these lessons out of sheer interest. It provides a lot of opportunity for the 'wizard' moments OP mentioned. I find a fairly cavalier attitude about things as a result. People at work think I'm crazy, but the longer this is prolonged - the more likely it is to hurt.
Supermancho 5y ago
> Ah yes, the good old days some people are mysteriously nostalgic about. > who knows if all the services will come back after a reboot. Better not risk it. This is not a story about how bad sysV was but about how people happily disregarded any sort of infrastructure rigor "in the good old days". Applying upgrades that would break their system then wrote workarounds on top of others for those who also didn't care to test the builds, was considered reasonable. This isn't considered a good plan today and it wasn't a good plan then.
hibbelig 5y ago
Ah, yes. I was having a discussion with BSD folks about how I was missing the equivalent of "service reload sendmail", and they replied that I have no business managing my own mail server if I don't know I should send SIGHUP to the process. I said that's great but this is my home box and I don't see how it helps me to be an expert in every service that's running there. For large installations, sure, there should be a dedicated email person. But are smaller installations not allowed to run sendmail so that cron can send its messages to some central place?

TacticalCoder 5y ago

> I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation. I, for one, I'm glad there are still options and that even Debian let the door open to, one day, switching to something else than systemd. I'm running Devuan (Chimaera) now and it's fine (for my needs). It's not all rosy: there's something to be said about the insane kitchen sink that systemd is. Having the PID1 so gigantic, with its tentacles everywhere in the system, ain't something the Linux community should be proud of. I'm very happy that, so far, Linus is resisting and nothing in the kernel is (yet?) tied to systemd. Linux is about choices too. The day there ain't no choice anymore, I'm switching to a BSD. I'm running Linus' Linux. Not IBM/Red Hat/Poettering's Linux.

dale_glass 5y ago
PID 1 isn't particularly gigantic under systemd. Most actual work is delegated to other binaries. Eg, journald isn't PID 1. Personally, I take the opposite stance. I have very little ideology besides things being open source. To me a machine is a tool to do a job, and if systemd does the job better, then that's the right tool, regardless of any violation of principles that it might involve. There's definitely good ideas in the Unix model, but if the model ceases to be useful, it's time to throw the bad parts out. I also have no issues whatsoever with running Red Hat's Linux (I use Fedora). It does the job, it provides the tools I like, that's what matters.
- trasz 5y ago
  It doesn’t matter if PID 1 isn’t gigantic if it depends on stuff that is.
  - dale_glass 5y ago
    There are legitimate concerns for PID 1 being small and stable, since if that doesn't work you have big issues. The rest doesn't really matter. They're separate processes and many are task specific and not used if you don't need that particular bit of functionality.

sjansen 5y ago

> Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard. You're right. It was an awesome intellectual high each time. But only the first time. Eventually seeing the same stupid problem come back got tedious. Thank goodness our systems continue to evolve. It's so long since I've seen a leaky environment at service startup cause problems that I'd forgotten it was a thing. I don't understand anyone that seems to think UNIX was perfected in 1983 with the release of System V. We've learned so much since then by standing on the shoulders of giants.

pjmlp 5y ago
Somehow there is this mysticism that UNIX System V was the best thing since sliced bread, only outperformed by Plan 9. More to the point is how Linux community tends to re-invent poorly or be against stuff that had been already available for ages on commercial UNIXes, the ones that I had more fun using actually.

mindslight 5y ago

The problem with systemd isn't the modernization, it's the utterly ungrokkable bloat caused by not making a clean break from many old customs. Countless .ini files splayed across /lib/systemd and /etc/systemd with symlinks setting additional properties? It (ab)uses the filesystem to create the illusion of being able to grok it with standard file manipulation tools, despite that being impossible. The general concept needs little more than a single dependency graph, and yet they've created like 3+. Services are enabled/disabled by a separate graph of Wants/WantedBy, plus a whole bunch of ad-hoc look-them-up-every-time PartOf/ReloadPropagatedFrom/etc. It's like they did the brainstorming design phase, but never did the design optimization bit where you try to simplify the model. Much of the pain of systemd goes away under NixOS, with the functional/exit (as contrasted with mutable/voice) thing of hiding all the tedious stateful complexity behind persistent objects, and leaving you with just an overly ugly config format. But then you've got to ask why the systemd designers couldn't have created that interface in the first place! One file per service config and one (conceptual) top level file for the dependency graph! Then the traditional distros could manage the complexity of needing to reference every possible service package with their usual include .d schemes, and anybody breaking from that could just augment or fork the toplevel config as usual.

dale_glass 5y ago
I'm not familiar with NixOS, so can't argue there. However: /lib/systemd and /etc/systemd make perfect sense to me -- /lib/systemd is what the package installs. /etc/systemd is the admin overriding what's in /lib. That's a perfectly fine solution in that the admin's modifications apply cleanly on top of whatever the package installs, and works nicely with package updates. Installing files into /etc is troublesome. Stuff like PropagatesReloadTo exists because there exist things that are built from multiple cooperating units that need such measures. The vast majority of things don't need anything more than a 5 line service file. One file for the dependency graph? How would that work, and why would that be a good idea? In systemd, the service declares what it needs. Why would I want to manually go mess with a file after installing some third party service to tell the system what that service needs?
- mindslight 5y ago

icedchai 5y ago

I did some contract work for a company that hadn't rebooted a couple of servers in almost 5 years because they were afraid of them not coming back up. Apparently the only employee who knew anything about that system left. This was an organization bringing in 100's of millions in revenue, not a tiny company.

bravetraveler 5y ago
This is a fun self-realizing thing, and I see it all over the place even today! Well, I haven't changed jobs in about 5 years - but there are plenty of indicators of the same problem around. Less the, 'we never reboot' thing but more... 'that person is the only one who understands this'; leaving the problem ripe for creation on any given day. Also a very large business/conglomerate - it's a household name. I'm part of the newer generation in a sense - I picked up Linux and the like in the mid-early 2000s. I was a glutton for punishment. I wanted to see everything break however it could. Eventually I learned quite a bit of these lessons out of sheer interest. It provides a lot of opportunity for the 'wizard' moments OP mentioned. I find a fairly cavalier attitude about things as a result. People at work think I'm crazy, but the longer this is prolonged - the more likely it is to hurt.

Supermancho 5y ago

> Ah yes, the good old days some people are mysteriously nostalgic about. > who knows if all the services will come back after a reboot. Better not risk it. This is not a story about how bad sysV was but about how people happily disregarded any sort of infrastructure rigor "in the good old days". Applying upgrades that would break their system then wrote workarounds on top of others for those who also didn't care to test the builds, was considered reasonable. This isn't considered a good plan today and it wasn't a good plan then.

hibbelig 5y ago

Ah, yes. I was having a discussion with BSD folks about how I was missing the equivalent of "service reload sendmail", and they replied that I have no business managing my own mail server if I don't know I should send SIGHUP to the process. I said that's great but this is my home box and I don't see how it helps me to be an expert in every service that's running there. For large installations, sure, there should be a dedicated email person. But are smaller installations not allowed to run sendmail so that cron can send its messages to some central place?