6 ms·

A nice story about Unix processes “infecting” each other

164 points by zoidb 5y ago

greyface- 5y ago
Previously: https://news.ycombinator.com/item?id=30281878 https://news.ycombinator.com/item?id=30281878
- deepsun 5y ago
  But this one has more points already :)
  - catach 5y ago
    Karma is a lottery.
- richardfey 5y ago
  Do we already have a Godwin's law for systemd? I am asking because this other short discussion inevitably ended up talking about systemd.
  - MichaelBurge 5y ago
    systemd originated in the article, and was inherited by the comments.
    - macintux 5y ago
      Thank you for providing clever humor is still possible and encouraged here.
z3t4 5y ago
would not want spawned processes to run as fresh root by default.
- asojfdowgh 5y ago
  Gotta start as root to start as any other user, as only root can setuid etc And since a lot of stuff is on lower ports, and there wasn't the complicated firewalls of today to dynamically flick around port routing depending on processes and the such, most programs grabbed the requested ports, and then setuid to the relevant daemon user, all by themselves (also forking, going into background, etc) Thankfully these days its all unneeded, yes
  - noitpmeder 5y ago
    I just started a new job (well, transferred to a new office of my old job), and discovered they are still running everything as root. Thankfully the main office doesn't do this, so the last 5 years of my life have been relatively pain-free, but I'm starting to dread the next 5...
  - tzs 5y ago
    > Gotta start as root to start as any other user, as only root can setuid etc TOPS-10, DEC's main operating system for PDP-10, had an interesting approach to this. The login system call, which was used to set the user ID for a user, was not privileged. However, it would return an error if it was executed when already logged in. As was common with systems back then, there was a CLI that was built in to the OS and ran on all terminals without being logged in. It allowed you to run a restricted set of programs without logging in, including the login program. One of those programs you could run without being logged in was queue, which was a program to view the print queue (and maybe other queues, like the queue of pending tape mounting requests?...I don't remember). Unlike the login program, the queue program was also useful when logged in and so included many features and options beyond just what was needed by people who were not logged in. That included a flag to run other programs: queue/run:foo would run run foo.exe. (No, I don't remember why /run was ever actually useful). Someone figured out that this allowed queue/run:ddt to run the DDT debugger, and that they could do this while not logged in thus ending up with a running DDT. They could then use the debugger to poke in a short assembly program to invoke the login system call and login as anyone they wanted, no password required.
geocrasher 5y ago
Wait, was this just an article that was designed to say systemd isn't all that bad?
- sjansen 5y ago
  As someone who watched this bit if history unfold, and someone who diagnosed similar issues "back in the day", I see it as an example of sitting around the virtual water cooler trading war stories. Sometimes it's nice to look back at the fun bugs we squashed, and be grateful they're not a thing anymore because our systems have improved.
noitpmeder 5y ago
As a related tidbit, I recently learned that, while it's possible to increase one's niceness, it's impossible (with stock settings on most distros) to decrease said niceness without root.
- CrLf 5y ago
  Makes sense, you can always drop privileges but can’t acquire them without elevated capabilities.
c0l0 5y ago
Cool story! I can only imagine how good the person solving the riddle felt when the penny finally dropped! :) Two somewhat topical observations in a similar vein: Back in a sysvinit world, many (Debian) users would run init-scripts directly via `/etc/init.d/someservice someaction`, instead of using `/usr/sbin/service someservice someaction`. This could have very unpleasant side effects at times, because `service` is/was actively trying to clean and sanitize the resulting child processes' environment, while most init scripts themselves usually did not. The second thing that the article made me think of is not directly related to most "admin wants the system to start a service"-scenarios, but COULD also happen in such a situation: On Linux, when a process tries to fork, the kernel requires certain guarantees in terms of usable memory for the process' copy, due to copy-on-write semantics for allocated pages. If these cannot be matched, forking might fail with ENOMEM. In my experience, this is much more likely to happen when the process trying to fork has a lot of allocated memory accounted for. Therefore, it makes perfect sense to have a "lightweight" kind of fork-and-exec-server around in the system that can be contacted via an IPC mechanism, and made to start daemon processes that are probably not meant to inherit anything from the caller's environment (like file descriptors, niceness, environment variables) in the first place.
dale_glass 5y ago
Ah yes, the good old days some people are mysteriously nostalgic about. I've seen quite a few people insist that SysV was perfectly fine and nothing about it needed fixing. But this article is a good example of how that kind of thing actually went: a myriad gotchas, bugs and undesirable characteristics eventually became so entrenched and familiar that knowing about all of this just became part of the sysadmin job. Having an elaborate dance around system upgrades because it can break ssh was a perfectly normal thing. You're just supposed to know that. And init scripts are fine as they are, you're just supposed to know processes inherit stuff, and take that into account. And stray pid files sometimes stop a service from starting, that's also basic knowledge. And rebooting? Nah, we'll keep the machine running for 2 years without a kernel upgrade, because who knows if all the services will come back after a reboot. Better not risk it. And I kinda think I know why some people actually liked that. Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard. The problem is that the competition didn't sit still and eventually Windows and Mac got good enough, and all this stuff became even more of a clunky impediment to getting stuff done. I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation.
- TacticalCoder 5y ago
  > I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation. I, for one, I'm glad there are still options and that even Debian let the door open to, one day, switching to something else than systemd. I'm running Devuan (Chimaera) now and it's fine (for my needs). It's not all rosy: there's something to be said about the insane kitchen sink that systemd is. Having the PID1 so gigantic, with its tentacles everywhere in the system, ain't something the Linux community should be proud of. I'm very happy that, so far, Linus is resisting and nothing in the kernel is (yet?) tied to systemd. Linux is about choices too. The day there ain't no choice anymore, I'm switching to a BSD. I'm running Linus' Linux. Not IBM/Red Hat/Poettering's Linux.
  - dale_glass 5y ago
    PID 1 isn't particularly gigantic under systemd. Most actual work is delegated to other binaries. Eg, journald isn't PID 1. Personally, I take the opposite stance. I have very little ideology besides things being open source. To me a machine is a tool to do a job, and if systemd does the job better, then that's the right tool, regardless of any violation of principles that it might involve. There's definitely good ideas in the Unix model, but if the model ceases to be useful, it's time to throw the bad parts out. I also have no issues whatsoever with running Red Hat's Linux (I use Fedora). It does the job, it provides the tools I like, that's what matters.
iqanq 5y ago
I was expecting something regarding LD_PRELOAD.
deleted 5y ago
[deleted]
temac 5y ago
Propagating the niceness of a process by default is perfectly normal and most OSes do it by default. E.g. Windows propagates IDLE_PRIORITY_CLASS or BELOW_NORMAL_PRIORITY by default. So much of the "Unix processes" theory. Given systemd did not stop to use "Unix processes", that should also have been a clue. The issue was about the postinst script which either missed that it should have done something differently, or is maybe just not designed to be run nicely so the users broke an (implied) contract by calling it that way.
teddyh 5y ago
I remember having the habit of always doing “cd /” before restarting services. This was because, when starting a service manually, this service would similarly inherit the current working directory of the shell, and many services would behave weirdly in various ways. Running a root shell, your current directory was most often either “/root”, or something like “/home/someuser” if you had done “su” or the like. The “/root” directory was unreadable by unprivileged users, and if the service lowered its privileges by changing user id to an unprivileged user, the service was no longer allowed to access its current directory, often leading to weird errors. The “/home/someuser” directory was also a source of errors, since the service might happily be creating weird files there, sometimes owned by the root user. Also, if the “/home/someuser” directory was mounted via NFS, you might get access permission errors if the NFS server had squashroot configured (as it should), but it might also lead to weird behavior when the NFS server itself was restarted or taken down; this seemingly unrelated service would start behaving weirdly, just because its current working directory was a directory served by the NFS server. In short, yeah, life before systemd was hard.
cryptonector 5y ago
Reminds me of a case where adding printers to print spoolers that used HP Jetadmin printer drivers would cause all printers to start spewing garbage on reams and reams of paper. The problem turned out to be that one sysadmin thought that the printer subsystem needed to be restarted after adding a printer queue, and to do it they used `sudo su` (not `sudo su -`) to gain privileges, and they liked a BSD-ish environment, so they had `/usr/ucb` ahead of `/usr/bin` (yes, this was on Solaris). How would that break printing? Well, those HP Jetadmin printer drivers were awful Bourne shell scripts that depended on System V echo behavior to emit escape sequences for PCL, and now you see the problem: /usr/ucb/echo did not do what /usr/bin/echo did, and failed to emit those escape sequences, leading to garbage rather than properly-formed PCL being sent to the printers. SMF (Solaris/Illumos) / systemd (Linux) fix the problem permanently.

164 points by zoidb 5y ago

greyface- 5y ago

Previously: https://news.ycombinator.com/item?id=30281878 https://news.ycombinator.com/item?id=30281878

deepsun 5y ago
But this one has more points already :)
- catach 5y ago
  Karma is a lottery.
richardfey 5y ago
Do we already have a Godwin's law for systemd? I am asking because this other short discussion inevitably ended up talking about systemd.
- MichaelBurge 5y ago
  systemd originated in the article, and was inherited by the comments.
  - macintux 5y ago
    Thank you for providing clever humor is still possible and encouraged here.

z3t4 5y ago

would not want spawned processes to run as fresh root by default.

asojfdowgh 5y ago
Gotta start as root to start as any other user, as only root can setuid etc And since a lot of stuff is on lower ports, and there wasn't the complicated firewalls of today to dynamically flick around port routing depending on processes and the such, most programs grabbed the requested ports, and then setuid to the relevant daemon user, all by themselves (also forking, going into background, etc) Thankfully these days its all unneeded, yes
- noitpmeder 5y ago
  I just started a new job (well, transferred to a new office of my old job), and discovered they are still running everything as root. Thankfully the main office doesn't do this, so the last 5 years of my life have been relatively pain-free, but I'm starting to dread the next 5...
- tzs 5y ago
  > Gotta start as root to start as any other user, as only root can setuid etc TOPS-10, DEC's main operating system for PDP-10, had an interesting approach to this. The login system call, which was used to set the user ID for a user, was not privileged. However, it would return an error if it was executed when already logged in. As was common with systems back then, there was a CLI that was built in to the OS and ran on all terminals without being logged in. It allowed you to run a restricted set of programs without logging in, including the login program. One of those programs you could run without being logged in was queue, which was a program to view the print queue (and maybe other queues, like the queue of pending tape mounting requests?...I don't remember). Unlike the login program, the queue program was also useful when logged in and so included many features and options beyond just what was needed by people who were not logged in. That included a flag to run other programs: queue/run:foo would run run foo.exe. (No, I don't remember why /run was ever actually useful). Someone figured out that this allowed queue/run:ddt to run the DDT debugger, and that they could do this while not logged in thus ending up with a running DDT. They could then use the debugger to poke in a short assembly program to invoke the login system call and login as anyone they wanted, no password required.

geocrasher 5y ago

Wait, was this just an article that was designed to say systemd isn't all that bad?

sjansen 5y ago
As someone who watched this bit if history unfold, and someone who diagnosed similar issues "back in the day", I see it as an example of sitting around the virtual water cooler trading war stories. Sometimes it's nice to look back at the fun bugs we squashed, and be grateful they're not a thing anymore because our systems have improved.

noitpmeder 5y ago

As a related tidbit, I recently learned that, while it's possible to increase one's niceness, it's impossible (with stock settings on most distros) to decrease said niceness without root.

CrLf 5y ago
Makes sense, you can always drop privileges but can’t acquire them without elevated capabilities.

c0l0 5y ago

Cool story! I can only imagine how good the person solving the riddle felt when the penny finally dropped! :) Two somewhat topical observations in a similar vein: Back in a sysvinit world, many (Debian) users would run init-scripts directly via `/etc/init.d/someservice someaction`, instead of using `/usr/sbin/service someservice someaction`. This could have very unpleasant side effects at times, because `service` is/was actively trying to clean and sanitize the resulting child processes' environment, while most init scripts themselves usually did not. The second thing that the article made me think of is not directly related to most "admin wants the system to start a service"-scenarios, but COULD also happen in such a situation: On Linux, when a process tries to fork, the kernel requires certain guarantees in terms of usable memory for the process' copy, due to copy-on-write semantics for allocated pages. If these cannot be matched, forking might fail with ENOMEM. In my experience, this is much more likely to happen when the process trying to fork has a lot of allocated memory accounted for. Therefore, it makes perfect sense to have a "lightweight" kind of fork-and-exec-server around in the system that can be contacted via an IPC mechanism, and made to start daemon processes that are probably not meant to inherit anything from the caller's environment (like file descriptors, niceness, environment variables) in the first place.

dale_glass 5y ago

Ah yes, the good old days some people are mysteriously nostalgic about. I've seen quite a few people insist that SysV was perfectly fine and nothing about it needed fixing. But this article is a good example of how that kind of thing actually went: a myriad gotchas, bugs and undesirable characteristics eventually became so entrenched and familiar that knowing about all of this just became part of the sysadmin job. Having an elaborate dance around system upgrades because it can break ssh was a perfectly normal thing. You're just supposed to know that. And init scripts are fine as they are, you're just supposed to know processes inherit stuff, and take that into account. And stray pid files sometimes stop a service from starting, that's also basic knowledge. And rebooting? Nah, we'll keep the machine running for 2 years without a kernel upgrade, because who knows if all the services will come back after a reboot. Better not risk it. And I kinda think I know why some people actually liked that. Figuring out problems like this when nobody else can and saving the day makes you feel like a wizard. The problem is that the competition didn't sit still and eventually Windows and Mac got good enough, and all this stuff became even more of a clunky impediment to getting stuff done. I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation.

TacticalCoder 5y ago
> I for one am very glad that systemd modernized things a bit and helped remove some of this stuff from the equation. I, for one, I'm glad there are still options and that even Debian let the door open to, one day, switching to something else than systemd. I'm running Devuan (Chimaera) now and it's fine (for my needs). It's not all rosy: there's something to be said about the insane kitchen sink that systemd is. Having the PID1 so gigantic, with its tentacles everywhere in the system, ain't something the Linux community should be proud of. I'm very happy that, so far, Linus is resisting and nothing in the kernel is (yet?) tied to systemd. Linux is about choices too. The day there ain't no choice anymore, I'm switching to a BSD. I'm running Linus' Linux. Not IBM/Red Hat/Poettering's Linux.
- dale_glass 5y ago
  PID 1 isn't particularly gigantic under systemd. Most actual work is delegated to other binaries. Eg, journald isn't PID 1. Personally, I take the opposite stance. I have very little ideology besides things being open source. To me a machine is a tool to do a job, and if systemd does the job better, then that's the right tool, regardless of any violation of principles that it might involve. There's definitely good ideas in the Unix model, but if the model ceases to be useful, it's time to throw the bad parts out. I also have no issues whatsoever with running Red Hat's Linux (I use Fedora). It does the job, it provides the tools I like, that's what matters.

iqanq 5y ago

I was expecting something regarding LD_PRELOAD.

deleted 5y ago

[deleted]

temac 5y ago

Propagating the niceness of a process by default is perfectly normal and most OSes do it by default. E.g. Windows propagates IDLE_PRIORITY_CLASS or BELOW_NORMAL_PRIORITY by default. So much of the "Unix processes" theory. Given systemd did not stop to use "Unix processes", that should also have been a clue. The issue was about the postinst script which either missed that it should have done something differently, or is maybe just not designed to be run nicely so the users broke an (implied) contract by calling it that way.

teddyh 5y ago

I remember having the habit of always doing “cd /” before restarting services. This was because, when starting a service manually, this service would similarly inherit the current working directory of the shell, and many services would behave weirdly in various ways. Running a root shell, your current directory was most often either “/root”, or something like “/home/someuser” if you had done “su” or the like. The “/root” directory was unreadable by unprivileged users, and if the service lowered its privileges by changing user id to an unprivileged user, the service was no longer allowed to access its current directory, often leading to weird errors. The “/home/someuser” directory was also a source of errors, since the service might happily be creating weird files there, sometimes owned by the root user. Also, if the “/home/someuser” directory was mounted via NFS, you might get access permission errors if the NFS server had squashroot configured (as it should), but it might also lead to weird behavior when the NFS server itself was restarted or taken down; this seemingly unrelated service would start behaving weirdly, just because its current working directory was a directory served by the NFS server. In short, yeah, life before systemd was hard.

cryptonector 5y ago

Reminds me of a case where adding printers to print spoolers that used HP Jetadmin printer drivers would cause all printers to start spewing garbage on reams and reams of paper. The problem turned out to be that one sysadmin thought that the printer subsystem needed to be restarted after adding a printer queue, and to do it they used `sudo su` (not `sudo su -`) to gain privileges, and they liked a BSD-ish environment, so they had `/usr/ucb` ahead of `/usr/bin` (yes, this was on Solaris). How would that break printing? Well, those HP Jetadmin printer drivers were awful Bourne shell scripts that depended on System V echo behavior to emit escape sequences for PCL, and now you see the problem: /usr/ucb/echo did not do what /usr/bin/echo did, and failed to emit those escape sequences, leading to garbage rather than properly-formed PCL being sent to the printers. SMF (Solaris/Illumos) / systemd (Linux) fix the problem permanently.