5 ms·

Linux has been at the "it just works" level for decades, where it matters -- on servers and appliances. It's "Linux on the desktop" where apparently boot times

Linux has been at the "it just works" level for decades, where it matters -- on servers and appliances. It's "Linux on the desktop" where apparently boot times are a big deal requiring screwing up the entire operating system for.

rst 12y ago
And not even there, in my experience; my laptop and desktop reboot once every few weeks or longer, usually for kernel upgrades, and even with legacy init, bios-to-login-screen is not the longest part of that process. For interactive machines, systemd is already a complicated solution for a simple non-problem. For servers, it's worse.
cwyers 12y ago
> Linux has been at the "it just works" level for decades, where it matters -- on servers and appliances. Like hell. Anything beyond the most basic tasks on a server (and even the most basic tasks, sometimes) requires configuration, and often that configuration ranges between "nonintuitive" and "actively spiteful." If a service crashes and the pid file doesn't get handled properly, you've got to go in there and fix it. And god forbid you have to touch SELinux for anything. "Just works." I want what you're smoking.
acdha 12y ago
The first few trivial counter-examples which come to mind: * someone adds a storage device. Now your server can't boot because what used to be /dev/sda1 is now /dev/sdc1. Yes, UUIDs partially address that but unless you were defining "decades" as "within the last year or so" you can't assume that they were used. * init script starts something, which crashes / is killed by OOM / etc. and you'd like it to be restarted. Note that this is still unsolved with upstart if the process crashes for more than your respawn limit, as is common with e.g. networking-related startup failures but, hey, it's the same braintrust which required multiple major releases to accept that sysadmins need logging and daemons to be started as users other than root or that "hang forever" is not a valid error handling strategy.
- felixgallo 12y ago
  Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It never even happened with SCSI. And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically.
  - acdha 12y ago
    > Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It's not especially common, particularly if you don't work with large servers or tons of local storage. Here are three variations on this theme which I had to deal with: * The Linux kernel changed the order of device enumeration in a point release (early 2.6, if memory serves). This caused some of our servers to start enumeration the on-board IDE CD-ROM ahead of the SCSI drives and on the next reboot the ones which had been setup before Debian started using UUIDs in /etc/fstab failed because /dev/sda was now the CD drive. * A very similar problem happened involving USB mass storage, which was a problem because our remote administration tools could forward a local drive or ISO image to the remote server as a USB device. This was particularly annoying on one early KVM system which only forwarded the device when you were connected so the logical device ordering changed if you rebooted it while connected to the console. * If you have multiple controllers in a system, the drives are enumerated in order per-controller. If the bays were not fully populated when the system was first setup, a new drive added to controller 1 will be enumerated ahead of the drives from controller 2 and all of the subsequent device names will be shifted by one. All of these are why it's now routine to use UUIDs to identify drives. It's a solvable problem but hasn't been so for even a single decade. > And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. Uptime and resilience are baseline requirements for what I work on, not "rare cases". I've used external supervisors but that's introducing a third-party tool with its own bugs and means that you can't assume that another sysadmin is familiar with it. If you're the monit person in a supervisord shop (or vice versa) you'll probably have something take longer to debug or break at some point because of a subtlety in the differences between the two. This is such a common case that it makes sense to solve it once at the system level, particularly since that allows me to e.g. not have to code an extra bit in my daemon that knows how to ask arbitrary third-party supervisors about a different process. > If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically. That's a fine approach if the failures are caused by something like a broken config file but it doesn't handle environmental failures gracefully. For example, consider a daemon which fails if it can't resolve a hostname or connect to a server (e.g. a database) or which exits if it can't write to disk. In either case, you're going to have downtime if the server / data-center drops offline or a partition fills up. The question is whether the service comes back quickly after the underlying problem is fixed or requires someone to localize it and restart the process. If you have one server and you happen to be on-duty, this isn't a big deal but for anyone with many servers running different apps it can be a headache after a network partition or power failure (oops, web servers all came online well before the database server finished checking its RAID array). In the case of upstart, the default respawn limit is only 10 times within 5 seconds after which it will stop the job until you manually restart it. Since they didn't add throttling or delays, there's no way to deal with this within upstart without either increasing downtime for every error (e.g. adding "|| sleep 10" to every script) or wasting more CPU restarting jobs as quickly as possible until you hit a larger limit. Again, not an unsolvable problem but had they not ignored a huge amount of prior art and perhaps even consulted with actual sysadmins there would be things like a respawn delay and backoff so you could say things like "restart this no more than n times per minute". Instead, most places either convert everything to use a different supervisor or support an external monitor which requests restarts for services periodically, both of which add maintenance overhead and risk for what is a standard feature on other systems.

rst 12y ago

And not even there, in my experience; my laptop and desktop reboot once every few weeks or longer, usually for kernel upgrades, and even with legacy init, bios-to-login-screen is not the longest part of that process. For interactive machines, systemd is already a complicated solution for a simple non-problem. For servers, it's worse.

cwyers 12y ago

> Linux has been at the "it just works" level for decades, where it matters -- on servers and appliances. Like hell. Anything beyond the most basic tasks on a server (and even the most basic tasks, sometimes) requires configuration, and often that configuration ranges between "nonintuitive" and "actively spiteful." If a service crashes and the pid file doesn't get handled properly, you've got to go in there and fix it. And god forbid you have to touch SELinux for anything. "Just works." I want what you're smoking.

acdha 12y ago

The first few trivial counter-examples which come to mind: * someone adds a storage device. Now your server can't boot because what used to be /dev/sda1 is now /dev/sdc1. Yes, UUIDs partially address that but unless you were defining "decades" as "within the last year or so" you can't assume that they were used. * init script starts something, which crashes / is killed by OOM / etc. and you'd like it to be restarted. Note that this is still unsolved with upstart if the process crashes for more than your respawn limit, as is common with e.g. networking-related startup failures but, hey, it's the same braintrust which required multiple major releases to accept that sysadmins need logging and daemons to be started as users other than root or that "hang forever" is not a valid error handling strategy.

felixgallo 12y ago
Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It never even happened with SCSI. And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically.
- acdha 12y ago
  > Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It's not especially common, particularly if you don't work with large servers or tons of local storage. Here are three variations on this theme which I had to deal with: * The Linux kernel changed the order of device enumeration in a point release (early 2.6, if memory serves). This caused some of our servers to start enumeration the on-board IDE CD-ROM ahead of the SCSI drives and on the next reboot the ones which had been setup before Debian started using UUIDs in /etc/fstab failed because /dev/sda was now the CD drive. * A very similar problem happened involving USB mass storage, which was a problem because our remote administration tools could forward a local drive or ISO image to the remote server as a USB device. This was particularly annoying on one early KVM system which only forwarded the device when you were connected so the logical device ordering changed if you rebooted it while connected to the console. * If you have multiple controllers in a system, the drives are enumerated in order per-controller. If the bays were not fully populated when the system was first setup, a new drive added to controller 1 will be enumerated ahead of the drives from controller 2 and all of the subsequent device names will be shifted by one. All of these are why it's now routine to use UUIDs to identify drives. It's a solvable problem but hasn't been so for even a single decade. > And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. Uptime and resilience are baseline requirements for what I work on, not "rare cases". I've used external supervisors but that's introducing a third-party tool with its own bugs and means that you can't assume that another sysadmin is familiar with it. If you're the monit person in a supervisord shop (or vice versa) you'll probably have something take longer to debug or break at some point because of a subtlety in the differences between the two. This is such a common case that it makes sense to solve it once at the system level, particularly since that allows me to e.g. not have to code an extra bit in my daemon that knows how to ask arbitrary third-party supervisors about a different process. > If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically. That's a fine approach if the failures are caused by something like a broken config file but it doesn't handle environmental failures gracefully. For example, consider a daemon which fails if it can't resolve a hostname or connect to a server (e.g. a database) or which exits if it can't write to disk. In either case, you're going to have downtime if the server / data-center drops offline or a partition fills up. The question is whether the service comes back quickly after the underlying problem is fixed or requires someone to localize it and restart the process. If you have one server and you happen to be on-duty, this isn't a big deal but for anyone with many servers running different apps it can be a headache after a network partition or power failure (oops, web servers all came online well before the database server finished checking its RAID array). In the case of upstart, the default respawn limit is only 10 times within 5 seconds after which it will stop the job until you manually restart it. Since they didn't add throttling or delays, there's no way to deal with this within upstart without either increasing downtime for every error (e.g. adding "|| sleep 10" to every script) or wasting more CPU restarting jobs as quickly as possible until you hit a larger limit. Again, not an unsolvable problem but had they not ignored a huge amount of prior art and perhaps even consulted with actual sysadmins there would be things like a respawn delay and backoff so you could say things like "restart this no more than n times per minute". Instead, most places either convert everything to use a different supervisor or support an external monitor which requests restarts for services periodically, both of which add maintenance overhead and risk for what is a standard feature on other systems.