3 ms·

Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It

Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It never even happened with SCSI. And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically.

acdha 12y ago
> Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It's not especially common, particularly if you don't work with large servers or tons of local storage. Here are three variations on this theme which I had to deal with: * The Linux kernel changed the order of device enumeration in a point release (early 2.6, if memory serves). This caused some of our servers to start enumeration the on-board IDE CD-ROM ahead of the SCSI drives and on the next reboot the ones which had been setup before Debian started using UUIDs in /etc/fstab failed because /dev/sda was now the CD drive. * A very similar problem happened involving USB mass storage, which was a problem because our remote administration tools could forward a local drive or ISO image to the remote server as a USB device. This was particularly annoying on one early KVM system which only forwarded the device when you were connected so the logical device ordering changed if you rebooted it while connected to the console. * If you have multiple controllers in a system, the drives are enumerated in order per-controller. If the bays were not fully populated when the system was first setup, a new drive added to controller 1 will be enumerated ahead of the drives from controller 2 and all of the subsequent device names will be shifted by one. All of these are why it's now routine to use UUIDs to identify drives. It's a solvable problem but hasn't been so for even a single decade. > And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. Uptime and resilience are baseline requirements for what I work on, not "rare cases". I've used external supervisors but that's introducing a third-party tool with its own bugs and means that you can't assume that another sysadmin is familiar with it. If you're the monit person in a supervisord shop (or vice versa) you'll probably have something take longer to debug or break at some point because of a subtlety in the differences between the two. This is such a common case that it makes sense to solve it once at the system level, particularly since that allows me to e.g. not have to code an extra bit in my daemon that knows how to ask arbitrary third-party supervisors about a different process. > If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically. That's a fine approach if the failures are caused by something like a broken config file but it doesn't handle environmental failures gracefully. For example, consider a daemon which fails if it can't resolve a hostname or connect to a server (e.g. a database) or which exits if it can't write to disk. In either case, you're going to have downtime if the server / data-center drops offline or a partition fills up. The question is whether the service comes back quickly after the underlying problem is fixed or requires someone to localize it and restart the process. If you have one server and you happen to be on-duty, this isn't a big deal but for anyone with many servers running different apps it can be a headache after a network partition or power failure (oops, web servers all came online well before the database server finished checking its RAID array). In the case of upstart, the default respawn limit is only 10 times within 5 seconds after which it will stop the job until you manually restart it. Since they didn't add throttling or delays, there's no way to deal with this within upstart without either increasing downtime for every error (e.g. adding "|| sleep 10" to every script) or wasting more CPU restarting jobs as quickly as possible until you hit a larger limit. Again, not an unsolvable problem but had they not ignored a huge amount of prior art and perhaps even consulted with actual sysadmins there would be things like a respawn delay and backoff so you could say things like "restart this no more than n times per minute". Instead, most places either convert everything to use a different supervisor or support an external monitor which requests restarts for services periodically, both of which add maintenance overhead and risk for what is a standard feature on other systems.

acdha 12y ago

> Nobody adds a storage device and ends up changing the boot drive on a server. I've never even heard of that in three decades of changing drives on servers. It's not especially common, particularly if you don't work with large servers or tons of local storage. Here are three variations on this theme which I had to deal with: * The Linux kernel changed the order of device enumeration in a point release (early 2.6, if memory serves). This caused some of our servers to start enumeration the on-board IDE CD-ROM ahead of the SCSI drives and on the next reboot the ones which had been setup before Debian started using UUIDs in /etc/fstab failed because /dev/sda was now the CD drive. * A very similar problem happened involving USB mass storage, which was a problem because our remote administration tools could forward a local drive or ISO image to the remote server as a USB device. This was particularly annoying on one early KVM system which only forwarded the device when you were connected so the logical device ordering changed if you rebooted it while connected to the console. * If you have multiple controllers in a system, the drives are enumerated in order per-controller. If the bays were not fully populated when the system was first setup, a new drive added to controller 1 will be enumerated ahead of the drives from controller 2 and all of the subsequent device names will be shifted by one. All of these are why it's now routine to use UUIDs to identify drives. It's a solvable problem but hasn't been so for even a single decade. > And in the rare case where automatic restart is a good idea, I wrap that script in a supervisor, like monit, as god intended. Uptime and resilience are baseline requirements for what I work on, not "rare cases". I've used external supervisors but that's introducing a third-party tool with its own bugs and means that you can't assume that another sysadmin is familiar with it. If you're the monit person in a supervisord shop (or vice versa) you'll probably have something take longer to debug or break at some point because of a subtlety in the differences between the two. This is such a common case that it makes sense to solve it once at the system level, particularly since that allows me to e.g. not have to code an extra bit in my daemon that knows how to ask arbitrary third-party supervisors about a different process. > If a process is crashing for more than the supervisor's respawn limit, then you have a problem that computers can't solve. Not sure how you thought that could be resolved automatically. That's a fine approach if the failures are caused by something like a broken config file but it doesn't handle environmental failures gracefully. For example, consider a daemon which fails if it can't resolve a hostname or connect to a server (e.g. a database) or which exits if it can't write to disk. In either case, you're going to have downtime if the server / data-center drops offline or a partition fills up. The question is whether the service comes back quickly after the underlying problem is fixed or requires someone to localize it and restart the process. If you have one server and you happen to be on-duty, this isn't a big deal but for anyone with many servers running different apps it can be a headache after a network partition or power failure (oops, web servers all came online well before the database server finished checking its RAID array). In the case of upstart, the default respawn limit is only 10 times within 5 seconds after which it will stop the job until you manually restart it. Since they didn't add throttling or delays, there's no way to deal with this within upstart without either increasing downtime for every error (e.g. adding "|| sleep 10" to every script) or wasting more CPU restarting jobs as quickly as possible until you hit a larger limit. Again, not an unsolvable problem but had they not ignored a huge amount of prior art and perhaps even consulted with actual sysadmins there would be things like a respawn delay and backoff so you could say things like "restart this no more than n times per minute". Instead, most places either convert everything to use a different supervisor or support an external monitor which requests restarts for services periodically, both of which add maintenance overhead and risk for what is a standard feature on other systems.