6 ms·
Kill Me Softly – Kill processes in a reliable manner
- alanfranzoni 12y agoOften I need to kill processes, and wait until they're dead. I want to send a SIGTERM first, then a SIGKILL if it fails. And I want this process to be automated. That's it.
- tspiteri 12y agoI know it's normally a good idea to avoid code duplication, but in this case I would have copied the function inside kms_functions into both kmsn and kmsp, since this removes the dependency on readlink and also makes it possible to move the executables around without care.
- alanfranzoni 12y agoI have been thinking at alternatives modes of operation that don't require an explicit install (it's in the TODOs), I'll probably improve the whole in the next days. Thanks for your suggestion.
- icebraining 12y agoYeah, and you can always use a build tool to generate the two scripts from a DRY codebase.
- wdewind 12y agoOne of the best things I learned from a very senior engineer was "There are far worse things in life than duplicating code." Dependencies are frequently one of those things.
- khrf 12y agoOh thank you, these are gold words. I was amused recently when my collegues wanted to avoid code duplication of one small library and ended up with creating a new repository, learning awful Python packaging and releasing a new package for every small change. Much more pain gained IMHO.
- peterwwillis 12y agoAlternative modes of operation that would be useful: * Send SIGUSR1 if TERM doesn't work * Send SIGCONT if TERM doesn't work * Attempt to reap the process * Attempt to ptrace the process * Give a hint as to why it can't be killed (like blocked I/O, a zombie, etc) * Kill the process group (dangerous!)
- alanfranzoni 12y agoYes, of course that would be useful, something is already in the TODOs. I don't want to overcomplicate what is a simple shell script by the way. Reaping may be interesting, ptrace may be a bit of overkill. Thanks.
- ad_hominem 12y agoSIGUSR1 is often caught by applications for their own IPC. SIGQUIT would probably be a safer one to send, but that can also be caught and repurposed. Sending other signal types would definitely have to be opt-in to prevent unintended side effects.
- dankilman 12y agoThose should be available on almost any Linux system, even on minimal installs: Linux
- cbd1984 12y agoTautologies aren't false.
- rdtsc 12y agoIf you are the one designing your applications, don't rely on SIGTERM and clean shutdowns. Design them so SIGKILL is the default way to stop them such that no data is corrupted and persistent state stays consistent. Then you don't have to rely on some rarely executed "recovery" code, or even worse take chances of what might happen if SIGTERM doesn't work. Handle SIGTERM only if you get a blackbox-like application (a database that will break if you pull the power plug on it) and so on.
- GauntletWizard 12y agoI don't disagree with the intent (Make crash-resistant, journaling programs), but TERM is an incredibly useful tool. Lame-Ducking (when your program continues processing current requests but refuses new ones) is a great way to handle rollouts and other planned maintenance in a way that minimises or prevents user visibility.
- dimman 12y agoA proper init system will send you a SIGTERM before a normal reboot/shutdown, and also for stopping the program. I do agree that "power failure safety" is a good way to go, but ignoring to handle SIGTERM if there is a use for it is just ignorant.
- rdtsc 12y ago> I do agree that "power failure safety" is a good way to go, So you have write code to handle abrupt power shutoff? And test it, and make sure it doesn't break your data at rest or your overwall service. And you have to write to handle a paralle set of code to also handle SIGTERM. > , but ignoring to handle SIGTERM if there is a use for it is just ignorant. It seems having 2 sets of shutdown/failure code (one abrupt power failure, "one gentle" is the more ignorant path.
- hyperpape 12y agoI thought you couldn't handle SIGKILL, so how do you have two sets of code? SIGTERM just lets you do a bit more of the operations you've guaranteed are always safe.
- vacri 12y agoSo instead of two kill messages, we now have two scripts to run, dependent on name or pid? Just put everything in one script - these scripts are shorter than an init boilerplate section - and use an option to switch. Do process names by default, then perhaps use a -p arg to switch the script into 'pid' mode. Bash's 'getopts' is pretty easy to implement. As it stands, you currently have to waste mental power to figure out whether you want the 'n' or 'p' version anyway, so may as well just make it an arg.
- ukandy 12y agoIs the functionality even needed? Doesn't take much to type.. ksm `pidof cupsd`
- benatkin 12y agoA lot of these bash scripts could be replaced with Go programs. It doesn't take much code for a bash script to get ugly.
- ukandy 12y agoSure they could, but then you would need different compiled versions for different architectures, and this is kind of OTT for the task it accomplishes as it is!
- benatkin 12y agoThis would be an incentive to package the code, which would mean being able to check it against SHAs. There are a lot of go programs on homebrew now. I'd like to see a trend of smaller ones getting added there.
- dimman 12y agoOr why not make them #!/bin/sh compatible instead, I don't have bash on my systems for instance.
- pepve 12y agoI really want to like this. But SIGKILL is such a special instrument in the administrator's toolbox that I don't want it automatically sent after a short timeout. Moreover I don't think anyone should want that. This tool will hide or allow you to ignore important issues with your system. I do unequivocally like the blocking behaviour though, I might implement that for myself. Edit: so just something like this: function killwait() { kill $1 while ps -p $1 &>/dev/null; do sleep 1 done }
- dimman 12y agoI'd recommend adding some kind of max retries mechanism because otherwise, if process ignores SIGTERM for instance, you can wait forever.
- khrf 12y agoAnd again you'll get this: > This tool will hide or allow you to ignore important issues with your system.
- pepve 12y agoI personally prefer utilities like this just do one simple task and perform it well. And adding a timeout would just complicate things unnecessarily, compared to the alternative of pressing Ctrl-C when you feel you've waited long enough.
- lomnakkus 12y agoI appreciate the sentiment, but if you think about it... If a process can't be killed arbitrarily at any point during its execution then it's definitely not behaving in a "transactional" way (as in: sort-of-like-ACID-except-the-durability-may-not-be-that-important-as-long-as-consistency-and-atomicity-is-preserved). You really want processes to behave "transactionally", otherwise you'll be really really screwed when the power/UPSs fail. (I'm ignoring the fact that a "kill -9" actually still flushes buffers and such, but that's tangential to the point I'm trying to make.) EDIT: In summary: I'd rather find out sooner rather than later :).
- faragon 12y agoIn my opinion, current process handling is braindead, both in POSIX and Windows. That could be "fixed" with an additional unique identifier during the OS uptime, in order to allow other processes track instances, and not just the ones doing the fork.
- dap 12y agoAs you point out, using the pid alone is not reliable. FWIW, Solaris and illumos systems have process contracts that allow you to reliably track processes[0]. If the child forks, it's the new child is still in the same contract (unless configured otherwise). And both the child or parent (the watcher) can restart and then return to the normal state (parent watching child processes). Solving this problem requires kernel support because the kernel is generally the only fault domain in the system that cannot crash while the system is still running. Edit: forgot the link the first time. [0] http://illumos.org/man/4/process http://illumos.org/man/4/process
- feld 12y agoLinux has this capability in recent kernel versions (3.x?). I don't know what the implementation is called (prctl?), but DragonflyBSD add this in procctl(2) in release 4.0. FreeBSD already had a procctl(2) and added the same functionality to be compatible. http://lists.dragonflybsd.org/pipermail/commits/2014-November/343844.html http://lists.dragonflybsd.org/pipermail/commits/2014-Novembe... So now FreeBSD and DragonflyBSD have plans to utilize this exactly like Illumos and build lightweight process supervision without the need for something as disruptive as systemd.
- _cipher_ 12y ago> Those should be available on almost any Linux system, even on minimal installs: > [..] > Bash Maybe I'm nitpicking here, but why Bash _should_ be available?
- alanfranzoni 12y agoThank to everybody for the feedback. Standalone executable scripts that are both OSX and Linux compatible are now available.