6 ms·
Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cle
by Zenst 7y ago
Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time.
Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be run by ops on production servers, what happens if they accidently run it twice!
An easy way to cover that is to have the script create a file, which you check at the start of the script and if present you exit. For this you can use the date/time, PID and script name for a unique file name, create in a /tmp directory that you cleanup weekly via sculker or whatever frequency you require. Handy way to handle scripts that you want run successfully once and never to be run again.
- cjvirtucio 7y agoI have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.
- prepend 7y agoIf possible, I’ll try to make a symlink just point to the staging dir after tests complete. This isn’t always possible, but makes it easy to roll back to the old prod folder if anything goes awry.
- nerdponx 7y agoWhat do you use to compute the diff? Just the diff command itself on individual files?
- cryptofistMonk 7y agoCould use something like rsync to automatically check for a difference and copy
- dredmorbius 7y agodiff -R You can use numerous tools to compare/manage branches; md5sum / shasum (md5 is usually useful, though not entirely safe), diff and kin, vimdiff, rsync, fdupes, jdupes, git, hg.
- fwip 7y agoYou might find rsync cleaner to use than a diff/cp loop. Another approach is to create a unique directory (named by hash or date) with the results, and once it's good, symlink the real directory to it. (Full disclosure, I stole this approach from nextflow.io, a great workflow system aimed at bioinformatics). ln -sfn "build-cache/2019-07-07-21:23:04" "build"
- pjscott 7y agoApple's new-ish APFS filesystem includes something for this pattern: the renamex_np(2) system call with the RENAME_SWAP flag. This atomically swaps two files, and can be used to atomically swap your staging directory with the existing directory. I'm not sure if there's command line support yet, though.
- cyphar 7y agoLinux has something similar with RENAME_EXCHANGE. Unfortunately MV(1) doesn't appear to have support for any of the renameat2(2) flags.
- grosswait 7y agoAssuming umask is one that bites any dev that has never worked with hardened systems.
- amelius 7y agoJust have a database table, and update it with a special key. When the transaction commits, you know the script has executed correctly, and the key is written as a result. When the script doesn't complete, the commit is not performed, and you can run the script again. No need for "idempotent" scripts. Trying to do this with a filesystem instead of a real database, and things get messy.
- dredmorbius 7y agoWithout proper locks, this sets you up for race conditions.
- amelius 7y agoAny modern database will perform the locking for you as part of running code inside a transaction.
- dredmorbius 7y agoIs the OS operation part of that transaction?
- zbentley 7y agoYou still have to make that messy choice between at-most-once and at-least-once execution of each transactionally guarded section.
- amelius 7y agoThe idea is that you run the entire script as a single transaction. Then you only have to check for a key once (which is trivial).
- zbentley 7y agoSeems risky. What if it blows up, or is killed, halfway through? I'm into using transactions as explicit guards around chunks of known-non-reversible logic as a robust way to warn that manual intervention may be necessary, but using them to replace lockfiles seems to have the worst drawbacks of both systems: it's inherently at-least-once, plus it's substantially more risk surface than a lockfile to establish and maintain a database connection/transaction. Other concerns: What if the database has gone away or crashed by the end of your possibly-long-running script (unless you're using sqlite, in which case, why not use a lockfile? On second thought, sqlite is probably better at lockfiles than flock...)? What about the tech choices that using a transaction precludes? You can't write ops scripts in pure shell any more (without some seriously weird file descriptor munging and job backgrounding to keep the transaction open while you do stuff). Installing your runtime of choice plus a database driver may be unnecessary and time/space-consuming on a lot of systems you need to manage this way. You now also have a bootstrap problem: your database driver and associated tooling may not already be available on your target system, so using such transaction-managed scripts to provision clean/empty systems is another area where additional challenges emerge. All of those can be mitigated or worked around, and this shouldn't be taken as advice to never use an RDBMS's transaction system to manage ops tasks, but I think the cases where it adds more than it costs are pretty rare.
- pdkl95 7y ago> traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state Traps are underused. They solve a lot of error-handling and cleanup problems really easily! Here[1] is a simple example of using a trap to cleanup a tempfile, that safely handles pipeline errors and unexpected exit/return. The same basic idea works for many kinds of error handling, cleanup, or similar work. If you ever need to cleanup something at the end of a function foo() { setup_command # ... cleanup_command } it using a trap might be as simple as moving the cleanup command to trap at the earliest possible time in the function the command can run: foo() { setup_command trap "cleanup_command" RETURN # ... } [1] https://gist.github.com/pdkl95/61fc242e7961cc2584a787ed1760c859 https://gist.github.com/pdkl95/61fc242e7961cc2584a787ed1760c...
- ddebernardy 7y agoIntriguing. I always trap before setting up, on the basis that a script could get interrupted before setup is completed. Is there any general agreement on what the correct order should be?
- pdkl95 7y agoIt probably depends on if the cleanup is safe to run without its setup. For "rm -f", that's true so the trap probably should be first in that case. If you have to worry about strict dependency ordering, trapping first may not be possible. Of course, the better solution to those situations is to make the cleanup command idempotent ... (that said, any order is much better than the unfortunately-common style of simply ignoring errors and exceptions.)
- kahlonel 7y ago> An easy way to cover that is to have the script create a file, which you check at the start of the script and if present you exit. This does sound a good solution on paper, but "checking at the start" and "creating a file" are two different steps (i.e. not atomic) and will cause trouble eventually if your system has the tendency to run the script twice. A better solution is to use the `flock` command before creating the flag file. For example, exec 999>/var/lock/my.lock flock -n 999 || exit 1 if [ ! -f flag_file ]; then echo "script not ran before, running" touch flag_file else echo " already ran, exiting" exit 1 fi # do stuff here
- dredmorbius 7y agoWithin a shell script context, 'mktemp' and 'mv' are almost always what you want, possibly 'ln -s' for libraries. The first safely creates a temporary file. The second (with noclobber set or -n argument) will atomically rename a file, but not overwrite existing contents. Linux uses symlinks to system library files in order to allow in-place atomic replavement without affecting running processes with open filehandles to earlier versions. MS Window's lack of this feature is what makes (or made, I'm very out of date) malware removal and system updates so painful.
- CameronNemo 7y ago>Linux uses symlinks to system library files in order to allow in-place atomic replavement without affecting running processes with open filehandles to earlier versions Nope. If you rm a library, the processes using that library keep using the old library. Symlinks have nothing to do with it.
- iteriteratedone 7y agoThe fact is they shouldnt have to. The env should be known. If you cant control the env then you shouldnt be writing the script. If someone runs your script on a windows machine and it fails its not the script writers fault. There is a level of assumed resources.