14 ms·
How to write idempotent Bash scripts
- argd678 7y agoThe problem with the idempotent trend in configuration management is that it’s all based on not tracking or knowing what the current state of something is. So reasoning about how these systems work is fundamentally impossible. It would be better to focus on systems where by we can always know the state and improve the tooling there.
- platistocrates 7y agoI don't see any conflict here. Idempotency is not a silver bullet and cannot fully replace state tracking, but it adds simplicity where appropriate.
- LfLxfxxLxfxx 7y agoOn the contrary. With idempotence you know the state after an action. rm -f will always delete the file (if possible). ln -sfn will always work, even for a directory. With the default behaviour of rm, ln -s, etc you know neither the state before, nor after.
- posix_me_less 7y agoThe option -f is making the script more heavy handed and less likely to fail. But this is not necessary for the script to be idempotent and sometimes is not desirable. You can deal with errors in a more safer way and still be idempotent.
- iforgotpassword 7y agoIt's a pragmatic approach to this problem. Usually tracking state means that you write some version number or similar to a file, but than if for some reason that goes out of sync with the actual state, for example because you put it in /etc and someone accidentally restores an older version of it, hilarity ensues, because it's guaranteed the devs never expected this to happen and don't handle it in any way. For the case of a setup script you'd need to update your state file after every step and in turn when running it again see what the file says and resume at whatever point it says. But then you risk running into above problem. I try to write idempotent scripts whenever possible, combined with general sanity checks specific to whatever environment the script expects.
- argd678 7y agoThe reason it’s pragmatic is due to how systems are designed to begin with, it’s not what you want. We know what the state is right after the command for an undetermined amount of time, but we have no knowledge of what’s changing it. Why not cover that? There’s no reason not to other than it hasn’t been worked on enough. Consider a database schema migration, there’s a reason those are not idempotent, you know each state. Seems quite solvable.
- aey 7y agoWhy wouldn’t you use make? Side-effect tracking and cleanup is the job of a build tool.
- da_chicken 7y agoWhy would you use a build tool when you're not writing a build script?
- throw0101a 7y agoInstead of "built tool" think "state machine". (What is the state of the .o file compared to the .c file? Newer or older?)
- bdcravens 7y agoIt's not necessarily a build tool - it's commonly used for that, but it's a useful DSL. rake in the Ruby world is used for all sorts of tasks for web apps and more, even though it was originally designed as the Ruby version of make. Look at npm - the NODE package manager, but it quickly grew to be used for more than server-side applications. A title is a description, not a constraint.
- da_chicken 7y agoJust because a language can do the task doesn't mean it's actually a good idea to use it for that task. Make is much less well known than shell script. In fact, I'd go so far as to say that make is infamously poorly understood and makefiles are infamously poorly written. That makes it less maintainable, or otherwise you'll need to put "experienced with GNU make" on your job description. If you've got the script scheduled, then you're going to have to document it everywhere that it's not actually a build script so please don't disable it because it doesn't obviously look like it needs to be running every 6 hours. And because it's a DSL and you're somewhat going out of scope, you're more likely to have a problem like needed to extend your script to do something you can't do as easily in make. Worse, you might want to do something that you're expressly not supposed to do [0]. You're going to have to defend your decision every time you present the script, too. And if you ever need to pass it off, the first thing you're going to have to do is explain why you used make. Whether or not the person you hand it of too is an idiot or not, you know the first question is going to be "why isn't this a shell script or Python script?" You're going to have to defend your decision to use make instead of shell script because of idempotence, even though you can write idempotent bash scripts. And if the person you're talking to is an idiot -- and let's be fair that there's a good chance that that is the case -- then they'll never figure it out. So, for me, you've got to go beyond "this language can technically do the task" in order for me to understand why you'd want to use something that's generally understood to be used for build scripts for, well, anything other than that. By choosing make, you're doing something unexpected. That's a bad idea. [0]: https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html#Utilities-in-Makefiles https://www.gnu.org/prep/standards/html_node/Utilities-in-Ma...
- Zenst 7y agoMany people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be run by ops on production servers, what happens if they accidently run it twice! An easy way to cover that is to have the script create a file, which you check at the start of the script and if present you exit. For this you can use the date/time, PID and script name for a unique file name, create in a /tmp directory that you cleanup weekly via sculker or whatever frequency you require. Handy way to handle scripts that you want run successfully once and never to be run again.
- cjvirtucio 7y agoI have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.
- prepend 7y agoIf possible, I’ll try to make a symlink just point to the staging dir after tests complete. This isn’t always possible, but makes it easy to roll back to the old prod folder if anything goes awry.
- nerdponx 7y agoWhat do you use to compute the diff? Just the diff command itself on individual files?
- cryptofistMonk 7y agoCould use something like rsync to automatically check for a difference and copy
- mnutt 7y agoIt’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef will determine if the directory already exists. There are of some downsides to that approach which stem from the fact that you can’t possibly declare every possible thing. For instance, if you were to run the above declaration and then later remove the declaration, chef wouldn’t know to delete the directory; at that point chef wouldn’t know anything about the directory at all. So ideally you can build infrastructure from scratch each time (Docker, etc) rather than having to converge an existing machine.
- itchyouch 7y agoChef does have an action on their primitives, so it is possible in the directory example to set the action to :delete instead of only implying :create
- mnutt 7y agoYou’re right, it’s totally functional, the (minor) challenge I see with it is that it breaks the illusion that you’re describing the state of the system, and feels more like describing a set of changes you want to make.
- pas 7y agoChef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the go from bash to something like Rust, instead of Chef/Ansible. Sure, again, this might sound rather unconventional, but you get safety, speed and convenience of a full fledged programming language, single binary output, etc. And you need to test cookbooks/playbooks too, so why spend enormous effort on cobbling together scripts and high-level abstractions instead of writing just what you need with the assumptions you really have - instead of what a general cookbook/playbook has to have (which is a vast difference).
- NelsonMinar 7y agoI got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.
- deleted 7y ago[deleted]
- joshspankit 7y agoIn other words: When promiscuous touching, always make sure you’re not making things unsanitary.
- stephenr 7y agoI had the same thought about `ln -sf` - removing and recreating the symlink aren't no-effect.
- fpoling 7y agoTo avoid that I use readlink in my scripts to read the previous value of the link and call ln -s only when the value is not what is expected.
- kousikk 7y agoLast modified timestamps in source code are a bad idea and they make builds non-hermetic. Better to fix up those scenarios in code than work around them. Irrespective you have a good point that touch does have side effects.
- intc 7y ago> This is an easy one. Touch is by default idempotent. This means you can call it multiple times without any issues. A second call won’t have any effects Every call to touch will alter the modification time of the file (example.txt in this case).
- jrumbut 7y agoIt will change the access time too. Almost every suggestion in the post has this problem which is unfortunate since access/modify/create time are used in a lot of scripts to see which files were or were not processed by prior runs of a script (whether this is a good idea or not). It is very difficult to write truly idempotent bash scripts (consider log files). Creating sort of idempotent or at least rerunnable scripts would be nice but I think even that would take more care than this.
- theamk 7y agoDo people actually use access time for anything? I occasionally do full-text searches on system files, and this resets access times on all files. Thus, I cannot imagine using a script which cares about it.
- mongol 7y agoThis is useful and I think it would be good if more tools had idempotency options available. Just today I needed to run a script that added a zypper repo in OpenSuse. It fails the second time because the repo is already added and there is no option to avoid it. Also the error code is not distinct so it is not easy to script around. Ideally every tool should have a mode "accomplish this result" in addition to "do this".
- michaelmcmillan 7y agoLet's try to avoid using a hammer on anything that resembles a nail: https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671
- dymk 7y agoHow is that related to the article aside from "both involve bash"?
- pmarreck 7y agoIt's because rm -rf "$STEAMROOT/"* will evaluate as rm -rf "/"* if $STEAMROOT for some reason is blank or unset, which is what happened here. (As someone in that discussion mentioned, a little more Bash knowledge might have suggested using rm -rf "${STEAMROOT:?}/"* instead, to force it to (at least) error if it is empty or unset.)
- dymk 7y agoSo literally the only thing in common with the article is that they involve bash
- michaelmcmillan 7y agoI can't help you if you don't see the connection.
- canadev 7y agopersonally, I feel like rm -f is dangerous, and never would recommend using it.
- zokier 7y agoThe first example ("idiom") is bit tricky, because the title is "Creating an empty file". An alternative would for example be echo -n > example.txt Both fit the description of "Creating an empty file". Is it idempotent, or more or less so than `touch example.txt`? If there is already an existing file, then touching it obviously will not empty it, so the end state is not "empty file exists" like one would expect from operation "Creating an empty file" Ironically article calls "This is an easy one"
- dylan604 7y agoIf a file has a line ending in it, is it truly empty?
- fwip 7y agoThe -n flag keeps echo from emitting a line ending.
- dylan604 7y agodoh! I read it as \n, but then that's for other languages, so i was just all sorts of not paying attention
- mappu 7y agoI used a distro in recent memory where `echo -n foo` really produced `-n foo`. The workaround was `printf "%s" foo`.
- fwip 7y agoAh yeah, posix doesn't require -n to do anything in particular. For most use cases, strict adherence to posix compatibility isn't important, but sometimes it really really is.
- gabrielblack 7y ago"echo -n " is useless, is sufficient: > example.txt without prepending the echo command. This will create an empty file or it will truncate an existent file. The touch command could be preferable to preserve the content, if needed.
- zamadatix 7y ago"The -f flag removes the target destination before creating the symbolic link, hence it’ll always succeed." Removing something and adding it back seems to be, by definition, not idempotent. What if something tries to access that file in between? What about the filesystem timestamps (same issues as with the "touch" claim)?
- grzm 7y agoIdempotency focuses on the result, rather than implementation. If the intent of the script is to ensure the file is there when the script is finished, then it’s idempotent. If the time stamps matter for the intent of the script, then, yes, what you pointed out would be an issue.
- Godel_unicode 7y agoThis is incorrect. Idempotency is about not needing to worry about running something more than once. If you introduce race conditions with follow-on runs, you're not idempotent.
- grzm 7y agoIf race conditions are an issue within the context of your intent, then yes, you'll need to take that into account. Again, one needs to take into account the intent and context of the script. If other aspects of the environment ensure that the operations are serialized (say you're making an update to a system which you've rotated out of production for the update), you won't need to worry about race conditions. Need to make sure you have no timing effects? That will effect how the script is written. Need to worry about CPU or IO utilization (including reads)? That will need to be considered.
- ggm 7y agoThe chmod command can alter the flagset on the link not the target, if you select the right options. How do you guarantee the replacement symlink had the same chmod flags and grp flags as the original? (chmod -h for reference*) I don't think these commands are idempotent in the wide. They satisfy only the narrowest "yea, most normal-ish cases are sort of maybe ok" but since we lost atomicity, there are significant windows of time in the filesystem where inodes are in flux, and the new thing has a different inode to the old thing, if you delete and replace it. The disk is different.
- zimbatm 7y agoAll the examples are more-or-less idempotent. For example the `blkid /dev/sda1 || mkfs.ext4 /dev/sda1` doesn't guarantee what type of filesystem the partition will have. `touch file` changes the mtime of the file. `ln -sf` will fail if the target is a directory and is not atomic. It would be nice to have a toolset of commands that are all idempotent to make that type of task easier.
- bonyt 7y agoDon’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp. https://linux.die.net/man/1/mktemp https://linux.die.net/man/1/mktemp
- arendtio 7y agoWell, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/
- achamayou 7y agoAnd anyway you really want mkstemp ;)
- faho 7y agoNot in a shell. There is no `mkstemp` command, at least on a typical GNU system. There is a `mktemp`, which may very well use the `mkstemp` function in its source. I.e. this is about `mktemp(1)`, not `mktemp(3)`.
- slrz 7y agoYou can invoke mktemp(1) in ways that work across most contemporary systems. At least if you don't care that the resulting names are going to look a bit weird (have lots of XXXXXX in them due to name prefix vs. template differences among implementations).
- arendtio 7y agoActually, I am not sure if that is true. I have no example at hand (I could look it up, if you are interested), but a few months ago I was trying hard to create a folder with mktemp using the same command for Linux, MacOS and Android, and as far as I remember, that wasn't as easy as it is supposed to be.
- united893 7y agoThis is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/
- godelski 7y agoI'm not sure why people see errors as a bad thing. Errors are good. They stop you from doing stupid things. But maybe this article is good for telling people why you should handle errors well or when you write code to make sure it fails gracefully. Because if you don't people are going to start telling others to use the force flag when removing (almost always a terrible idea). Learn to handle errors, don't learn to brute force your way through them.
- jmts 7y agoMy understanding of 'set -e' is to ensure that errors are handled appropriately, not to brute force around them. If an error occurs that is not handled, this is treated as a problem and the script exits to ensure that one failure doesn't cause additional unforeseen problems as a result of the script running to completion.
- godelski 7y agoThe brute force method is the force flag (eg: `rm -f`). `set -e` is error handling. Force flag was constantly used in the post which is what I disagree with.
- dymk 7y agoThat's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.
- empath75 7y agoThis is surely a good collection of flags to know, but I sort of object to the premise of the article. There is a good reason that many of these flags are not default behavior, and it’s that they can be quite destructive. If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite stuff if it encounters an unexpected state? Telling people to just always use these flags and not only when they’re quite sure what their script is doing, is probably playing with fire. This may make the script idempotent in repeated runs, but the first run might not be what you expect at all.
- iforgotpassword 7y ago> If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite stuff if it encounters an unexpected state? Where are they saying this? These flags are about making sure the script wouldn't error out right away if you run it again, eg how a regular mkdir for a path that already exists would not exit 0 and thus end script execution (given set -e is active, which the article seems to imply). Also it is not so much about purposely writing a script that contains errors, because that would just mean the error is encountered every time, but more about transient errors like running out of disk space, a curl call failing because of network hiccups etc. Everything your script did up until that failure shouldn't prevent a second call from succeeding.
- mooneater 7y agoGreat article, but a common scenario is missing: I want to run a long running script, but not if another copy of it is already running.
- dymk 7y agohttps://linux.die.net/man/1/flock https://linux.die.net/man/1/flock would probably help
- fwip 7y agoA word of caution on flock: it may fail when used on some network filesystems, and it is tricky to make sure you've got it right. If you're locking on-disk it's pretty straight-forward, though.
- sirn 7y agoflock(1) is not POSIX, though. mkdir(1) can be used if you absolutely want a POSIX way to manage locks. For example: if ! mkdir .lock; then printf >&2 "Already running?\\n" exit 1 fi Some network file system implementation do not guarantee atomic mkdir, so you still need an extra caution with this method.
- posix_me_less 7y agoExistence of a file is an unreliable indicator of script instance running. Much more reliable is to search the script name or other characteristic in the list of running processes. To use this, the script has to have unique name though.
- unixhero 7y agoGreat article. Very helpful.
- gist 7y agoA good way to remember idempotent is to think of the 'delete' or 'mark as read' function for gmail. You can delete an email or mark it as read even if it's been done before (on another open screen as an example).
- bklaasen 7y agoActually I think the "summon lift" button is a better example of idempotence. It doesn't matter how many times you mash the button, the lift is coming ASAP, no quicker. Hitting the button when it's lit doesn't cancel summoning the lift.
- gist 7y agoBy 'lift' you mean 'elevator'? When I first read this I thought you meant 'summon lyft' the ride service but then realized what you meant! Interestingly in the 'old days' of elevator operators hitting it multiple times would be either a 'hurry up' for the operator or an annoyance that maybe made it come slower!
- finchisko 7y agoI think the problem should be solved within the OS using sandboxing, like in iOS (or to some degree Android too) and macOS and Windows store applications. For example, when you uninstall an app, there are no leftovers in the system. You don't need to check, if file exists or not ... To some degree, this has been also solved with containers. It's much easier to create new container image, then create proper idempotent script. And you can create create image from running system too. So you can just "bash" commands and don't care about the state of the system. When done, you just create final image and you can be sure, that state of the system in the container would same as you intended. I'm talking mostly about install scripts. Of course there are valid use cases for idempotent bash scripts.
- rb808 7y agoI think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.
- sametmax 7y agoBesides, EAFP is actually a good way to make things idempotent AND free of race conditions.
- Godel_unicode 7y agoPython startup time is just not in the same league as bash though, if you tried to replace bash across the board you'd end up with lots of unnecessary lag.
- achamayou 7y agoIs that really true with limited imports though, and/or with precompiled bytecode? I was surprised to find that Ubuntu’s handler for unknown commands that tries to suggest what you could apt install is written in Python. I’ve never noticed it being slow.
- pmarreck 7y agoWhile not disagreeing, I would not agree that "everyone should go to Python" since I would personally prefer (for example) Ruby or Elixir, which can work just as well as scripting languages and are at least as readable, while also not triggering too much "VM startup" time. Of course, others would prefer, say, Clojure, or Kotlin (although the JVM startup time is the worst of all), but therein lies the issue... We're stuck with Bash scripting because everyone (more or less) knows/understands it to some extent, it's installed everywhere, and it has essentially no startup-time cost.
- pletnes 7y agoIf you’re into minimal docker images, bash or similar shell might be the only script interpreter available. Python might not exist on your windows machine. Etc.
- tjoff 7y agoIs there any hope for a more modern ubiquitous light weight scripting language? It can be learned, and one can even learn to love it. But scripting is often done by people that very seldom write scripts, the barrier to make decent scripts is much too high and readability isn't much better. The nonsensical syntax is also easy to forget, the result is just an awful lot of headache and poor scripts floating around. Surely we can do better?
- deepsun 7y ago> ln -s source target Terminology is broken. Correct is "ln -s TARGET LINK_NAME" per its man page.
- arendtio 7y agoFor everybody interested in portable shell programming I like to recommend the following page which offers a good overview over POSIX compliant commands with links directly into the POSIX standard: https://shellhaters.org https://shellhaters.org
- theamk 7y agoIs POSIX relevant today? I'd think that overwhelming majority of the shell scripts would require either Linux or BSD dialects. And if one wanted to limit my shell scripts significantly, I'd limit myself to commands supported by busybox's "ash" shell -- this is a limited shell environment which is pretty widely used in initrd's and embedded devices.
- deleted 7y ago[deleted]
- rwestergren 7y agoContext and purpose of the bash script in question is important here. In the example, the author is writing a simple bootstrap script for a dev machine. A number of the critiques here, while valid, are aimed at different use-cases.
- suprgeek 7y agoA lot of the suggestions in this post are "idempotent" only in the sense of the specific use case that the author is interested in. Please do not take this advise to be generally applicable for the uses of "idempotent" or in general to mean best practice while writing bash scripts. The advice such as replace rm <FileName> with rm -f <FileName> could lead to Disaster depending on the scenario. So a HUGE YMMV
- delinka 7y ago> This means you can call it multiple times without any issues. Unless you have processes that depend on modification time. > ln -sfn If you don’t mind the possibility of ownership changing, this is OK.
- ape4 7y agoHey I just learned about `mountpoint`. Great. I used to grep the output of `df`. No more.
- vonseel 7y agoSounds like a bunch of half-baked advice from a mid-level developer who knows a little bit but thinks he knows a lot. As others have mentioned, some of this advice is rather dangerous or only appropriate in particular use-cases. Call me an old bastard, but I'm always suspicious people just write blogs like this to make themselves look like an expert and throw another professional activity on their resumes.
- forty 7y agoI use ansible only not to have to do this :) nice tricks though
- nurettin 7y agoTo make your bash scripts truly idempotent, go to root and turn your system into a git repo. You're welcome.
- jakub_g 7y agoAnother tricky thing: recursively copying a folder is not easy to get right in an idempotent way due to arcane bash behaviors: https://unix.stackexchange.com/questions/228597/how-to-copy-a-folder-recursively-in-an-idempotent-way-using-cp https://unix.stackexchange.com/questions/228597/how-to-copy-...
- posix_me_less 7y agoThat is hardly arcane, the context dependent behaviour of cp and mv is a well-known standard. You are right it does make making a copy in a given path more complicated task.
- sharperguy 7y agoI like to use stuff like mountpoint -q $MOUNTPOINT || mount ... Instead of if ! mountpoint -q $MOUNTPOINT; then mount ... fi It's much more compact in the case of having a lot of commands that need to be checked in case they don't need to be run again.
- CameronNemo 7y agoI was going to mention this, but frankly it is just a style preference. No real difference between either variant. Also, you can do something like: mountpoint -q /proc || { mount -t proc none /proc chown 0400 /proc/slabinfo } if you need multiple statements.