12 ms·

Take care editing bash scripts (2019)

405 points by cxr 7y ago

yjftsjthsd-h 7y ago
Has this been reported as a bug? Because it strains credulity to think that this is really intended behavior.
- greggyb 7y ago
  I wouldn't predict this, but I don't find it terribly out of line. Bash, and shells in general, are designed to be interactive command execution environments. The syntax easily allows multiple commands on a single line, and this is not uncommon in interactive or scripting use, in my experience. Many shells also favor permissive execution. What I mean by this is that they will keep running as far as they can. It is perfectly reasonable (at least based on the behavior of many shells) for a script to run several commands successfully and then fail later on when facing a syntax error or nonexistent command. Many shells are also carrying design decisions from a much more resource-constrained era of computing. It certainly makes sense in terms of preserving system resources to simply execute a command at a time, to full completion, before even reading forward. This minimizes the amount of parsing and the amount of script required to keep in memory. I am not saying that this is the best way to do things, but that it is an optimization along certain axes. Separately, it is not unreasonable in terms of design and implementation of a shell to unify as much as possible the scripting environment and the interactive environment. Keeping this in mind, the way to think of a script is just as an interactive session. Each line is just a command entered at the prompt. This ensures that behavior in scripts is the same as the behavior the user sees every day. How to implement this, especially in the face of resource constraints, in another way? I am not saying that the implementation as it stands is correct or good, but just that this is not an unreasonable behavior.
  - ciprian_craciun 7y ago
    Furthermore I would say this is actually a "feature", as some scripts might be too large to fit into limited memory, and also reading-and-executing one step at a time allows one to write "dynamic" scripts that are generated as they are executed. Although I haven't written such "dynamic" scripts (yet), I'm sure a few exist that based on `expect` and `bash` work in a "feedback-loop" manner.
    - ArchD 7y ago
      If you want to use a 'dynamic' script, why not pipe from stdin instead?
      - ciprian_craciun 7y ago
        Yes, you are right, pipes are mandatory here. When I was referring to a "dynamic" script I didn't mean "just append to a file", but instead I was referring to the fact that `bash` doesn't first try to load the whole script, parse and then execute it, but instead it "streams" through the script.
  - IngoBlechschmid 7y ago
    I agree it's definitely on purpose. As you write, there is no reason to keep the full script in memory. Sure, human-written scripts will always fit, and probably did so even in the old days, but some scripts are computer-generated. (Recall those "self-extracting shell scripts"? Their size is unbounded. Nowadays they even have a hip .io domain: makeself.io) Secondly, bash does not always seek back and reread parts of the script it already read. For instance it will skip doing that when only executing plain echo commands instead of commands which necessitate forking. Perhaps someone could look up the relevant part of the source? See it for yourself by stracing (with strace -e %process,%file,read,write,lseek) a file containing echo a echo b echo c versus date +a date +b date +c
  - ArchD 7y ago
    Even when allowing incremental reading of the script, it would be safer to use inotify to detect changes to the script being executed and fail when changes are detected. Also, flock() could be used to signal to other processes that the script file is not to be written to.
    - smichel17 7y ago
      This seems like the correct solution to me. If I run a 4GB script (eg, a self-extracting script with a zip embedded, like another comment mentioned), I don't want to oom myself. Locking during execution or exiting with an appropriate error code seems like much more desirable behavior.
  - foobarian 7y ago
    Also, what is the use case exactly? In-place replacement of scripts in production that run almost like services? I would think that even if this behavior was not there, doing the above is a huge red flag and outside of the scope of what the shell was designed for.
    - greggyb 7y ago
      My post was all about why this is not surprising behavior for bash, and my justification for the same. To be very clear, I don't find this surprising given the circumstances of bash's birth into this world and its primary use case as an interactive shell. I don't suggest any specific use case for this behavior, or even that there is an intended use case at all. I am choosing "behavior" as the word to use very specifically. It's not a feature. It is a behavior that falls out of a combination of implementation goals and constraints. I do not find this behavior surprising. I am not suggesting that someone sat down and reasoned through why a language should behave in this way when executing a script.
      - foobarian 7y ago
        Understood; I was not trying to dispute your post or anything. It's an overall point that when faced with this bash behavior, the correct response should be not to be more careful, but redesign whatever system is running it in such a way that this can be a production problem in the first place.
  - lukeschlather 7y ago
    There are a lot of features of bash that are probably fundamentally incompatible with this, but for non-interactive scripts I really only want to use languages that read in the entire file and parse it before executing a single line. This can even allow stuff like JIT compilation which, though it does bear some startup cost is faster in the long run.
    - greggyb 7y ago
      Oh, for sure. I agree. I don't think bash is the tool for that job.
- jefftk 7y ago
  This is a feature I knew about, and one I've found useful. It means I can keep working on a long shell script while the first part is running. The "be careful" applies to editing parts of the script that have already run. Bash in general is a really helpful tool because it lets you trade off some safety for the ability to move very quickly, and this kind of on-the-fly editing is a good example.
deleted 7y ago
[deleted]
atorodius 7y ago
Oh wow! I had the feeling that something like this affected me a handful of times but I was never sure and I couldn‘t reproduce so I just ignored. That‘s quite crazy!
- kristopolous 7y ago
  Stylistically it's outside the scope. Bash is a classic style tool, if you hammer your thumb instead of a nail, that doesn't mean the design of hammers need to be fixed. It's meant to be useful, not smart. This attitude is why the core utils have remained so damn useful for 50 years and haven't devolved into dysfunctional messes. choosing what not to do is important
kristianp 7y ago
I've noticed this behaviour in the past. I remember a sun solaris server at work back around 2009 getting taken down by someone editing a script while it was running. The scripts always had a rm -rf as part of cleanup of working files.
chrismorgan 7y ago
Batch files on Windows do the same thing. That’s caught me out far more often, because I make errors in my batch files much more often than I do with shell scripts.
- malkia 7y ago
  I was just going to comment on this. I believe batch files buffer on line, but can't confirm.
  - jz_ 7y ago
    They do. Can be used it to interact with the execution of the current program. It is simple as appending further code with a label and jumping to it using goto. I first noticed it in 2007 started using it in production.
- leghifla 7y ago
  On the other hand, I used this behaviour to my advantage in the DOS era. I had written a small utility to easily start my games (a Pascal program). My goal was to get as much memory free for the games. To allow unloading my program before launching the game, I wrote a bat script which first launched my program. My program would replace the last line of the script with the game to start (and a goto begining), and then terminate. Now, command.com would start my game, without any memory overhead. When I quit my game, I come back to my utility. At the time, security was not my concern...
Uehreka 7y ago
It’s weird, I’ve been using bash for over ten years now, and just started noticing this behavior a couple weeks ago. It probably has to do with the nature of the scripts I’m running these days (bash scripts that call docker commands to build complicated images and then do other stuff). I could tell that bash wasn’t reading the whole script before executing, but I didn’t spend the time to figure out exactly what the real behavior was. Thanks to the author for this analysis!
- alexis_fr 7y ago
  I have noticed this behavior on Ubuntu in 13.13. On the other hand, I kept the habit of expecting a script to fail when modifying it (most of the time, starting a command in the middle makes it fail) but it never seems to occur on macOS’s bash.
  - Uehreka 7y ago
    If shell scripting on macOS works different (and this sounds like exactly the kind of thing that might work different) then that would explain why I didn’t come across this years ago. Most of my 10 years of shell scripting have been on macOS.
  - Y_Y 7y ago
    Was that the Ubuntu released in the 13th month of 2013, Fictional Ferret?
    - deleted 7y ago
      [deleted]
- yrro 7y ago
  Is it possible your editor used to perform atomic replacement of files, and has now started to overwrite them.in place instead?
noisy_boy 7y ago
I have seen this years ago when I was running a script and then thought of a change and edit it while it was still running. The odd execution thereafter gave it away.
jcynix 7y ago
It doesn't happen with the z-shell (zsh) as I just tested under Debian. Bash shows the described behaviour. Interesting, need to look at other shells too, later. But it's definitely better to be careful and expect the bash ("classic"?) behaviour.
- sneak 7y ago
  That was my next question, thank you. (Just switched to zsh last month after 20+ years on bash.)
- clort 7y ago
  Bear in mind that bash is obviously restarting from the character position specifically but other shells might not handle this at all, meaning that they just use buffered stdin. For such a small script the whole thing has been read into the buffer but a longer script might fault at an otherwise unobvious point. Unless you know that it is explicitly safe (ie the whole file has been cached before it starts) then you should really not modify any running file.
alok4nand 7y ago
I may be missing something here, but why are things like this not repotrted as a bug ?
- jchw 7y ago
  It is not really a bug, just counter intuitive behavior. Their example is somewhat contrived, usually this will just manifest in something being broken. If you use an editor with safe saves this might not even happen because that works by saving to a temp file and doing an atomic swap, and unlinking the old file. In that case, I believe the script should continue as written, because the fd should point to the unlinked file. I believe vim does this by default, for example.
  - larschdk 7y ago
    Using rename() does this for you. It's actually guaranteed to preserve the contents of the old file for any process still having it open while atomically letting any future process only see the new file.
- noja 7y ago
  This is a bug because the downside of this behaviour is outweighed by some tiny upside. But I've never seen a person editing a bash script without an editor that swaps in a new version of the file until today, so never knowingly encountered this bug before. It should be fixed.
  - yachtman 7y ago
    I don't think that's how you define a bug. I think technically you could argue its a "logical error", however that assumes that you know the intent of the creator of bash was specifically to not allow this behavior which I don't think is the case.
    - noja 7y ago
      If nearly everyone doesn't expect the behaviour, and the behaviour has a serious downside, it's a bug.
tannhaeuser 7y ago
There used to be the "text file busy" error to prevent that from happening, where "text" refers to the code segment of an ELF binary, but I've seen the error for shell script "text" as well. Isn't there anymore?
- duskwuff 7y ago
  That error only applies to executables that get mapped into memory while running -- which is to say, binary executables. Script executables have never had that behavior.
  - tannhaeuser 7y ago
    That makes sense, but I have faint memories of vi rejecting writes to running scripts. Maybe it was on Solaris or AIX.
rasz 7y ago
DOS/Windows Batch files behave same way. COMMAND.COM rereads whole batch file every time it executes one command line. PING localhost -n 10 #echo nothing to see here echo finish D:\123>PING localhost -n 10 Pinging ... [127.0.0.1] with 32 bytes of data: Reply from 127.0.0.1: bytes=32 time<1ms TTL=64 ... Ping statistics for 127.0.0.1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms D:\123>#echo nothing to see here '#echo' is not recognized as an internal or external command, operable program or batch file. D:\123>echo finish finish now rerun it and edit first line to 'PING localhost -n 1' and suddenly ... Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms D:\123>echo nothing to see here nothing to see here D:\123>echo finish finish
- weinzierl 7y ago
  What you describe is also interesting behaviour but it is different from the effect seen in bash. For bash it works rougly like this (discounting the shebang to keep things simple): 1. bash reads first line and remembers the byte offset where to continue next. In this case at the octothorpe at byte position 10 (assuming UNIX line endings). 2. bash executes first line 3. while bash is still busy with the first line someone removes a character before position 10 4. Bash reads what it assumes to be the next line at byte position 10, which is now where the 'r' is. 5. bash executes 'rm -rf --no-preserve-root' EDIT: The observed behaviour for a DOS/Windows batch file is indeed the same. I assumed that the trick would not work because supposedly Windows reads the whole file again after every command, but apparently it does not remember the line it was executing but also a byte offset echo start pause echo stop Removing the last character from the first line while the script is paused results in Bad command or filename - "cho".
- JdeBP 7y ago
  This is why Rex Conn's add-on command interpreter for MS/PC/DR-DOS, 4DOS, had the concept of BTM scripts. BTM stands for "Batch To Memory" and 4DOS (and all of its successors through NDOS, 4NT, and 4OS2 to the current Take Command) reads the entirety of a BTM script before running it. * https://jpsoft.com/help/batchtype.htm https://jpsoft.com/help/batchtype.htm The reason that this could not be simply turned on globally for BAT scripts is that various tools made use of the original behaviour, perhaps most notably "fancy change directory" scripts and similar programs, where a wrapped executable was called from a wrapper script, and the executable rewrote the next lines of the script on the fly to do the selected action.
ciprian_craciun 7y ago
I'm glad someone finally wrote about this issue. I have also stumbled upon this a few years ago, but I blamed myself (for using an editor that saves in place), and just applied the following "workaround" when needed: wrap the whole script in a function, and as the last line of the script just call it with `_main "${@}"`.
- Pawamoy 7y ago
  To expand on this, yes, you can force Bash to read more lines and keep them in memory before executing them by wrapping them in blocks. These blocks can be (), {}, functions, and maybe other structures. In addition to this, you can make sure no appended code is ever executed by explicitly running "exit" at the end of your block. I actually used this trick in a self git-updating script (the updated version could contain more lines at the end).
pnut 7y ago
Hot patching is dangerous in all execution environments. This is software engineering... Separate your dev environment from production and do a controlled deployment.
- arthurcolle 7y ago
  I have enough bureaucracy in my life, hard and fast rules like this is what kills actual innovation. Sometimes you just want to run something to get it working. Software engineering isn't religion. At the end of the day the focus should be on feature development, with a slight bias towards decent process.
  - spicyramen 7y ago
    I agree on the innovation side, but when you have a public service or paying customers, and large Engineering teams you definitely need rules and basic environment separation is not too hard
  - Cthulhu_ 7y ago
    While true, this assumes you work in a controlled environment, with a pipeline of checks and tests before things go into production; this particular article is about changing a script while it's running somewhere. Which is probably not a good idea in the first place.
- kleiba 7y ago
  But this is not hot patching, at least not if I understand correctly: changing 30 to 3 was not meant to have an effect on the running script, it was meant lower the waiting time for next time you're running the script. This is not an unusual thing to do in other situations. I'm sure you change your code all the time while an instance of the program is running, no matter whether it's compiled or interpreted. At least I don't habitually close a stop program first before I make changes to the source code. The example in the article shows that that's a bad idea in bash scripts because they, unlike other code, don't get read in entirely (as in, say, Python, Perl, etc.) before they get executed.
- rkangel 7y ago
theodpHN 7y ago
FWIW, the IBM mainframe world's partitioned dataset (PDS) addressed this issue 40+ years ago. Old processes continue to run with the version of the PDS member they started with, while new processes start with a newer version, if one exists. Similar to read consistency features of Oracle.
photon-torpedo 7y ago
Whether this happens or not depends on how the file is edited, I think. From the strace output shown in TFA, it appears the file is not opened again between executing the lines. E.g. using vim as editor, the modified file is actually written as a new file under the same filename. In Linux, the bash process will keep the original file open so that the inodes are not reclaimed by the filesystem until the file is closed. Hence the bash process won't "see" the modified script file. I've tested this, and indeed when editing the file with vim, I can't reproduce the "dangerous" behavior. To trigger this behavior, I'd have to overwrite the script file, e.g. by 1. create script run.sh with "sleep 30" etc 2. create script run1.sh with "sleep 3" instead 3. start run.sh 4. overwrite script: cat run1.sh > run.sh 5. after the sleep, the commented line does execute Are there editors that actually overwrite the original file, instead of creating a new file? I thought the latter is the recommended approach as it can prevent data loss e.g. if there's a crash during writing the file. P.S. If the script in question is on a network file system, things may be different even for vim. E.g. on NFS the file handle doesn't behave fully like a regular POSIX file handle, and the NFS server might actually reopen the file between reads, thus delivering the modified contents to the bash process.
- scbrg 7y ago
  > Are there editors that actually overwrite the original file, instead of creating a new file? Yes, there are. This bit me at one point, since I created a Make target depending on directories rather than individual files in the directories (for performance). A directory gets updated when a new file is created, so it worked for all developers except one - who had an editor that saved files in place. I think that developer was using Sublime, but I might remember wrong.
  - rrobukef 7y ago
    I know `nano` does this.
  - uryga 7y ago
    iirc Sublime Text overwrites if you set "atomic_save": false
michaelmrose 7y ago
I can reproduce this in bash but not in fish or zsh. This seems to be a uniquely bad feature of bash.
- 0ld 7y ago
  yep, i've also checked bourne shell and csh/tcsh, cannot reproduce too. only in bash
acqq 7y ago
I've continuously observed the described problem, and have read about a workaround: when writing the "top level" lines of script, never write them alone, but inside of a { } It looks like this in my case: somefunction() { } # main { ... } Now whatever is inside of any of the functions or inside of the "main" braces will be read at once and it won't be replaced with some new lines if I edit the file while the execution still hasn't finished. I would surely prefer having some global flag "reread the script" for those who need the old behavior and "sane" default for the most users (never reread). But general user friendliness and reasonable defaults was seldom a desired goal in the circles that decide about the development of these programs.
- chubot 7y ago
  Hm I put all shell code in functions for other reasons, but it's interesting to know it solves another problem. http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-0-dispatch-pattern-solves-three-important-problems http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... To summarize: 1) saving many interactive snippets in a single file -- each one is a function 2) expose entire shell functions to tools like xargs and chroot 3) if $0 myfunc solves the "ignored errexit" problem now 4) safely modify a bash script while it's executing
tragomaskhalos 7y ago
Windows .bat files are notorious for this behaviour; disappointing to see it in bash.
amelius 7y ago
If you put such dangerous commands on your website, then please make them not user-selectable.
- netsharc 7y ago
  Oh lord... Are there any further baby-proofing you'd like to see?
  - yachtman 7y ago
    No one post any fork bombs...
    - Filligree 7y ago
      Such as this? eval $(echo "I<RA('1E<W3t`rYWdl&r()(Y29j&r{,3Rl7Ig}&r{,T31wo});r`26<F]F;==" | uudecode)
dooglius 7y ago
This isn't specific to bash scripts, this can happen to pretty much any file. What we really need is a way for a process to atomically "check out" a copy of a file to read from that won't change.
- usefulcat 7y ago
  While technically true, there's an enormous difference between a window of vulnerability that lasts some fraction of a second (the time needed to read the entire file from the filesystem) and multiple seconds, hours, days or even months depending on how long the script runs.
- wahern 7y ago
  An atomic read fork isn't sufficient. The file could be inconsistent if another process is currently writing to the file at the moment of the snapshot. What you need is an atomic write, which is already provided by rename(2). I've always found it odd that binaries and scripts are often installed 755, including in /bin and /sbin.[1] Perhaps it's because install scripts don't bother changing the read-write permissions, so executable end up with 755 because the default umask is 022. Anyhow, I've taken to removing all write permissions from most of my files, not just executables. I haven't yet experimented with changing my umask to 222, but I suspect it would cause many programs, especially editors, to fail. [1] At least on Linux. I just checked OpenBSD and they're 555. But even on OpenBSD most files in /usr/local/{bin,sbin}, installed by third-party packages, are 755.
- wtetzner 7y ago
  MVCC for filesystems?
cracauer 7y ago
This is why my directories with shell scripts have a "make install" target. For execution they are copied to a place where they are unlinked before written new.
- JdeBP 7y ago
  As M. Ahern points out at https://news.ycombinator.com/item?id=23098813 https://news.ycombinator.com/item?id=23098813 , that's only part of the job. Assuming that you actually use rename() to do the unlinking and atomic updating, the "make install" should also ensure that write permission is removed from the new files. * https://github.com/jdebp/nosh/blob/79b1c0aab9834a09a59e15d47710f355c5c0417a/package/makeinstall#L30 https://github.com/jdebp/nosh/blob/79b1c0aab9834a09a59e15d47...
downerending 7y ago
IIRC, older versions of Python would run with a (writable) fd open to the initial script. If you weren't paying attention, it was possible to accidentally modify that script.
lisper 7y ago
Title should be: Take Care Editing Running BASH Scripts.
- jolmg 7y ago
  Yeah, I was just thinking about how this could only happen when editing scripts while they're running. It doesn't make sense to do that with regular, non-interactive CLI scripts, but I guess it does with interactive ones that display a TUI or prompt.
renewiltord 7y ago
I wonder if folks using Chef to ensure a file is present used a "clobber-always" mode and had it behave correctly. I would imagine so since Chef should make a new file. Anyway, a problem for an older age.
oddline 7y ago
Yikes, $ cat rewrite_me #!/bin/bash C=$(sed -r 's,^(C.+sed.+)#$,\1,g' ./$0); echo "$C" > $0 ##### # date $ ./rewrite_me Fri 08 May 2020 11:41:52 PM CEST
einpoklum 7y ago
I had always assumed that bash first read the whole script into memory (unless it was huge), then parsed and interpreted it. Guess I was wrong. Thanks.

405 points by cxr 7y ago

yjftsjthsd-h 7y ago

Has this been reported as a bug? Because it strains credulity to think that this is really intended behavior.

greggyb 7y ago
I wouldn't predict this, but I don't find it terribly out of line. Bash, and shells in general, are designed to be interactive command execution environments. The syntax easily allows multiple commands on a single line, and this is not uncommon in interactive or scripting use, in my experience. Many shells also favor permissive execution. What I mean by this is that they will keep running as far as they can. It is perfectly reasonable (at least based on the behavior of many shells) for a script to run several commands successfully and then fail later on when facing a syntax error or nonexistent command. Many shells are also carrying design decisions from a much more resource-constrained era of computing. It certainly makes sense in terms of preserving system resources to simply execute a command at a time, to full completion, before even reading forward. This minimizes the amount of parsing and the amount of script required to keep in memory. I am not saying that this is the best way to do things, but that it is an optimization along certain axes. Separately, it is not unreasonable in terms of design and implementation of a shell to unify as much as possible the scripting environment and the interactive environment. Keeping this in mind, the way to think of a script is just as an interactive session. Each line is just a command entered at the prompt. This ensures that behavior in scripts is the same as the behavior the user sees every day. How to implement this, especially in the face of resource constraints, in another way? I am not saying that the implementation as it stands is correct or good, but just that this is not an unreasonable behavior.
- ciprian_craciun 7y ago
  Furthermore I would say this is actually a "feature", as some scripts might be too large to fit into limited memory, and also reading-and-executing one step at a time allows one to write "dynamic" scripts that are generated as they are executed. Although I haven't written such "dynamic" scripts (yet), I'm sure a few exist that based on `expect` and `bash` work in a "feedback-loop" manner.
  - ArchD 7y ago
    If you want to use a 'dynamic' script, why not pipe from stdin instead?
    - ciprian_craciun 7y ago
      Yes, you are right, pipes are mandatory here. When I was referring to a "dynamic" script I didn't mean "just append to a file", but instead I was referring to the fact that `bash` doesn't first try to load the whole script, parse and then execute it, but instead it "streams" through the script.
- IngoBlechschmid 7y ago
  I agree it's definitely on purpose. As you write, there is no reason to keep the full script in memory. Sure, human-written scripts will always fit, and probably did so even in the old days, but some scripts are computer-generated. (Recall those "self-extracting shell scripts"? Their size is unbounded. Nowadays they even have a hip .io domain: makeself.io) Secondly, bash does not always seek back and reread parts of the script it already read. For instance it will skip doing that when only executing plain echo commands instead of commands which necessitate forking. Perhaps someone could look up the relevant part of the source? See it for yourself by stracing (with strace -e %process,%file,read,write,lseek) a file containing echo a echo b echo c versus date +a date +b date +c
- ArchD 7y ago
  Even when allowing incremental reading of the script, it would be safer to use inotify to detect changes to the script being executed and fail when changes are detected. Also, flock() could be used to signal to other processes that the script file is not to be written to.
  - smichel17 7y ago
    This seems like the correct solution to me. If I run a 4GB script (eg, a self-extracting script with a zip embedded, like another comment mentioned), I don't want to oom myself. Locking during execution or exiting with an appropriate error code seems like much more desirable behavior.
- foobarian 7y ago
  Also, what is the use case exactly? In-place replacement of scripts in production that run almost like services? I would think that even if this behavior was not there, doing the above is a huge red flag and outside of the scope of what the shell was designed for.
  - greggyb 7y ago
    My post was all about why this is not surprising behavior for bash, and my justification for the same. To be very clear, I don't find this surprising given the circumstances of bash's birth into this world and its primary use case as an interactive shell. I don't suggest any specific use case for this behavior, or even that there is an intended use case at all. I am choosing "behavior" as the word to use very specifically. It's not a feature. It is a behavior that falls out of a combination of implementation goals and constraints. I do not find this behavior surprising. I am not suggesting that someone sat down and reasoned through why a language should behave in this way when executing a script.
    - foobarian 7y ago
      Understood; I was not trying to dispute your post or anything. It's an overall point that when faced with this bash behavior, the correct response should be not to be more careful, but redesign whatever system is running it in such a way that this can be a production problem in the first place.
- lukeschlather 7y ago
  There are a lot of features of bash that are probably fundamentally incompatible with this, but for non-interactive scripts I really only want to use languages that read in the entire file and parse it before executing a single line. This can even allow stuff like JIT compilation which, though it does bear some startup cost is faster in the long run.
  - greggyb 7y ago
    Oh, for sure. I agree. I don't think bash is the tool for that job.
jefftk 7y ago
This is a feature I knew about, and one I've found useful. It means I can keep working on a long shell script while the first part is running. The "be careful" applies to editing parts of the script that have already run. Bash in general is a really helpful tool because it lets you trade off some safety for the ability to move very quickly, and this kind of on-the-fly editing is a good example.

deleted 7y ago

[deleted]

atorodius 7y ago

Oh wow! I had the feeling that something like this affected me a handful of times but I was never sure and I couldn‘t reproduce so I just ignored. That‘s quite crazy!

kristopolous 7y ago
Stylistically it's outside the scope. Bash is a classic style tool, if you hammer your thumb instead of a nail, that doesn't mean the design of hammers need to be fixed. It's meant to be useful, not smart. This attitude is why the core utils have remained so damn useful for 50 years and haven't devolved into dysfunctional messes. choosing what not to do is important

kristianp 7y ago

I've noticed this behaviour in the past. I remember a sun solaris server at work back around 2009 getting taken down by someone editing a script while it was running. The scripts always had a rm -rf as part of cleanup of working files.

chrismorgan 7y ago

Batch files on Windows do the same thing. That’s caught me out far more often, because I make errors in my batch files much more often than I do with shell scripts.

malkia 7y ago
I was just going to comment on this. I believe batch files buffer on line, but can't confirm.
- jz_ 7y ago
  They do. Can be used it to interact with the execution of the current program. It is simple as appending further code with a label and jumping to it using goto. I first noticed it in 2007 started using it in production.
leghifla 7y ago
On the other hand, I used this behaviour to my advantage in the DOS era. I had written a small utility to easily start my games (a Pascal program). My goal was to get as much memory free for the games. To allow unloading my program before launching the game, I wrote a bat script which first launched my program. My program would replace the last line of the script with the game to start (and a goto begining), and then terminate. Now, command.com would start my game, without any memory overhead. When I quit my game, I come back to my utility. At the time, security was not my concern...

Uehreka 7y ago

It’s weird, I’ve been using bash for over ten years now, and just started noticing this behavior a couple weeks ago. It probably has to do with the nature of the scripts I’m running these days (bash scripts that call docker commands to build complicated images and then do other stuff). I could tell that bash wasn’t reading the whole script before executing, but I didn’t spend the time to figure out exactly what the real behavior was. Thanks to the author for this analysis!

alexis_fr 7y ago
I have noticed this behavior on Ubuntu in 13.13. On the other hand, I kept the habit of expecting a script to fail when modifying it (most of the time, starting a command in the middle makes it fail) but it never seems to occur on macOS’s bash.
- Uehreka 7y ago
  If shell scripting on macOS works different (and this sounds like exactly the kind of thing that might work different) then that would explain why I didn’t come across this years ago. Most of my 10 years of shell scripting have been on macOS.
- Y_Y 7y ago
  Was that the Ubuntu released in the 13th month of 2013, Fictional Ferret?
  - deleted 7y ago
    [deleted]
yrro 7y ago
Is it possible your editor used to perform atomic replacement of files, and has now started to overwrite them.in place instead?

noisy_boy 7y ago

I have seen this years ago when I was running a script and then thought of a change and edit it while it was still running. The odd execution thereafter gave it away.

jcynix 7y ago

It doesn't happen with the z-shell (zsh) as I just tested under Debian. Bash shows the described behaviour. Interesting, need to look at other shells too, later. But it's definitely better to be careful and expect the bash ("classic"?) behaviour.

sneak 7y ago
That was my next question, thank you. (Just switched to zsh last month after 20+ years on bash.)
clort 7y ago
Bear in mind that bash is obviously restarting from the character position specifically but other shells might not handle this at all, meaning that they just use buffered stdin. For such a small script the whole thing has been read into the buffer but a longer script might fault at an otherwise unobvious point. Unless you know that it is explicitly safe (ie the whole file has been cached before it starts) then you should really not modify any running file.

alok4nand 7y ago

I may be missing something here, but why are things like this not repotrted as a bug ?

jchw 7y ago
It is not really a bug, just counter intuitive behavior. Their example is somewhat contrived, usually this will just manifest in something being broken. If you use an editor with safe saves this might not even happen because that works by saving to a temp file and doing an atomic swap, and unlinking the old file. In that case, I believe the script should continue as written, because the fd should point to the unlinked file. I believe vim does this by default, for example.
- larschdk 7y ago
  Using rename() does this for you. It's actually guaranteed to preserve the contents of the old file for any process still having it open while atomically letting any future process only see the new file.
noja 7y ago
This is a bug because the downside of this behaviour is outweighed by some tiny upside. But I've never seen a person editing a bash script without an editor that swaps in a new version of the file until today, so never knowingly encountered this bug before. It should be fixed.
- yachtman 7y ago
  I don't think that's how you define a bug. I think technically you could argue its a "logical error", however that assumes that you know the intent of the creator of bash was specifically to not allow this behavior which I don't think is the case.
  - noja 7y ago
    If nearly everyone doesn't expect the behaviour, and the behaviour has a serious downside, it's a bug.

tannhaeuser 7y ago

There used to be the "text file busy" error to prevent that from happening, where "text" refers to the code segment of an ELF binary, but I've seen the error for shell script "text" as well. Isn't there anymore?

duskwuff 7y ago
That error only applies to executables that get mapped into memory while running -- which is to say, binary executables. Script executables have never had that behavior.
- tannhaeuser 7y ago
  That makes sense, but I have faint memories of vi rejecting writes to running scripts. Maybe it was on Solaris or AIX.

rasz 7y ago

DOS/Windows Batch files behave same way. COMMAND.COM rereads whole batch file every time it executes one command line. PING localhost -n 10 #echo nothing to see here echo finish D:\123>PING localhost -n 10 Pinging ... [127.0.0.1] with 32 bytes of data: Reply from 127.0.0.1: bytes=32 time<1ms TTL=64 ... Ping statistics for 127.0.0.1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms D:\123>#echo nothing to see here '#echo' is not recognized as an internal or external command, operable program or batch file. D:\123>echo finish finish now rerun it and edit first line to 'PING localhost -n 1' and suddenly ... Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms D:\123>echo nothing to see here nothing to see here D:\123>echo finish finish

weinzierl 7y ago
What you describe is also interesting behaviour but it is different from the effect seen in bash. For bash it works rougly like this (discounting the shebang to keep things simple): 1. bash reads first line and remembers the byte offset where to continue next. In this case at the octothorpe at byte position 10 (assuming UNIX line endings). 2. bash executes first line 3. while bash is still busy with the first line someone removes a character before position 10 4. Bash reads what it assumes to be the next line at byte position 10, which is now where the 'r' is. 5. bash executes 'rm -rf --no-preserve-root' EDIT: The observed behaviour for a DOS/Windows batch file is indeed the same. I assumed that the trick would not work because supposedly Windows reads the whole file again after every command, but apparently it does not remember the line it was executing but also a byte offset echo start pause echo stop Removing the last character from the first line while the script is paused results in Bad command or filename - "cho".
JdeBP 7y ago
This is why Rex Conn's add-on command interpreter for MS/PC/DR-DOS, 4DOS, had the concept of BTM scripts. BTM stands for "Batch To Memory" and 4DOS (and all of its successors through NDOS, 4NT, and 4OS2 to the current Take Command) reads the entirety of a BTM script before running it. * https://jpsoft.com/help/batchtype.htm https://jpsoft.com/help/batchtype.htm The reason that this could not be simply turned on globally for BAT scripts is that various tools made use of the original behaviour, perhaps most notably "fancy change directory" scripts and similar programs, where a wrapped executable was called from a wrapper script, and the executable rewrote the next lines of the script on the fly to do the selected action.

ciprian_craciun 7y ago

I'm glad someone finally wrote about this issue. I have also stumbled upon this a few years ago, but I blamed myself (for using an editor that saves in place), and just applied the following "workaround" when needed: wrap the whole script in a function, and as the last line of the script just call it with `_main "${@}"`.

Pawamoy 7y ago
To expand on this, yes, you can force Bash to read more lines and keep them in memory before executing them by wrapping them in blocks. These blocks can be (), {}, functions, and maybe other structures. In addition to this, you can make sure no appended code is ever executed by explicitly running "exit" at the end of your block. I actually used this trick in a self git-updating script (the updated version could contain more lines at the end).

pnut 7y ago

Hot patching is dangerous in all execution environments. This is software engineering... Separate your dev environment from production and do a controlled deployment.

arthurcolle 7y ago
I have enough bureaucracy in my life, hard and fast rules like this is what kills actual innovation. Sometimes you just want to run something to get it working. Software engineering isn't religion. At the end of the day the focus should be on feature development, with a slight bias towards decent process.
- spicyramen 7y ago
  I agree on the innovation side, but when you have a public service or paying customers, and large Engineering teams you definitely need rules and basic environment separation is not too hard
- Cthulhu_ 7y ago
  While true, this assumes you work in a controlled environment, with a pipeline of checks and tests before things go into production; this particular article is about changing a script while it's running somewhere. Which is probably not a good idea in the first place.
kleiba 7y ago
But this is not hot patching, at least not if I understand correctly: changing 30 to 3 was not meant to have an effect on the running script, it was meant lower the waiting time for next time you're running the script. This is not an unusual thing to do in other situations. I'm sure you change your code all the time while an instance of the program is running, no matter whether it's compiled or interpreted. At least I don't habitually close a stop program first before I make changes to the source code. The example in the article shows that that's a bad idea in bash scripts because they, unlike other code, don't get read in entirely (as in, say, Python, Perl, etc.) before they get executed.
rkangel 7y ago

theodpHN 7y ago

FWIW, the IBM mainframe world's partitioned dataset (PDS) addressed this issue 40+ years ago. Old processes continue to run with the version of the PDS member they started with, while new processes start with a newer version, if one exists. Similar to read consistency features of Oracle.

photon-torpedo 7y ago

Whether this happens or not depends on how the file is edited, I think. From the strace output shown in TFA, it appears the file is not opened again between executing the lines. E.g. using vim as editor, the modified file is actually written as a new file under the same filename. In Linux, the bash process will keep the original file open so that the inodes are not reclaimed by the filesystem until the file is closed. Hence the bash process won't "see" the modified script file. I've tested this, and indeed when editing the file with vim, I can't reproduce the "dangerous" behavior. To trigger this behavior, I'd have to overwrite the script file, e.g. by 1. create script run.sh with "sleep 30" etc 2. create script run1.sh with "sleep 3" instead 3. start run.sh 4. overwrite script: cat run1.sh > run.sh 5. after the sleep, the commented line does execute Are there editors that actually overwrite the original file, instead of creating a new file? I thought the latter is the recommended approach as it can prevent data loss e.g. if there's a crash during writing the file. P.S. If the script in question is on a network file system, things may be different even for vim. E.g. on NFS the file handle doesn't behave fully like a regular POSIX file handle, and the NFS server might actually reopen the file between reads, thus delivering the modified contents to the bash process.

scbrg 7y ago
> Are there editors that actually overwrite the original file, instead of creating a new file? Yes, there are. This bit me at one point, since I created a Make target depending on directories rather than individual files in the directories (for performance). A directory gets updated when a new file is created, so it worked for all developers except one - who had an editor that saved files in place. I think that developer was using Sublime, but I might remember wrong.
- rrobukef 7y ago
  I know `nano` does this.
- uryga 7y ago
  iirc Sublime Text overwrites if you set "atomic_save": false

michaelmrose 7y ago

I can reproduce this in bash but not in fish or zsh. This seems to be a uniquely bad feature of bash.

0ld 7y ago
yep, i've also checked bourne shell and csh/tcsh, cannot reproduce too. only in bash

acqq 7y ago

I've continuously observed the described problem, and have read about a workaround: when writing the "top level" lines of script, never write them alone, but inside of a { } It looks like this in my case: somefunction() { } # main { ... } Now whatever is inside of any of the functions or inside of the "main" braces will be read at once and it won't be replaced with some new lines if I edit the file while the execution still hasn't finished. I would surely prefer having some global flag "reread the script" for those who need the old behavior and "sane" default for the most users (never reread). But general user friendliness and reasonable defaults was seldom a desired goal in the circles that decide about the development of these programs.

chubot 7y ago
Hm I put all shell code in functions for other reasons, but it's interesting to know it solves another problem. http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-0-dispatch-pattern-solves-three-important-problems http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... To summarize: 1) saving many interactive snippets in a single file -- each one is a function 2) expose entire shell functions to tools like xargs and chroot 3) if $0 myfunc solves the "ignored errexit" problem now 4) safely modify a bash script while it's executing

tragomaskhalos 7y ago

Windows .bat files are notorious for this behaviour; disappointing to see it in bash.

amelius 7y ago

If you put such dangerous commands on your website, then please make them not user-selectable.

netsharc 7y ago
Oh lord... Are there any further baby-proofing you'd like to see?
- yachtman 7y ago
  No one post any fork bombs...
  - Filligree 7y ago
    Such as this? eval $(echo "I<RA('1E<W3t`rYWdl&r()(Y29j&r{,3Rl7Ig}&r{,T31wo});r`26<F]F;==" | uudecode)

dooglius 7y ago

This isn't specific to bash scripts, this can happen to pretty much any file. What we really need is a way for a process to atomically "check out" a copy of a file to read from that won't change.

usefulcat 7y ago
While technically true, there's an enormous difference between a window of vulnerability that lasts some fraction of a second (the time needed to read the entire file from the filesystem) and multiple seconds, hours, days or even months depending on how long the script runs.
wahern 7y ago
An atomic read fork isn't sufficient. The file could be inconsistent if another process is currently writing to the file at the moment of the snapshot. What you need is an atomic write, which is already provided by rename(2). I've always found it odd that binaries and scripts are often installed 755, including in /bin and /sbin.[1] Perhaps it's because install scripts don't bother changing the read-write permissions, so executable end up with 755 because the default umask is 022. Anyhow, I've taken to removing all write permissions from most of my files, not just executables. I haven't yet experimented with changing my umask to 222, but I suspect it would cause many programs, especially editors, to fail. [1] At least on Linux. I just checked OpenBSD and they're 555. But even on OpenBSD most files in /usr/local/{bin,sbin}, installed by third-party packages, are 755.
wtetzner 7y ago
MVCC for filesystems?

cracauer 7y ago

This is why my directories with shell scripts have a "make install" target. For execution they are copied to a place where they are unlinked before written new.

JdeBP 7y ago
As M. Ahern points out at https://news.ycombinator.com/item?id=23098813 https://news.ycombinator.com/item?id=23098813 , that's only part of the job. Assuming that you actually use rename() to do the unlinking and atomic updating, the "make install" should also ensure that write permission is removed from the new files. * https://github.com/jdebp/nosh/blob/79b1c0aab9834a09a59e15d47710f355c5c0417a/package/makeinstall#L30 https://github.com/jdebp/nosh/blob/79b1c0aab9834a09a59e15d47...

downerending 7y ago

IIRC, older versions of Python would run with a (writable) fd open to the initial script. If you weren't paying attention, it was possible to accidentally modify that script.

lisper 7y ago

Title should be: Take Care Editing Running BASH Scripts.

jolmg 7y ago
Yeah, I was just thinking about how this could only happen when editing scripts while they're running. It doesn't make sense to do that with regular, non-interactive CLI scripts, but I guess it does with interactive ones that display a TUI or prompt.

renewiltord 7y ago

I wonder if folks using Chef to ensure a file is present used a "clobber-always" mode and had it behave correctly. I would imagine so since Chef should make a new file. Anyway, a problem for an older age.

oddline 7y ago

Yikes, $ cat rewrite_me #!/bin/bash C=$(sed -r 's,^(C.+sed.+)#$,\1,g' ./$0); echo "$C" > $0 ##### # date $ ./rewrite_me Fri 08 May 2020 11:41:52 PM CEST

einpoklum 7y ago

I had always assumed that bash first read the whole script into memory (unless it was huge), then parsed and interpreted it. Guess I was wrong. Thanks.