4 ms·

I wouldn't predict this, but I don't find it terribly out of line. Bash, and shells in general, are designed to be interactive command execution environments.

I wouldn't predict this, but I don't find it terribly out of line. Bash, and shells in general, are designed to be interactive command execution environments. The syntax easily allows multiple commands on a single line, and this is not uncommon in interactive or scripting use, in my experience. Many shells also favor permissive execution. What I mean by this is that they will keep running as far as they can. It is perfectly reasonable (at least based on the behavior of many shells) for a script to run several commands successfully and then fail later on when facing a syntax error or nonexistent command. Many shells are also carrying design decisions from a much more resource-constrained era of computing. It certainly makes sense in terms of preserving system resources to simply execute a command at a time, to full completion, before even reading forward. This minimizes the amount of parsing and the amount of script required to keep in memory. I am not saying that this is the best way to do things, but that it is an optimization along certain axes. Separately, it is not unreasonable in terms of design and implementation of a shell to unify as much as possible the scripting environment and the interactive environment. Keeping this in mind, the way to think of a script is just as an interactive session. Each line is just a command entered at the prompt. This ensures that behavior in scripts is the same as the behavior the user sees every day. How to implement this, especially in the face of resource constraints, in another way? I am not saying that the implementation as it stands is correct or good, but just that this is not an unreasonable behavior.

ciprian_craciun 6y ago
Furthermore I would say this is actually a "feature", as some scripts might be too large to fit into limited memory, and also reading-and-executing one step at a time allows one to write "dynamic" scripts that are generated as they are executed. Although I haven't written such "dynamic" scripts (yet), I'm sure a few exist that based on `expect` and `bash` work in a "feedback-loop" manner.
- ArchD 6y ago
  If you want to use a 'dynamic' script, why not pipe from stdin instead?
  - ciprian_craciun 6y ago
    Yes, you are right, pipes are mandatory here. When I was referring to a "dynamic" script I didn't mean "just append to a file", but instead I was referring to the fact that `bash` doesn't first try to load the whole script, parse and then execute it, but instead it "streams" through the script.
IngoBlechschmid 6y ago
I agree it's definitely on purpose. As you write, there is no reason to keep the full script in memory. Sure, human-written scripts will always fit, and probably did so even in the old days, but some scripts are computer-generated. (Recall those "self-extracting shell scripts"? Their size is unbounded. Nowadays they even have a hip .io domain: makeself.io) Secondly, bash does not always seek back and reread parts of the script it already read. For instance it will skip doing that when only executing plain echo commands instead of commands which necessitate forking. Perhaps someone could look up the relevant part of the source? See it for yourself by stracing (with strace -e %process,%file,read,write,lseek) a file containing echo a echo b echo c versus date +a date +b date +c
ArchD 6y ago
Even when allowing incremental reading of the script, it would be safer to use inotify to detect changes to the script being executed and fail when changes are detected. Also, flock() could be used to signal to other processes that the script file is not to be written to.
- smichel17 6y ago
  This seems like the correct solution to me. If I run a 4GB script (eg, a self-extracting script with a zip embedded, like another comment mentioned), I don't want to oom myself. Locking during execution or exiting with an appropriate error code seems like much more desirable behavior.
foobarian 6y ago
Also, what is the use case exactly? In-place replacement of scripts in production that run almost like services? I would think that even if this behavior was not there, doing the above is a huge red flag and outside of the scope of what the shell was designed for.
- greggyb 6y ago
  My post was all about why this is not surprising behavior for bash, and my justification for the same. To be very clear, I don't find this surprising given the circumstances of bash's birth into this world and its primary use case as an interactive shell. I don't suggest any specific use case for this behavior, or even that there is an intended use case at all. I am choosing "behavior" as the word to use very specifically. It's not a feature. It is a behavior that falls out of a combination of implementation goals and constraints. I do not find this behavior surprising. I am not suggesting that someone sat down and reasoned through why a language should behave in this way when executing a script.
  - foobarian 6y ago
    Understood; I was not trying to dispute your post or anything. It's an overall point that when faced with this bash behavior, the correct response should be not to be more careful, but redesign whatever system is running it in such a way that this can be a production problem in the first place.
lukeschlather 6y ago
There are a lot of features of bash that are probably fundamentally incompatible with this, but for non-interactive scripts I really only want to use languages that read in the entire file and parse it before executing a single line. This can even allow stuff like JIT compilation which, though it does bear some startup cost is faster in the long run.
- greggyb 6y ago
  Oh, for sure. I agree. I don't think bash is the tool for that job.

ciprian_craciun 6y ago

Furthermore I would say this is actually a "feature", as some scripts might be too large to fit into limited memory, and also reading-and-executing one step at a time allows one to write "dynamic" scripts that are generated as they are executed. Although I haven't written such "dynamic" scripts (yet), I'm sure a few exist that based on `expect` and `bash` work in a "feedback-loop" manner.

ArchD 6y ago
If you want to use a 'dynamic' script, why not pipe from stdin instead?
- ciprian_craciun 6y ago
  Yes, you are right, pipes are mandatory here. When I was referring to a "dynamic" script I didn't mean "just append to a file", but instead I was referring to the fact that `bash` doesn't first try to load the whole script, parse and then execute it, but instead it "streams" through the script.

IngoBlechschmid 6y ago

I agree it's definitely on purpose. As you write, there is no reason to keep the full script in memory. Sure, human-written scripts will always fit, and probably did so even in the old days, but some scripts are computer-generated. (Recall those "self-extracting shell scripts"? Their size is unbounded. Nowadays they even have a hip .io domain: makeself.io) Secondly, bash does not always seek back and reread parts of the script it already read. For instance it will skip doing that when only executing plain echo commands instead of commands which necessitate forking. Perhaps someone could look up the relevant part of the source? See it for yourself by stracing (with strace -e %process,%file,read,write,lseek) a file containing echo a echo b echo c versus date +a date +b date +c

ArchD 6y ago

Even when allowing incremental reading of the script, it would be safer to use inotify to detect changes to the script being executed and fail when changes are detected. Also, flock() could be used to signal to other processes that the script file is not to be written to.

smichel17 6y ago
This seems like the correct solution to me. If I run a 4GB script (eg, a self-extracting script with a zip embedded, like another comment mentioned), I don't want to oom myself. Locking during execution or exiting with an appropriate error code seems like much more desirable behavior.

foobarian 6y ago

Also, what is the use case exactly? In-place replacement of scripts in production that run almost like services? I would think that even if this behavior was not there, doing the above is a huge red flag and outside of the scope of what the shell was designed for.

greggyb 6y ago
My post was all about why this is not surprising behavior for bash, and my justification for the same. To be very clear, I don't find this surprising given the circumstances of bash's birth into this world and its primary use case as an interactive shell. I don't suggest any specific use case for this behavior, or even that there is an intended use case at all. I am choosing "behavior" as the word to use very specifically. It's not a feature. It is a behavior that falls out of a combination of implementation goals and constraints. I do not find this behavior surprising. I am not suggesting that someone sat down and reasoned through why a language should behave in this way when executing a script.
- foobarian 6y ago
  Understood; I was not trying to dispute your post or anything. It's an overall point that when faced with this bash behavior, the correct response should be not to be more careful, but redesign whatever system is running it in such a way that this can be a production problem in the first place.

lukeschlather 6y ago

There are a lot of features of bash that are probably fundamentally incompatible with this, but for non-interactive scripts I really only want to use languages that read in the entire file and parse it before executing a single line. This can even allow stuff like JIT compilation which, though it does bear some startup cost is faster in the long run.

greggyb 6y ago
Oh, for sure. I agree. I don't think bash is the tool for that job.