4 ms·

It's also extremely fast.

kazinator 7y ago
Sure, until you need to do something advanced like have a function return a string and use it in the caller. Test case: #!/bin/bash fun() { echo "result" } RES=$(fun) echo $RES Snippet from system call trace (via "strace bash ./test.sh"): pipe([3, 4]) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 lseek(255, -10, SEEK_CUR) = 51 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccbed72a10) = 3870 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGCHLD, {0x446240, [], SA_RESTORER|SA_RESTART, 0x7fccbe3b0ff0}, {0x446240, [], SA_RESTORER|SA_RESTART, 0x7fccbe3b0ff0}, 8) = 0 close(4) = 0 read(3, "result\n", 128) = 7 read(3, "", 128) = 0 In other words, this requires a pipe to be created a child process to be spawned, and its output to be read from that pipe. In any reasonable scripting language, and even some unreasonable ones, this is all done in one address space. How about, oh, Awk: function fun() { return "result" } One of the most common mantras for effective shell programming is "avoid writing loops; orchestrate external utilities".
- Spivak 7y ago
  The way one typically accomplishes this in bash is the following since the $() construct spawns a subshell. #!/bin/bash fun() { res="result" } fun && echo "$res" The same strace snippet. openat(AT_FDCWD, "./test.sh", O_RDONLY) = 3 stat("./test.sh", {st_mode=S_IFREG|0755, st_size=53, ...}) = 0 ioctl(3, TCGETS, 0x7ffd65c6d890) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_CUR) = 0 read(3, "#!/bin/bash\n\nfun() {\n res=\"resu"..., 80) = 53 lseek(3, 0, SEEK_SET) = 0 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=512*1024}) = 0 fcntl(255, F_GETFD) = -1 EBADF (Bad file descriptor) dup2(3, 255) = 255 close(3) = 0 fcntl(255, F_SETFD, FD_CLOEXEC) = 0 fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fstat(255, {st_mode=S_IFREG|0755, st_size=53, ...}) = 0 lseek(255, 0, SEEK_CUR) = 0 read(255, "#!/bin/bash\n\nfun() {\n res=\"resu"..., 53) = 53 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}) = 0 write(1, "result\n", 7result ) = 7 read(255, "", 53) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 exit_group(0) = ? If you want use the old magics to return values like the built-ins then it's a little uglier but a lot cleaner. #!/bin/bash fun() { eval $1=\""result"\" } fun res && echo $res
  - kazinator 7y ago
    Assignment to hard-coded global variable is not a return mechanism, and generally a nonstarter. It's not a viable approach for generally returning strings out shell functions everywhere in a codebase as a matter of habit. Use of eval: still slow, because the shell eval re-processes input from the character level and up. eval should be generally avoided as much as possible in shell programming. Careless use of eval can introduce security holes (piece of untrusted datum gets evaled as an expression). You really need to have your black belt in "shell escaping karate". Producing output and capturing with command substitution is the primary idiom for getting text out of a shell function. It has no visible side effect. The rebinding of standard output is scoped to the process substitution (and is confined to the child process, in fact), and the creation of the temporary process and pipe, expensive as they might be, are invisible to the program semantics. > The way one typically accomplishes this in bash In summary, what you propose here is not only vanishingly atypical, but also bad coding practice.
    - Spivak 7y ago
      > In summary, what you propose here is not only vanishingly atypical, but also bad coding practice. What I'm describing has been SOP in bash for twenty years. You can program in a more modern style and replace some uses of eval with ${!indirect_references}, declare, and associative arrays but eval is still in wide use today. Search through /etc for eval with grep and you'll return plenty of results in the wild. For example the eval in my previous post could be rewritten as the following declare -g "$1"="result" but this form, although safer, is the less common usage. Assigning to global variables or passing in a variable name to get the return value is just how bash works. See $MAPFILE $OPTIND $OPTARG for examples of the former and read as an example of the latter.
      - LukeShu 7y ago
        That [declare -g] is bad too, because it means you can't use it to set a local variable (see my sibling comment about dynamic scoping), which is surprising and confusing to the caller. While I agree with the parent that the whole thing is disgusting, I also recognize that it's a valid optimization technique, and it's sometimes necessary. So, if you do go down that route, I'd encourage you to do it as printf -v "$1" '%s' "$result"
aidenn0 7y ago
I find it to be quite slow; I wrote a command-line arguments parser library using shell metaprogramming, and I had to precompile the output for bash, while in dash it was fast enough to run as-is.
- LukeShu 7y ago
  I suspect that the slowness was caused by Bash having a couple of pathological cases where it converts back and forth between the LANG/LC encoding and internal wide-characters repeatedly, which can cause it to be stupidly slow. I suspect that setting LC_ALL=C for that portion of the code would speed Bash up to Dash speeds.
  - aidenn0 7y ago
    Factor of 3 speedup, thanks!
deleted 7y ago
[deleted]
3xblah 7y ago
The article mentions loops with an incrementing variable, e.g., i=$((i+1)). It is also possible to do this without loops. Is it faster. Left to the reader to decide. I used this technique sometimes so I could check on the progress of a script while it is running and also continue a script later where I left off. It also allowed me to stop the script by removing a file. At the time, the shell I used had not yet implemented LINENO, which is also very useful when scripts are terminated before finishing. For example, if I wanted to "loop" 15 times^1 To begin, first create a file that stores the count.^2 It also acts as a way to stop the "loop" from advancing if it is removed. echo 0 > x Then create a file that acts as an "on/off switch" to stop the script, e.g., if it is not runnning in the foreground. > x-on Then run the script. The "prologue" and "epilogue" use only shell built-ins. No external programs or Bash-isms are required.^3 #! /bin/sh CRAWL_DELAY=30; test -f x||exit $LINENO; test -f x-on||exit $LINENO; read x < x; test $x -le 14||exit $LINENO; # do stuff; echo https://example.com/$x; sleep CRAWL_DELAY; echo $((x+1)) >x; test -f x-on||exit $LINENO; test -f $0||exit $LINENO; $0 To see progress, cat x To stop the script before it finishes, rm x-on To restart the script and continue where left off, > x-on and run the script. 1. Sometimes I did not have access to a program like seq or jot so something like this was not possible for x in $(seq 15);do echo https://example.com/$x; done 2. I always work in a tmpfs-mounted directory so these "files" are just memory, they are not saved on "disk". 3. Bash has too many features for this author to keep track of, like, e.g., base conversion echo $((16#a))

Sure, until you need to do something advanced like have a function return a string and use it in the caller. Test case: #!/bin/bash fun() { echo "result" } RES=$(fun) echo $RES Snippet from system call trace (via "strace bash ./test.sh"): pipe([3, 4]) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 lseek(255, -10, SEEK_CUR) = 51 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fccbed72a10) = 3870 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGCHLD, {0x446240, [], SA_RESTORER|SA_RESTART, 0x7fccbe3b0ff0}, {0x446240, [], SA_RESTORER|SA_RESTART, 0x7fccbe3b0ff0}, 8) = 0 close(4) = 0 read(3, "result\n", 128) = 7 read(3, "", 128) = 0 In other words, this requires a pipe to be created a child process to be spawned, and its output to be read from that pipe. In any reasonable scripting language, and even some unreasonable ones, this is all done in one address space. How about, oh, Awk: function fun() { return "result" } One of the most common mantras for effective shell programming is "avoid writing loops; orchestrate external utilities".

Spivak 7y ago
The way one typically accomplishes this in bash is the following since the $() construct spawns a subshell. #!/bin/bash fun() { res="result" } fun && echo "$res" The same strace snippet. openat(AT_FDCWD, "./test.sh", O_RDONLY) = 3 stat("./test.sh", {st_mode=S_IFREG|0755, st_size=53, ...}) = 0 ioctl(3, TCGETS, 0x7ffd65c6d890) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_CUR) = 0 read(3, "#!/bin/bash\n\nfun() {\n res=\"resu"..., 80) = 53 lseek(3, 0, SEEK_SET) = 0 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=512*1024}) = 0 fcntl(255, F_GETFD) = -1 EBADF (Bad file descriptor) dup2(3, 255) = 255 close(3) = 0 fcntl(255, F_SETFD, FD_CLOEXEC) = 0 fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fstat(255, {st_mode=S_IFREG|0755, st_size=53, ...}) = 0 lseek(255, 0, SEEK_CUR) = 0 read(255, "#!/bin/bash\n\nfun() {\n res=\"resu"..., 53) = 53 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}) = 0 write(1, "result\n", 7result ) = 7 read(255, "", 53) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 exit_group(0) = ? If you want use the old magics to return values like the built-ins then it's a little uglier but a lot cleaner. #!/bin/bash fun() { eval $1=\""result"\" } fun res && echo $res
- kazinator 7y ago
  Assignment to hard-coded global variable is not a return mechanism, and generally a nonstarter. It's not a viable approach for generally returning strings out shell functions everywhere in a codebase as a matter of habit. Use of eval: still slow, because the shell eval re-processes input from the character level and up. eval should be generally avoided as much as possible in shell programming. Careless use of eval can introduce security holes (piece of untrusted datum gets evaled as an expression). You really need to have your black belt in "shell escaping karate". Producing output and capturing with command substitution is the primary idiom for getting text out of a shell function. It has no visible side effect. The rebinding of standard output is scoped to the process substitution (and is confined to the child process, in fact), and the creation of the temporary process and pipe, expensive as they might be, are invisible to the program semantics. > The way one typically accomplishes this in bash In summary, what you propose here is not only vanishingly atypical, but also bad coding practice.
  - Spivak 7y ago
    > In summary, what you propose here is not only vanishingly atypical, but also bad coding practice. What I'm describing has been SOP in bash for twenty years. You can program in a more modern style and replace some uses of eval with ${!indirect_references}, declare, and associative arrays but eval is still in wide use today. Search through /etc for eval with grep and you'll return plenty of results in the wild. For example the eval in my previous post could be rewritten as the following declare -g "$1"="result" but this form, although safer, is the less common usage. Assigning to global variables or passing in a variable name to get the return value is just how bash works. See $MAPFILE $OPTIND $OPTARG for examples of the former and read as an example of the latter.
    - LukeShu 7y ago
      That [declare -g] is bad too, because it means you can't use it to set a local variable (see my sibling comment about dynamic scoping), which is surprising and confusing to the caller. While I agree with the parent that the whole thing is disgusting, I also recognize that it's a valid optimization technique, and it's sometimes necessary. So, if you do go down that route, I'd encourage you to do it as printf -v "$1" '%s' "$result"

aidenn0 7y ago

I find it to be quite slow; I wrote a command-line arguments parser library using shell metaprogramming, and I had to precompile the output for bash, while in dash it was fast enough to run as-is.

LukeShu 7y ago
I suspect that the slowness was caused by Bash having a couple of pathological cases where it converts back and forth between the LANG/LC encoding and internal wide-characters repeatedly, which can cause it to be stupidly slow. I suspect that setting LC_ALL=C for that portion of the code would speed Bash up to Dash speeds.
- aidenn0 7y ago
  Factor of 3 speedup, thanks!

deleted 7y ago

[deleted]

3xblah 7y ago

The article mentions loops with an incrementing variable, e.g., i=$((i+1)). It is also possible to do this without loops. Is it faster. Left to the reader to decide. I used this technique sometimes so I could check on the progress of a script while it is running and also continue a script later where I left off. It also allowed me to stop the script by removing a file. At the time, the shell I used had not yet implemented LINENO, which is also very useful when scripts are terminated before finishing. For example, if I wanted to "loop" 15 times^1 To begin, first create a file that stores the count.^2 It also acts as a way to stop the "loop" from advancing if it is removed. echo 0 > x Then create a file that acts as an "on/off switch" to stop the script, e.g., if it is not runnning in the foreground. > x-on Then run the script. The "prologue" and "epilogue" use only shell built-ins. No external programs or Bash-isms are required.^3 #! /bin/sh CRAWL_DELAY=30; test -f x||exit $LINENO; test -f x-on||exit $LINENO; read x < x; test $x -le 14||exit $LINENO; # do stuff; echo https://example.com/$x; sleep CRAWL_DELAY; echo $((x+1)) >x; test -f x-on||exit $LINENO; test -f $0||exit $LINENO; $0 To see progress, cat x To stop the script before it finishes, rm x-on To restart the script and continue where left off, > x-on and run the script. 1. Sometimes I did not have access to a program like seq or jot so something like this was not possible for x in $(seq 15);do echo https://example.com/$x; done 2. I always work in a tmpfs-mounted directory so these "files" are just memory, they are not saved on "disk". 3. Bash has too many features for this author to keep track of, like, e.g., base conversion echo $((16#a))