6 ms·

> A script that processes N files passed as arguments? for filename in "$@"; do ... done > Or that processes a listing of files piped in from

> A script that processes N files passed as arguments? for filename in "$@"; do ... done > Or that processes a listing of files piped in from stdin? That has the tricky bit of "what about filenames that contain newlines" that would come up in an implementation in any language. Since the one byte not allowed in filenames is '\0', let's say that the listing uses '\0' to separate filesnames. while read -d '' -r filename; do ... done There are things that are hard in Bash... but the things you mentioned aren't them. > but which lets you strictly control the ARGV and env of those function calls without the whole thing passing through a TCL-like stringification and re-tokenization. I'm pretty sure that's just Bash, if you remember to wrap everything in double-quotes.

derefr 8y ago
That second example is write-only code, and becomes moreso the more things you have to do with the filename. What if you have to reassemble a few of these into an array, filter that array by matching tokens within it, and then feed those arguments through as NUL-separated stdin to another binary, with error checking all along the way to make sure you never call the final command with invalid arguments? In bash, that's a 50-line monstrosity. In Ruby (or Python, or even Perl), it's ~5 clean, readable lines. > I'm pretty sure that's just Bash, if you remember to wrap everything in double-quotes. Now try composing a command-line in Bash out of those "simply double-quoted values" to pass to ssh as a command to start an ssh exec channel with, where the command also contains a `su -c` stanza. Again, in another language, that's just a repeated application of a shellescape function as you're composing things. In bash... I have no idea. --- These are both examples of the type of code you have to write all the time in Bash when dealing with file manipulation, because any file could have any byte-sequence (except NUL) in its name, and you're dealing with the results of calling commands that want to pipe those filenames out to you as raw delimited data (hopefully NUL-delimited.) Let's say you're writing a script that copies a bunch of files into a dir, and then archives them into a tar file. Are you sure your command reads and recreates both relative and absolute symlinks correctly, in a way that they'll unpack correctly for the tar file? In most scripting languages, you've got functions like Pathname#relative_path_from(dir) and Pathname#clean_path. To get the "correct" symlink in bash, you've got `readlink -f` and a comparison and then a bunch of string manipulations using `basename` and `dirname`. Or how about a "recursively collapse empty or pseudo-empty directory" function? Where, by "pseudo-empty", I mean "empty except for .DS_Store; Thumbs.db; Desktop.ini; those resource-fork files on FAT volumes/SMB shares that start with ._; or other empty or pseudo-empty directories". That's not a function anything ships with, but it's ~10 lines of Ruby. Try to implement it in Bash! (Yes, you can decide whether to collapse the whole hierarchy in one pass with `find`. But the mission here is to remove all the empty and pseudo-empty dirs found in the hierarchy, like an enhanced version of `find -empty`.) (Notice that these are both examples of things where a Bash programmer would be likely to say "why not write a new binary in C that does this thing, and then call it from your script"? Well, because the whole reason I'm stuck writing in a scripting language is that this code needs to be portable to unknown target CPU ISAs and OS ABIs, silly.) There is a reason that the default kind of cloud-init script is a YAML file, rather than a Bash script. There is a reason that Vagrantfiles aren't Bash; a reason that Ansible is less popular than Chef or Puppet or Fabric; a reason why every /bin/init that isn't sysvinit has soundly beaten sysvinit. And it's why, even in Ansible, or autotools, or any package manager's pre-install and post-install hooks, the parts where you do file manipulation are abstracted out, either being done through YAML (Ansible), or through domain-specific file-manipulation binaries (dpkg) or through pre-written and thoroughly-checked file-manipulation Bash scripts (autotools). And that reason is that doing file manipulation directly in Bash is not worth the time and effort, and especially not worth the maintenance burden in a codebase shared with other programmers who are probably not Bash experts.
- LukeShu 8y ago
  > That second example is write-only code, Tacking on the `-d ''` flag to `read` makes it write-only!? > In bash, that's a 50-line monstrosity. In Ruby (or Python, or even Perl), it's ~5 clean, readable lines. As an example, let's take the input, batch it in to groups of 5, filter-out files that don't contain "token1" or "token2", and pass those as a NUL-separated list to stdin of another binary. I wrote that up. It came out to 10 lines in either Ruby or Bash. https://pastebin.com/EksRxmVJ https://pastebin.com/EksRxmVJ It's certainly not a "50-line monstrosity" > Again, in another language, that's just a repeated application of a shellescape function as you're composing things. In bash... I have no idea. In Bash, "shellescape" is pronounced "printf %q" (and shelljoin is pronounced "printf '%q '"). In Bash, that's similarly just repeated application of printf %q: base_cmd=('whatever' 'arg with space') nested_cmd=(sh -c "$(printf '%q ' "${base_cmd[@]}")") ssh user@hostname "$(printf '%q ' "${nested_cmd[@]}")" For direct comparison with Ruby: escaped=$(printf '%q ' "${base_cmd[@]}") escaped=base_cmd.map{|a|shellescape(a)}.join(" ") escaped=shelljoin(base_cmd) Or, Bash 4.4 gained @Q as a shorthand for that: base_cmd=('whatever' 'arg with space') nested_cmd=(sh -c "${base_cmd[*]@Q}") ssh user@hostname "${nested_cmd[*]@Q}" > In most scripting languages, you've got functions like Pathname#relative_path_from(dir) and Pathname#clean_path. You mean $(realpath --relative-to=DIR) and $(realpath)? > Or how about a "recursively collapse empty or pseudo-empty directory" function? ... it's ~10 lines of Ruby. Try to implement it in Bash!... And it's 6 lines of Bash? find $ -not -type d \ -not -name .DS_Store \ -not -name Thumbs.db \ -not -name Desktop.ini \ -not -name '._*' \ $ -printf '%h\0%p\0' | sort -uz Or were you looking for a listing of pseudo-empty directories? comm -z23 <(find -type d -print0 | sort -z) \ <(find $ -not -type d \ -not -name .DS_Store \ -not -name Thumbs.db \ -not -name Desktop.ini \ -not -name '._*' \ $ -printf '%h\0%p\0' | sort -uz) Like I said, there are things that are hard to do in Bash, but I'm not certain you've pointed any of them out. > There is a reason that the default kind of cloud-init script is a YAML file, rather than a Bash script. And there's also a reason why a "GNU/Linux distro" is essentially a giant pile of shell scripts (maybe embedded in to Makefiles). > a reason why every /bin/init that isn't sysvinit has soundly beaten sysvinit. I mean, I'm happy with systemd on my systems, but: OpenRC is a giant pile of shell scripts. > the parts where you do file manipulation are abstracted out, ... through domain-specific file-manipulation binaries (dpkg) Great example, because the core of Debian is the "rules" file for each package, which is... a Makefile, containing shell. Or, look at Arch Linux, the core of which is the "PKGBUILD" file for each package, which is... a Bash script. > or through pre-written and thoroughly-checked file-manipulation Bash scripts (autotools). Autoconf is targeting shells that aren't Bash. It's pre-written and thoroughly-checked shell that contains wonky workarounds, in order to be compatible with buggy shells from 25 years ago. If you're assuming "modern shell / utilities", file manipulation in Bash isn't hard.

derefr 8y ago

That second example is write-only code, and becomes moreso the more things you have to do with the filename. What if you have to reassemble a few of these into an array, filter that array by matching tokens within it, and then feed those arguments through as NUL-separated stdin to another binary, with error checking all along the way to make sure you never call the final command with invalid arguments? In bash, that's a 50-line monstrosity. In Ruby (or Python, or even Perl), it's ~5 clean, readable lines. > I'm pretty sure that's just Bash, if you remember to wrap everything in double-quotes. Now try composing a command-line in Bash out of those "simply double-quoted values" to pass to ssh as a command to start an ssh exec channel with, where the command also contains a `su -c` stanza. Again, in another language, that's just a repeated application of a shellescape function as you're composing things. In bash... I have no idea. --- These are both examples of the type of code you have to write all the time in Bash when dealing with file manipulation, because any file could have any byte-sequence (except NUL) in its name, and you're dealing with the results of calling commands that want to pipe those filenames out to you as raw delimited data (hopefully NUL-delimited.) Let's say you're writing a script that copies a bunch of files into a dir, and then archives them into a tar file. Are you sure your command reads and recreates both relative and absolute symlinks correctly, in a way that they'll unpack correctly for the tar file? In most scripting languages, you've got functions like Pathname#relative_path_from(dir) and Pathname#clean_path. To get the "correct" symlink in bash, you've got `readlink -f` and a comparison and then a bunch of string manipulations using `basename` and `dirname`. Or how about a "recursively collapse empty or pseudo-empty directory" function? Where, by "pseudo-empty", I mean "empty except for .DS_Store; Thumbs.db; Desktop.ini; those resource-fork files on FAT volumes/SMB shares that start with ._; or other empty or pseudo-empty directories". That's not a function anything ships with, but it's ~10 lines of Ruby. Try to implement it in Bash! (Yes, you can decide whether to collapse the whole hierarchy in one pass with `find`. But the mission here is to remove all the empty and pseudo-empty dirs found in the hierarchy, like an enhanced version of `find -empty`.) (Notice that these are both examples of things where a Bash programmer would be likely to say "why not write a new binary in C that does this thing, and then call it from your script"? Well, because the whole reason I'm stuck writing in a scripting language is that this code needs to be portable to unknown target CPU ISAs and OS ABIs, silly.) There is a reason that the default kind of cloud-init script is a YAML file, rather than a Bash script. There is a reason that Vagrantfiles aren't Bash; a reason that Ansible is less popular than Chef or Puppet or Fabric; a reason why every /bin/init that isn't sysvinit has soundly beaten sysvinit. And it's why, even in Ansible, or autotools, or any package manager's pre-install and post-install hooks, the parts where you do file manipulation are abstracted out, either being done through YAML (Ansible), or through domain-specific file-manipulation binaries (dpkg) or through pre-written and thoroughly-checked file-manipulation Bash scripts (autotools). And that reason is that doing file manipulation directly in Bash is not worth the time and effort, and especially not worth the maintenance burden in a codebase shared with other programmers who are probably not Bash experts.

LukeShu 8y ago
> That second example is write-only code, Tacking on the `-d ''` flag to `read` makes it write-only!? > In bash, that's a 50-line monstrosity. In Ruby (or Python, or even Perl), it's ~5 clean, readable lines. As an example, let's take the input, batch it in to groups of 5, filter-out files that don't contain "token1" or "token2", and pass those as a NUL-separated list to stdin of another binary. I wrote that up. It came out to 10 lines in either Ruby or Bash. https://pastebin.com/EksRxmVJ https://pastebin.com/EksRxmVJ It's certainly not a "50-line monstrosity" > Again, in another language, that's just a repeated application of a shellescape function as you're composing things. In bash... I have no idea. In Bash, "shellescape" is pronounced "printf %q" (and shelljoin is pronounced "printf '%q '"). In Bash, that's similarly just repeated application of printf %q: base_cmd=('whatever' 'arg with space') nested_cmd=(sh -c "$(printf '%q ' "${base_cmd[@]}")") ssh user@hostname "$(printf '%q ' "${nested_cmd[@]}")" For direct comparison with Ruby: escaped=$(printf '%q ' "${base_cmd[@]}") escaped=base_cmd.map{|a|shellescape(a)}.join(" ") escaped=shelljoin(base_cmd) Or, Bash 4.4 gained @Q as a shorthand for that: base_cmd=('whatever' 'arg with space') nested_cmd=(sh -c "${base_cmd[*]@Q}") ssh user@hostname "${nested_cmd[*]@Q}" > In most scripting languages, you've got functions like Pathname#relative_path_from(dir) and Pathname#clean_path. You mean $(realpath --relative-to=DIR) and $(realpath)? > Or how about a "recursively collapse empty or pseudo-empty directory" function? ... it's ~10 lines of Ruby. Try to implement it in Bash!... And it's 6 lines of Bash? find $ -not -type d \ -not -name .DS_Store \ -not -name Thumbs.db \ -not -name Desktop.ini \ -not -name '._*' \ $ -printf '%h\0%p\0' | sort -uz Or were you looking for a listing of pseudo-empty directories? comm -z23 <(find -type d -print0 | sort -z) \ <(find $ -not -type d \ -not -name .DS_Store \ -not -name Thumbs.db \ -not -name Desktop.ini \ -not -name '._*' \ $ -printf '%h\0%p\0' | sort -uz) Like I said, there are things that are hard to do in Bash, but I'm not certain you've pointed any of them out. > There is a reason that the default kind of cloud-init script is a YAML file, rather than a Bash script. And there's also a reason why a "GNU/Linux distro" is essentially a giant pile of shell scripts (maybe embedded in to Makefiles). > a reason why every /bin/init that isn't sysvinit has soundly beaten sysvinit. I mean, I'm happy with systemd on my systems, but: OpenRC is a giant pile of shell scripts. > the parts where you do file manipulation are abstracted out, ... through domain-specific file-manipulation binaries (dpkg) Great example, because the core of Debian is the "rules" file for each package, which is... a Makefile, containing shell. Or, look at Arch Linux, the core of which is the "PKGBUILD" file for each package, which is... a Bash script. > or through pre-written and thoroughly-checked file-manipulation Bash scripts (autotools). Autoconf is targeting shells that aren't Bash. It's pre-written and thoroughly-checked shell that contains wonky workarounds, in order to be compatible with buggy shells from 25 years ago. If you're assuming "modern shell / utilities", file manipulation in Bash isn't hard.