14 ms·

Pure Sh Bible

406 points by akraker 3y ago

dorfwald 3y ago
Dylan is a phenomenon. I feel like most people here would know about his projects, though.
- zoover2020 3y ago
  Who are they and why should we know them?
  - dorfwald 3y ago
    Creator of neofetch.
    - sevg 3y ago
      And kisslinux. He seems to have gone off the grid though? Seemingly no public GitHub activity since 2021.
      - akraker 3y ago
        Unfortunately, it seems that he has dropped off the grid. Didn't realize this until you mentioned it actually. It's really too bad. Not a lot is known about why, but more information can be found here: https://www.reddit.com/r/kisslinux/comments/lsbz8n/an_update_on_dylan/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/kisslinux/comments/lsbz8n/an_update...
cde-v 3y ago
[flagged]
cduzz 3y ago
I have to take exception to some of this; it's "technically correct" but like much of shell, likely full of terrifying edge cases that are best avoided. For instance, using eval to have variables with variable names is madness. $ var="world" $ eval "hello_$var=value" $ eval printf '%s\n' "\$hello_$var" I suppose the fancy business with printf is flexing, but I suspect the majority of people scanning documents like this would prefer to have examples with echo "${variable%%string_modifier}" Escape sequences are mostly dependent on you being on a VT102 compatible terminal; I'm not sure what happens if you're on an ASR33 and you send this to it... I'd consider type checking (is this a float) to be similarly cursed witchcraft -- after all I've run into all sorts of "floats" that aren't just 123.456 strings... Overall -- there's a huge range of things shell can do that you probably should just avoid. Similarly, even if you can do it in a clever shell way (the bit shifting math for instance, or extensive string manipulation) should likely be done with external tools just because people understand "oh, awk or sed -- I know this!" instead of "what the hell is this line noise?" If you're writing performance optimized shell, well, probably put the keyboard down and walk away for a bit and reconsider your life choices. Good doc; I'll probably stick to the man page though.
- Linux-Fan 3y ago
  > I suppose the fancy business with printf is flexing, but I suspect the majority of people scanning documents like this would prefer to have examples with echo "${variable%%string_modifier}" POSIX itself contains some interesting notes about `echo` and `printf`, e.g. from the `printf` page: "The printf utility was added to provide functionality that has historically been provided by echo. However, due to irreconcilable differences in the various versions of echo extant, the version has few special features, leaving those to this new printf utility, which is based on one in the Ninth Edition system." The behaviour of `echo` is especially inconsistent for escape sequences. Often, `echo -e` is used to enable escape sequences for it but this is not POSIX compliant. For any "standards-perferring" shell scripting pages I'd thus think `printf` is the right way to present it if only to advertise that this builtin exists. > should likely be done with external tools just because people understand "oh, awk or sed -- I know this!" instead of "what the hell is this line noise?" I am not actually sure about this "I know this" part. I think many users of AWK only ever use it to do some sort of "access the n-th column of input" as in `echo a b c | awk '{ print $2 }'`. Similar for `sed` which is often only known and used for "replace string a by string b in input" use cases. > If you're writing performance optimized shell, well, probably put the keyboard down There is some difference between performance optimized and "hugely inefficient" that can make the difference between a task taking minutes vs. a few seconds. It may not seem much but shell scripts are often integrated into automated processes (think of build environments or system startup tasks) and there, seconds quickly accumulate. Whether this is relevant to your use case, only you can know. I found this "Pure Sh Bible" to be highly interesting, but there are some places where it has some rough edges and it may not show the "best practices" for standard use cases. Still a very useful resource.
  - palunon 3y ago
    > Similar for `sed` which is often only known and used for "replace string a by string b in input" use cases. Sed has at least the vi users who may know ex command from the vi command mode. Awk needs to be learned for itself.
  - cduzz 3y ago
    There are a huge variety of things you can do in shell that you simply shouldn't. If you're adventuring into a world where it matters if there's a newline at the end of your output as a result of your script running on noodlebsd or ache or freeGum, you've already lost. Don't play the game. Don't do things where it matters what the exact output of echo gives you. Did you know that you can't put a null into a shell variable? You can't. Also, you shouldn't ever be in a position to care. Did you know that you can fiddle with the "IFS" to let you parse CSVs? Don't ever do that; it works but nobody will ever understand what the hell you did. You can make an case statement and evaluate it with eval to make your own shell script that writes its own shell scripts. Please don't. All of these adventures are possible, and work fine, and I've done them and come back to the code a decade later and I both understand what I was trying to do and the code still works across a variety of bourne shell interpreters. Nevertheless, these are things that shouldn't be done except to flex.
INTPenis 3y ago
If we're going to be purists, sh is a far cry from bash. /s tongue in cheek, please don't kill me. I love bash and I use it often. I love Greg Wooledge's bash guides and all the people in #bash@libera.
- chasil 3y ago
  I like busybox bash (especially on Windows) which is very much not bash. Busybox bash does not implement arrays, for starters, which is a deal breaker for many scripts. The Windows kernel forks processes about 10x slower than Linux, so these tricks have real performance value for POSIX-family shells running there.
  - _kst_ 3y ago
    As far as I can tell, there is no busybox bash. busybox's built-in shell is ash. There are options to enable a few bash-compatible extensions, but there is no "bash" applet.
    - chasil 3y ago
      Try the Windows version on frippery.org. I see it in the screen capture. https://frippery.org/busybox/index.html https://frippery.org/busybox/index.html
      - arp242 3y ago
        There's an option to install the "bash" applet as a link to either ash or hush, the two shells that busybox comes with. Turns out that a large number of "bash scripts" use no bash-specific features in spite of using "bash" in the #!, or only a few bash-specific features like [[ ]]. It's disabled by default, and arguably not a good idea to enable it because compatibility is not great as you found out. Either way, "busybox bash" doesn't exist: only the option to alias ash or hush to bash.
        chasil 3y ago
juujian 3y ago
```bash lstrip() { # Usage: lstrip "string" "pattern" printf '%s\n' "${1##$2}" } ``` Got me right at the start. I look this up about once a week. Time to put it .bashrc already.
orf 3y ago
# Remove all leading white-space. trim=${1#${1%%[![:space:]]*}} # Remove all trailing white-space. trim=${trim%${trim##*[![:space:]]}} Your scientists were so preoccupied with whether they could, they didn't stop to think if they should
indigodaddy 3y ago
Before clicking, I really thought this was some kind of shell command front-end to spit out Bible verses or something. Woops.
- catiopatio 3y ago
  For something like that, you’d want to use fortune(1) with a Bible fortune file, e.g. - KJV: https://sourceforge.net/projects/fortunebible/ https://sourceforge.net/projects/fortunebible/ - NIV: https://github.com/optio50/Bible-NIV-Fortune https://github.com/optio50/Bible-NIV-Fortune
- rubatuga 3y ago
  You've been watching too many Terry Davis videos
chasil 3y ago
Problems with this advice: ::looping over a file The "loop over the contents of a file" entry does not protect stdin. Use an alternate descriptor. while read -r line <&9 do printf '%s\n' "$line" done 9< "$file" Particularly problematic is ssh, that will pass $line to any remote command that is able to consume it. The looped commands will receive stdin of the loop when an alternate descriptor is used. I use 9 by habit after reading the flock manual page many years ago. ::EXIT trap dash will only call the EXIT trap on a normal exit. Add INT (and any other signal that you want from "kill -l") if you also want to catch abnormal terminations. Windows busybox sh/bash only catches regular exits, no matter what else you add.
- ndsipa_pomu 3y ago
  You might also want to clear IFS if you don't want to trim leading/trailing whitespace (taken from https://mywiki.wooledge.org/BashFAQ/001 https://mywiki.wooledge.org/BashFAQ/001): while IFS= read -r line <&9; do cat > ignoredfile printf '%s\n' "$line" done 9< "$file" ("cat" is used as an example of a command eating stdin)
djha-skin 3y ago
Mildly annoyed when people constantly refer to their article as the Bible of X. For those of us who are religious, it is in poor taste.
- jjgreen 3y ago
  Fair point
- catiopatio 3y ago
  Even as someone who grew up in a very religious household, I am still struggling to come up with a “steel-man” argument on your behalf. Can you explain why you believe it is in poor taste?
  - comprev 3y ago
    How common is the expression "Qur'an of X" instead of Bible?
    - burnished 3y ago
      What? You appear to be saying nonsense.
    - tyingq 3y ago
      The "Torah of X" is fairly common.
      - einpoklum 3y ago
        But in Hebrew, "The Torah" doesn't mean a certain book anyway. There are "The five one-part-of-five's of Torah". The Torah is like the teachings, or the postulates, or the theory. So there's Jehova's Torah, or Torat Hashem; but there's also your mother's torah: "Remember, my son, your father's Mussar (= mores), and do not abandon your mother's Torah (= teachings)".
        tyingq 3y ago
tpoacher 3y ago
Regarding conditional expressions: I found something neat recently. The coreutils version of `test` doesn't have this, and when you use `test`, typically what you're using is the coreutils one. But if you use `builtin test` to force the bash builtin variant of `test`, this has a nice `-v` switch, which allows you to check if a variable is set. I found out about this recently when I had to use it in my bash argument processing library, to check if an option expecting a value had not been provided a value (since checking for an empty variable instead would mean that an actual empty string would also be ignored). (see here: https://git.sr.ht/~tpapastylianou/process_optargs/tree/main/item/sourceable_functions/process_optargs#L464 https://git.sr.ht/~tpapastylianou/process_optargs/tree/main/...)
- teddyh 3y ago
  If you already assume bash, then you don’t need the “-v” option to the “test” builtin; just do if [ "${foo+some_string}" = "" ]; then echo "foo is unset" fi or, in your case: if test "${2+x}" = "" # i.e. if $2 is not set
  - tpoacher 3y ago
    True, but I consider this very hacky, error-prone, and unnecessary when a clear, bespoke test flag exists. Also, if you prefer the "[ ... ]" syntax for testing, then you can use `-v` directly anyway (since that is equivalent to the builtin test keyword anyway).
  - hddqsb 3y ago
    In fact `${foo+some_string}` is supported by POSIX sh (unlike `test -v`).
- hddqsb 3y ago
  Note that `test -v` is not in POSIX sh (see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...). And you don't need `builtin` because `test ...` invokes the builtin by default; if you want to run coreutils test you need to use `/usr/bin/test ...` or `env test ...` or `(exec test ...)`. The usual way to check if $2 is present is to use `[ $# -ge 2 ]`. For named variables, I like to use `[ "${myvar+set}" != "set" ]` (the parameter expansion `${myvar+word}` expands to "word" if myvar is set or "" if it is unset, see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...). (Often people want to treat an empty value as unset, so they just use `[ -n "$myvar" ]` to check that the value is non-empty. If you enable `set -u` to treat references to unset variables as errors, this should instead be `[ -n "${myvar-}" ]`.) Some comments on the script you linked: I highly recommend ShellCheck (https://www.shellcheck.net/ https://www.shellcheck.net/), it picks up a legitimate issue in your script: `tr -s [:alpha:] Y` should be `tr -s "[:alpha:]" Y` (otherwise it fails if the current directory contains a single-letter filename). Also, you should use `printf "%s\n" "$var"` instead of `echo "$var"` in case $var is "-n" for example.
galaxyLogic 3y ago
Can you "pipe" these Posix-Shell functions together? I read somewhere that Bash-shell functions can be?
- BirAdam 3y ago
  Many of them yes.
awaythrow98765 3y ago
If you are at this level of required complexity as in those examples, you should use a proper programming language, not shell. Half those snippets fail with spaces in the wrong place, newlines, control characters, etc. I think all such shell "magic" should come with a huge disclaimer pointing the user at better alternatives such as perl or python. And all snippets should have the necessary caveats on horribly broken and dangerous stuff such as "if this variable contains anything other than letters, your 'eval' will summon demons" in eval "hello_$var=value" and stuff...
- bloopernova 3y ago
  Yeah, I'm a diehard bash scripter but even I know to switch to Python once the script gets above a certain size/complexity. I should really just start with Python, but old habits and all that.
- throwawaaarrgh 3y ago
  Using a more advanced language just because you find shell syntax to be wacky is like using a car to get groceries because you find panniers or a backpack to be wacky. It's the use case that matters; if you're 60 meters from the store, just use your bike, or walk. There are plenty of cases in which Perl or Python will make things much more complicated than 5 lines of spooky-looking shell script. Sometimes a little mystical sorcery is what the doctor ordered.
  - atomicnumber3 3y ago
    Shell is full of ridiculous footguns though. It's like saying if the store is only 60m away (across an Indiana Jones-tier trap gauntlet) then just walk there. Remember that time bumblebee accidentally deleted everyone's /usr? Or that time steam deleted everyone's homedir? Both because of the easiest to avoid bash footguns - spaces and empty variables. My hard and fast rule that I've never regretted is - if you use a bash if statement, consider python. If you're going to do a for loop, you must instead use python. Typically as a side effect once the programmer is in python at my prompting, they find having such easy access to libraries suddenly lets them make the whole script more robust - proper argparsing is an immediate one that comes to mind. Frequently the reticence I see from people, especially juniors, is that they're worried about people thinking "haha they have to pull an entire python into their image just to run this script" or "wow they're so newbie/young that they can't even write some shell". I reassure them: don't worry, there's a reason we used Perl even back then too.
rmm 3y ago
i dont know about anyone else. but since ChatGPT, my shell game is probably the one that is most levelled up. A lot of things i would either do manually because i had forgotten shell programming, or i would have done in a seperate pyenv with 2-3 packages i can get done in about 4 minutes.
- darkteflon 3y ago
  Same. Shell and glue-scripts in Python. The activation energy required is now close to zero.
  - teddyh 3y ago
    I hope I never have to be the one to read those scripts.
    - darkteflon 3y ago
      There’s a world of difference between having GPT4 write your scripts, and writing scripts with GPT4.
    - dmd 3y ago
      Why? They're generally much clearer and well-commented than what a rushed harried sysadmin would write.
      - teddyh 3y ago
        But the lines and code snippets presumably won’t match the style of the rest of the program (or each other), so I guess that you’ll get programs which does things wildly differently from line to line.
        dmd 3y ago
        Sounds like you've never actually used it for writing code, and are basing this on how you think it works. As someone who uses it for hours a day every day for writing code ... no, it does not have that problem.
lamontcg 3y ago
What is everyone's current favorite one-liner for replacing a string in all files that match? In other words alternative implementations of this: ruby -pi -e "gsub(/Net::HTTPServerException/, 'Net::HTTPClientException')" {lib,knife}/**/*.rb
- cutler 3y ago
  I think you need `ruby -i -ple` to handle newlines properly.
- michaelcampbell 3y ago
  And the perl version from which ruby's kind of sprang. perl -pi -e 's/from/to/g' list_of_files
- arp242 3y ago
  sed -i? Not 100% portable across all implementations, but that usually doesn't matter. Some languages also have nice tools for this, such as gofmt -r; I don't know if something like that exists for Ruby.
- rascul 3y ago
  find with an -exec sed
cutler 3y ago
The Holy Grail of POSIX-portable shell scripts is irrelevant these days. Just use bash which is the default on most *nix systems.
- Koshkin 3y ago
  > *nix "*nux", you mean.
- arp242 3y ago
  I'll go a step further and say "just use zsh", which is clearly superior for scripting and installed so easily it hardly matters. (aside: I don't think bash the "the default on most *nix systems"; it's not on BSD or macOS (which does ship with a very old bash), and while it's certainly common on Linux even there it's not universal)
  - ocrow 3y ago
    Bash comes free. To get zsh I have to convince the sysadmins of the need for it. I'll stick with the (currently) lowest common denominator.
- TheAdamist 3y ago
  Debian/ubuntu use dash for sh, instead of bash, which breaks a lot of scripts lazily assuming bash is sh. Scripts that specify bash will work, assuming its installed though.
29athrowaway 3y ago
Use shellcheck to make the process of using sh easier.
rubatuga 3y ago
Is pure shell the programming language that will never die?
tandav 3y ago
I recommend shellcheck and shfmt pre-commit hooks: - repo: https://github.com/shellcheck-py/shellcheck-py rev: v0.9.0.2 hooks: - id: shellcheck args: [-e, SC2154, -e, SC1091, -e, SC1090] - repo: https://github.com/scop/pre-commit-shfmt rev: v3.6.0-2 hooks: - id: shfmt # requires Go to build args: [-i, '4', -ci, -sr, -w, -s]
1vuio0pswjnm7 3y ago
"A collection of pure POSIX sh alternatives to external processes." "Ternary Tests" Can anyone point to where in the POSIX standard ternary test are described. (NB. Pure POSIX sh is less featureful than Bash.) What am I missing. https://web.archive.org/web/20201219013931/https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://web.archive.org/web/20201219013931/https://pubs.open... Also these operators from C that the author includes. Is he suggesting these are available in POSIX sh. Quoting from the Pure Sh Bible: += Plus-Equal (Increment a variable.) -= Minus-Equal (Decrement a variable.) *= Times-Equal (Multiply a variable.) /= Slash-Equal (Divide a variable.) %= Mod-Equal (Remainder of dividing a variable.) << Bitwise Left Shift <<= Left-Shift-Equal >> Bitwise Right Shift >>= Right-Shift-Equal & Bitwise AND &= Bitwise AND-Equal | Bitwise OR |= Bitwise OR-Equal ~ Bitwise NOT ^ Bitwise XOR ^= Bitwise XOR-Equal One could probably learn from reading the POSIX standard, the Almquist shell source code and then experimenting. I also like reading other peoples' shell scripts, if they are good. Always something new to learn. Unfortunately this "Bible" did not teach me anything new.
- lmz 3y ago
  Those operators are in "2.6.4 Arithmetic Expansion" in your linked doc. See the link to "Arithmetic Precision and Operations"[1]. [1]: https://web.archive.org/web/20201219013931/https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_01_02_01 https://web.archive.org/web/20201219013931/https://pubs.open...
  - 1vuio0pswjnm7 3y ago
    Thank you. This helps. But does that mean a POSIX shell, or any other UNIX utility, must implement these operators. Using them in shell scripts makes the scripts non-portable, e.g., Almquist sh, NetBSD Almquist sh or Debian Almquist sh do not support them. Maybe the author of the "Pure POSIX Sh Bible" always runs Bash in --posix mode. Hence "Pure POSIX sh".
    - lmz 3y ago
      Well it doesn't say that arithmetic expansion is "optional" or an "extension". Maybe those shells are conforming to an older version?
asicsp 3y ago
See also: "Serious Shell Programming" https://freebsdfrau.gitbook.io/serious-shell-programming/ https://freebsdfrau.gitbook.io/serious-shell-programming/
protomikron 3y ago
Some nice stuff, but also some stuff, where things get easier if you just switch to a language with less caveats (like Python). My rule of thumb is to use Bash to write simple scripts, if they fit on the command line and have just one level of loop or if-else (lines can get very long, though ...). However for more complicated stuff I use the alternative programs (find, sed, awk, ...), as they behave more predictable. Furthermore I am not sure one should use such simplified versions, that are just wrong: is_float() { # Usage: is_float "number" # The test checks to see that the input contains # a '.'. This filters out whole numbers. [ -z "${1##*.*}" ] && printf %f "$1" >/dev/null 2>&1 } I mean having a '.' in the string does make it a non-integer, but what about other floats like 2e3?
- tlamponi 3y ago
  > where things get easier if you just switch to a language with less caveats (like Python). For general, non-domain specific things, it gets IMO easier by two main reasons 1) obviously when using tools where one has more experience with, but 2) also when the tool is more widely available, and POSIX shell is probably one of the most (by default) available interpreters with a somewhat human-friendly input mode there is. I know if I write a script or some functions in POSIX shell I can copy it and run it wherever I want (besides Windows, but don't really know anybody close to me that still uses that, especially in a server environment). Sure, if everywhere you want to run it python/perl/... is just available, or simple to install (i.e., any modern Linux or *BSD distro), then the second point hasn't that much weight anymore besides maybe for minimal Alpine Linux CTs (< 8 MB for a full fledged distro!). > I mean having a '.' in the string does make it a non-integer, but what about other floats like 2e3? They use the printf "%f" float format conversion so `printf "%f" 2e3` will work just fine, whereas using something that definitively is not a float makes printf exit with an error code as it fails to format the value as float, failing this check.
  - protomikron 3y ago
    > They use the printf "%f" float format conversion so `printf "%f" 2e3` will work just fine, whereas using something that definitively is not a float makes printf exit with an error code as it fails to format the value as float, failing this check. You are of course correct (sorry to sound like a language model), I missed the printf.
arendtio 3y ago
POSIX shell scripting is just broken. Don't get me wrong. I love doing it, but from a language design point it is abysmal. Just this week I tried to write a script to apply a custom function to a list of files. But the amount of time you spend just to make sure it works with special cases like spaces or new line characters in file names is not healthy. After all, it is probably one of the most standard scenarios. First, I thought I would use 'find -execdir' or 'find | xargs', but neither supports shell functions. So I went on to take the 'while read' road, just to learn, that POSIX 'read', does not support all options you would need to make that work with all special cases.
- JackSlateur 3y ago
  You can xargs a bash function That function must be exported before, for instance: my_function(){ ... } export -f my_function find | xargs my_function
  - arendtio 3y ago
    Yeah, I know, but 'export -f' is not POSIX compliant [1]. In addition, I have read, that this solution has quite a bad performance [2]. 1: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... 2: https://stackoverflow.com/a/67780632/1149404 https://stackoverflow.com/a/67780632/1149404
- Night_Thastus 3y ago
  Not only broken, but every single implementation of it is broken in different ways - even between those that claim to be POSIX-compliant.
ulrischa 3y ago
This should be forbidden. Terrible idea to use the Shell for more than three lines of code!
jake_morrison 3y ago
I feel the same way about "advanced" shell scripting techniques as I do about "cutting-edge" accounting techniques: someone's probably going to jail. I have written some big complex systems in shell and regretted it. My rule of thumb is that if a script needs error handling, it's time to write it in a real programming language. Tricky shell programs running as root are a real danger. It's nice to know that some of these techniques exist, but better not to use them. Many of them have to do with string manipulation, which is a sign that a proper language with data structures would be a win. Many people don't realize that perl is part of the Debian base system. If you are going to go crazy with one-liners, then that's a better tool. I would generally recommend Python, though. Avoiding bash-isms is useful so that you can run under busybox shell, reducing the size and dependencies of containers. Embedded systems often use shell. If you are tempted to install bash for more power, a better solution is probably lua. It's smaller and saner.

Pure Sh Bible

406 points by akraker 3y ago

dorfwald 3y ago

Dylan is a phenomenon. I feel like most people here would know about his projects, though.

zoover2020 3y ago
Who are they and why should we know them?
- dorfwald 3y ago
  Creator of neofetch.
  - sevg 3y ago
    And kisslinux. He seems to have gone off the grid though? Seemingly no public GitHub activity since 2021.
    - akraker 3y ago
      Unfortunately, it seems that he has dropped off the grid. Didn't realize this until you mentioned it actually. It's really too bad. Not a lot is known about why, but more information can be found here: https://www.reddit.com/r/kisslinux/comments/lsbz8n/an_update_on_dylan/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/kisslinux/comments/lsbz8n/an_update...

cde-v 3y ago

[flagged]

cduzz 3y ago

I have to take exception to some of this; it's "technically correct" but like much of shell, likely full of terrifying edge cases that are best avoided. For instance, using eval to have variables with variable names is madness. $ var="world" $ eval "hello_$var=value" $ eval printf '%s\n' "\$hello_$var" I suppose the fancy business with printf is flexing, but I suspect the majority of people scanning documents like this would prefer to have examples with echo "${variable%%string_modifier}" Escape sequences are mostly dependent on you being on a VT102 compatible terminal; I'm not sure what happens if you're on an ASR33 and you send this to it... I'd consider type checking (is this a float) to be similarly cursed witchcraft -- after all I've run into all sorts of "floats" that aren't just 123.456 strings... Overall -- there's a huge range of things shell can do that you probably should just avoid. Similarly, even if you can do it in a clever shell way (the bit shifting math for instance, or extensive string manipulation) should likely be done with external tools just because people understand "oh, awk or sed -- I know this!" instead of "what the hell is this line noise?" If you're writing performance optimized shell, well, probably put the keyboard down and walk away for a bit and reconsider your life choices. Good doc; I'll probably stick to the man page though.

Linux-Fan 3y ago
> I suppose the fancy business with printf is flexing, but I suspect the majority of people scanning documents like this would prefer to have examples with echo "${variable%%string_modifier}" POSIX itself contains some interesting notes about `echo` and `printf`, e.g. from the `printf` page: "The printf utility was added to provide functionality that has historically been provided by echo. However, due to irreconcilable differences in the various versions of echo extant, the version has few special features, leaving those to this new printf utility, which is based on one in the Ninth Edition system." The behaviour of `echo` is especially inconsistent for escape sequences. Often, `echo -e` is used to enable escape sequences for it but this is not POSIX compliant. For any "standards-perferring" shell scripting pages I'd thus think `printf` is the right way to present it if only to advertise that this builtin exists. > should likely be done with external tools just because people understand "oh, awk or sed -- I know this!" instead of "what the hell is this line noise?" I am not actually sure about this "I know this" part. I think many users of AWK only ever use it to do some sort of "access the n-th column of input" as in `echo a b c | awk '{ print $2 }'`. Similar for `sed` which is often only known and used for "replace string a by string b in input" use cases. > If you're writing performance optimized shell, well, probably put the keyboard down There is some difference between performance optimized and "hugely inefficient" that can make the difference between a task taking minutes vs. a few seconds. It may not seem much but shell scripts are often integrated into automated processes (think of build environments or system startup tasks) and there, seconds quickly accumulate. Whether this is relevant to your use case, only you can know. I found this "Pure Sh Bible" to be highly interesting, but there are some places where it has some rough edges and it may not show the "best practices" for standard use cases. Still a very useful resource.
- palunon 3y ago
  > Similar for `sed` which is often only known and used for "replace string a by string b in input" use cases. Sed has at least the vi users who may know ex command from the vi command mode. Awk needs to be learned for itself.
- cduzz 3y ago
  There are a huge variety of things you can do in shell that you simply shouldn't. If you're adventuring into a world where it matters if there's a newline at the end of your output as a result of your script running on noodlebsd or ache or freeGum, you've already lost. Don't play the game. Don't do things where it matters what the exact output of echo gives you. Did you know that you can't put a null into a shell variable? You can't. Also, you shouldn't ever be in a position to care. Did you know that you can fiddle with the "IFS" to let you parse CSVs? Don't ever do that; it works but nobody will ever understand what the hell you did. You can make an case statement and evaluate it with eval to make your own shell script that writes its own shell scripts. Please don't. All of these adventures are possible, and work fine, and I've done them and come back to the code a decade later and I both understand what I was trying to do and the code still works across a variety of bourne shell interpreters. Nevertheless, these are things that shouldn't be done except to flex.

INTPenis 3y ago

If we're going to be purists, sh is a far cry from bash. /s tongue in cheek, please don't kill me. I love bash and I use it often. I love Greg Wooledge's bash guides and all the people in #bash@libera.

chasil 3y ago
I like busybox bash (especially on Windows) which is very much not bash. Busybox bash does not implement arrays, for starters, which is a deal breaker for many scripts. The Windows kernel forks processes about 10x slower than Linux, so these tricks have real performance value for POSIX-family shells running there.
- _kst_ 3y ago
  As far as I can tell, there is no busybox bash. busybox's built-in shell is ash. There are options to enable a few bash-compatible extensions, but there is no "bash" applet.
  - chasil 3y ago
    Try the Windows version on frippery.org. I see it in the screen capture. https://frippery.org/busybox/index.html https://frippery.org/busybox/index.html
    - arp242 3y ago
      There's an option to install the "bash" applet as a link to either ash or hush, the two shells that busybox comes with. Turns out that a large number of "bash scripts" use no bash-specific features in spite of using "bash" in the #!, or only a few bash-specific features like [[ ]]. It's disabled by default, and arguably not a good idea to enable it because compatibility is not great as you found out. Either way, "busybox bash" doesn't exist: only the option to alias ash or hush to bash.
      - chasil 3y ago

juujian 3y ago

```bash lstrip() { # Usage: lstrip "string" "pattern" printf '%s\n' "${1##$2}" } ``` Got me right at the start. I look this up about once a week. Time to put it .bashrc already.

orf 3y ago

# Remove all leading white-space. trim=${1#${1%%[![:space:]]*}} # Remove all trailing white-space. trim=${trim%${trim##*[![:space:]]}} Your scientists were so preoccupied with whether they could, they didn't stop to think if they should

indigodaddy 3y ago

Before clicking, I really thought this was some kind of shell command front-end to spit out Bible verses or something. Woops.

catiopatio 3y ago
For something like that, you’d want to use fortune(1) with a Bible fortune file, e.g. - KJV: https://sourceforge.net/projects/fortunebible/ https://sourceforge.net/projects/fortunebible/ - NIV: https://github.com/optio50/Bible-NIV-Fortune https://github.com/optio50/Bible-NIV-Fortune
rubatuga 3y ago
You've been watching too many Terry Davis videos

chasil 3y ago

Problems with this advice: ::looping over a file The "loop over the contents of a file" entry does not protect stdin. Use an alternate descriptor. while read -r line <&9 do printf '%s\n' "$line" done 9< "$file" Particularly problematic is ssh, that will pass $line to any remote command that is able to consume it. The looped commands will receive stdin of the loop when an alternate descriptor is used. I use 9 by habit after reading the flock manual page many years ago. ::EXIT trap dash will only call the EXIT trap on a normal exit. Add INT (and any other signal that you want from "kill -l") if you also want to catch abnormal terminations. Windows busybox sh/bash only catches regular exits, no matter what else you add.

ndsipa_pomu 3y ago
You might also want to clear IFS if you don't want to trim leading/trailing whitespace (taken from https://mywiki.wooledge.org/BashFAQ/001 https://mywiki.wooledge.org/BashFAQ/001): while IFS= read -r line <&9; do cat > ignoredfile printf '%s\n' "$line" done 9< "$file" ("cat" is used as an example of a command eating stdin)

djha-skin 3y ago

Mildly annoyed when people constantly refer to their article as the Bible of X. For those of us who are religious, it is in poor taste.

jjgreen 3y ago
Fair point
catiopatio 3y ago
Even as someone who grew up in a very religious household, I am still struggling to come up with a “steel-man” argument on your behalf. Can you explain why you believe it is in poor taste?
- comprev 3y ago
  How common is the expression "Qur'an of X" instead of Bible?
  - burnished 3y ago
    What? You appear to be saying nonsense.
  - tyingq 3y ago
    The "Torah of X" is fairly common.
    - einpoklum 3y ago
      But in Hebrew, "The Torah" doesn't mean a certain book anyway. There are "The five one-part-of-five's of Torah". The Torah is like the teachings, or the postulates, or the theory. So there's Jehova's Torah, or Torat Hashem; but there's also your mother's torah: "Remember, my son, your father's Mussar (= mores), and do not abandon your mother's Torah (= teachings)".
      - tyingq 3y ago

tpoacher 3y ago

Regarding conditional expressions: I found something neat recently. The coreutils version of `test` doesn't have this, and when you use `test`, typically what you're using is the coreutils one. But if you use `builtin test` to force the bash builtin variant of `test`, this has a nice `-v` switch, which allows you to check if a variable is set. I found out about this recently when I had to use it in my bash argument processing library, to check if an option expecting a value had not been provided a value (since checking for an empty variable instead would mean that an actual empty string would also be ignored). (see here: https://git.sr.ht/~tpapastylianou/process_optargs/tree/main/item/sourceable_functions/process_optargs#L464 https://git.sr.ht/~tpapastylianou/process_optargs/tree/main/...)

teddyh 3y ago
If you already assume bash, then you don’t need the “-v” option to the “test” builtin; just do if [ "${foo+some_string}" = "" ]; then echo "foo is unset" fi or, in your case: if test "${2+x}" = "" # i.e. if $2 is not set
- tpoacher 3y ago
  True, but I consider this very hacky, error-prone, and unnecessary when a clear, bespoke test flag exists. Also, if you prefer the "[ ... ]" syntax for testing, then you can use `-v` directly anyway (since that is equivalent to the builtin test keyword anyway).
- hddqsb 3y ago
  In fact `${foo+some_string}` is supported by POSIX sh (unlike `test -v`).
hddqsb 3y ago
Note that `test -v` is not in POSIX sh (see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...). And you don't need `builtin` because `test ...` invokes the builtin by default; if you want to run coreutils test you need to use `/usr/bin/test ...` or `env test ...` or `(exec test ...)`. The usual way to check if $2 is present is to use `[ $# -ge 2 ]`. For named variables, I like to use `[ "${myvar+set}" != "set" ]` (the parameter expansion `${myvar+word}` expands to "word" if myvar is set or "" if it is unset, see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...). (Often people want to treat an empty value as unset, so they just use `[ -n "$myvar" ]` to check that the value is non-empty. If you enable `set -u` to treat references to unset variables as errors, this should instead be `[ -n "${myvar-}" ]`.) Some comments on the script you linked: I highly recommend ShellCheck (https://www.shellcheck.net/ https://www.shellcheck.net/), it picks up a legitimate issue in your script: `tr -s [:alpha:] Y` should be `tr -s "[:alpha:]" Y` (otherwise it fails if the current directory contains a single-letter filename). Also, you should use `printf "%s\n" "$var"` instead of `echo "$var"` in case $var is "-n" for example.

galaxyLogic 3y ago

Can you "pipe" these Posix-Shell functions together? I read somewhere that Bash-shell functions can be?

BirAdam 3y ago
Many of them yes.

awaythrow98765 3y ago

If you are at this level of required complexity as in those examples, you should use a proper programming language, not shell. Half those snippets fail with spaces in the wrong place, newlines, control characters, etc. I think all such shell "magic" should come with a huge disclaimer pointing the user at better alternatives such as perl or python. And all snippets should have the necessary caveats on horribly broken and dangerous stuff such as "if this variable contains anything other than letters, your 'eval' will summon demons" in eval "hello_$var=value" and stuff...

bloopernova 3y ago
Yeah, I'm a diehard bash scripter but even I know to switch to Python once the script gets above a certain size/complexity. I should really just start with Python, but old habits and all that.
throwawaaarrgh 3y ago
Using a more advanced language just because you find shell syntax to be wacky is like using a car to get groceries because you find panniers or a backpack to be wacky. It's the use case that matters; if you're 60 meters from the store, just use your bike, or walk. There are plenty of cases in which Perl or Python will make things much more complicated than 5 lines of spooky-looking shell script. Sometimes a little mystical sorcery is what the doctor ordered.
- atomicnumber3 3y ago
  Shell is full of ridiculous footguns though. It's like saying if the store is only 60m away (across an Indiana Jones-tier trap gauntlet) then just walk there. Remember that time bumblebee accidentally deleted everyone's /usr? Or that time steam deleted everyone's homedir? Both because of the easiest to avoid bash footguns - spaces and empty variables. My hard and fast rule that I've never regretted is - if you use a bash if statement, consider python. If you're going to do a for loop, you must instead use python. Typically as a side effect once the programmer is in python at my prompting, they find having such easy access to libraries suddenly lets them make the whole script more robust - proper argparsing is an immediate one that comes to mind. Frequently the reticence I see from people, especially juniors, is that they're worried about people thinking "haha they have to pull an entire python into their image just to run this script" or "wow they're so newbie/young that they can't even write some shell". I reassure them: don't worry, there's a reason we used Perl even back then too.

rmm 3y ago

i dont know about anyone else. but since ChatGPT, my shell game is probably the one that is most levelled up. A lot of things i would either do manually because i had forgotten shell programming, or i would have done in a seperate pyenv with 2-3 packages i can get done in about 4 minutes.

darkteflon 3y ago
Same. Shell and glue-scripts in Python. The activation energy required is now close to zero.
- teddyh 3y ago
  I hope I never have to be the one to read those scripts.
  - darkteflon 3y ago
    There’s a world of difference between having GPT4 write your scripts, and writing scripts with GPT4.
  - dmd 3y ago
    Why? They're generally much clearer and well-commented than what a rushed harried sysadmin would write.
    - teddyh 3y ago
      But the lines and code snippets presumably won’t match the style of the rest of the program (or each other), so I guess that you’ll get programs which does things wildly differently from line to line.
      - dmd 3y ago
        Sounds like you've never actually used it for writing code, and are basing this on how you think it works. As someone who uses it for hours a day every day for writing code ... no, it does not have that problem.

lamontcg 3y ago

What is everyone's current favorite one-liner for replacing a string in all files that match? In other words alternative implementations of this: ruby -pi -e "gsub(/Net::HTTPServerException/, 'Net::HTTPClientException')" {lib,knife}/**/*.rb

cutler 3y ago
I think you need `ruby -i -ple` to handle newlines properly.
michaelcampbell 3y ago
And the perl version from which ruby's kind of sprang. perl -pi -e 's/from/to/g' list_of_files
arp242 3y ago
sed -i? Not 100% portable across all implementations, but that usually doesn't matter. Some languages also have nice tools for this, such as gofmt -r; I don't know if something like that exists for Ruby.
rascul 3y ago
find with an -exec sed

cutler 3y ago

The Holy Grail of POSIX-portable shell scripts is irrelevant these days. Just use bash which is the default on most *nix systems.

Koshkin 3y ago
> *nix "*nux", you mean.
arp242 3y ago
I'll go a step further and say "just use zsh", which is clearly superior for scripting and installed so easily it hardly matters. (aside: I don't think bash the "the default on most *nix systems"; it's not on BSD or macOS (which does ship with a very old bash), and while it's certainly common on Linux even there it's not universal)
- ocrow 3y ago
  Bash comes free. To get zsh I have to convince the sysadmins of the need for it. I'll stick with the (currently) lowest common denominator.
TheAdamist 3y ago
Debian/ubuntu use dash for sh, instead of bash, which breaks a lot of scripts lazily assuming bash is sh. Scripts that specify bash will work, assuming its installed though.

29athrowaway 3y ago

Use shellcheck to make the process of using sh easier.

rubatuga 3y ago

Is pure shell the programming language that will never die?

tandav 3y ago

I recommend shellcheck and shfmt pre-commit hooks: - repo: https://github.com/shellcheck-py/shellcheck-py rev: v0.9.0.2 hooks: - id: shellcheck args: [-e, SC2154, -e, SC1091, -e, SC1090] - repo: https://github.com/scop/pre-commit-shfmt rev: v3.6.0-2 hooks: - id: shfmt # requires Go to build args: [-i, '4', -ci, -sr, -w, -s]

1vuio0pswjnm7 3y ago

"A collection of pure POSIX sh alternatives to external processes." "Ternary Tests" Can anyone point to where in the POSIX standard ternary test are described. (NB. Pure POSIX sh is less featureful than Bash.) What am I missing. https://web.archive.org/web/20201219013931/https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://web.archive.org/web/20201219013931/https://pubs.open... Also these operators from C that the author includes. Is he suggesting these are available in POSIX sh. Quoting from the Pure Sh Bible: += Plus-Equal (Increment a variable.) -= Minus-Equal (Decrement a variable.) *= Times-Equal (Multiply a variable.) /= Slash-Equal (Divide a variable.) %= Mod-Equal (Remainder of dividing a variable.) << Bitwise Left Shift <<= Left-Shift-Equal >> Bitwise Right Shift >>= Right-Shift-Equal & Bitwise AND &= Bitwise AND-Equal | Bitwise OR |= Bitwise OR-Equal ~ Bitwise NOT ^ Bitwise XOR ^= Bitwise XOR-Equal One could probably learn from reading the POSIX standard, the Almquist shell source code and then experimenting. I also like reading other peoples' shell scripts, if they are good. Always something new to learn. Unfortunately this "Bible" did not teach me anything new.

lmz 3y ago
Those operators are in "2.6.4 Arithmetic Expansion" in your linked doc. See the link to "Arithmetic Precision and Operations"[1]. [1]: https://web.archive.org/web/20201219013931/https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_01_02_01 https://web.archive.org/web/20201219013931/https://pubs.open...
- 1vuio0pswjnm7 3y ago
  Thank you. This helps. But does that mean a POSIX shell, or any other UNIX utility, must implement these operators. Using them in shell scripts makes the scripts non-portable, e.g., Almquist sh, NetBSD Almquist sh or Debian Almquist sh do not support them. Maybe the author of the "Pure POSIX Sh Bible" always runs Bash in --posix mode. Hence "Pure POSIX sh".
  - lmz 3y ago
    Well it doesn't say that arithmetic expansion is "optional" or an "extension". Maybe those shells are conforming to an older version?

asicsp 3y ago

See also: "Serious Shell Programming" https://freebsdfrau.gitbook.io/serious-shell-programming/ https://freebsdfrau.gitbook.io/serious-shell-programming/

protomikron 3y ago

Some nice stuff, but also some stuff, where things get easier if you just switch to a language with less caveats (like Python). My rule of thumb is to use Bash to write simple scripts, if they fit on the command line and have just one level of loop or if-else (lines can get very long, though ...). However for more complicated stuff I use the alternative programs (find, sed, awk, ...), as they behave more predictable. Furthermore I am not sure one should use such simplified versions, that are just wrong: is_float() { # Usage: is_float "number" # The test checks to see that the input contains # a '.'. This filters out whole numbers. [ -z "${1##*.*}" ] && printf %f "$1" >/dev/null 2>&1 } I mean having a '.' in the string does make it a non-integer, but what about other floats like 2e3?

tlamponi 3y ago
> where things get easier if you just switch to a language with less caveats (like Python). For general, non-domain specific things, it gets IMO easier by two main reasons 1) obviously when using tools where one has more experience with, but 2) also when the tool is more widely available, and POSIX shell is probably one of the most (by default) available interpreters with a somewhat human-friendly input mode there is. I know if I write a script or some functions in POSIX shell I can copy it and run it wherever I want (besides Windows, but don't really know anybody close to me that still uses that, especially in a server environment). Sure, if everywhere you want to run it python/perl/... is just available, or simple to install (i.e., any modern Linux or *BSD distro), then the second point hasn't that much weight anymore besides maybe for minimal Alpine Linux CTs (< 8 MB for a full fledged distro!). > I mean having a '.' in the string does make it a non-integer, but what about other floats like 2e3? They use the printf "%f" float format conversion so `printf "%f" 2e3` will work just fine, whereas using something that definitively is not a float makes printf exit with an error code as it fails to format the value as float, failing this check.
- protomikron 3y ago
  > They use the printf "%f" float format conversion so `printf "%f" 2e3` will work just fine, whereas using something that definitively is not a float makes printf exit with an error code as it fails to format the value as float, failing this check. You are of course correct (sorry to sound like a language model), I missed the printf.

arendtio 3y ago

POSIX shell scripting is just broken. Don't get me wrong. I love doing it, but from a language design point it is abysmal. Just this week I tried to write a script to apply a custom function to a list of files. But the amount of time you spend just to make sure it works with special cases like spaces or new line characters in file names is not healthy. After all, it is probably one of the most standard scenarios. First, I thought I would use 'find -execdir' or 'find | xargs', but neither supports shell functions. So I went on to take the 'while read' road, just to learn, that POSIX 'read', does not support all options you would need to make that work with all special cases.

JackSlateur 3y ago
You can xargs a bash function That function must be exported before, for instance: my_function(){ ... } export -f my_function find | xargs my_function
- arendtio 3y ago
  Yeah, I know, but 'export -f' is not POSIX compliant [1]. In addition, I have read, that this solution has quite a bad performance [2]. 1: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... 2: https://stackoverflow.com/a/67780632/1149404 https://stackoverflow.com/a/67780632/1149404
Night_Thastus 3y ago
Not only broken, but every single implementation of it is broken in different ways - even between those that claim to be POSIX-compliant.

ulrischa 3y ago

This should be forbidden. Terrible idea to use the Shell for more than three lines of code!

jake_morrison 3y ago

I feel the same way about "advanced" shell scripting techniques as I do about "cutting-edge" accounting techniques: someone's probably going to jail. I have written some big complex systems in shell and regretted it. My rule of thumb is that if a script needs error handling, it's time to write it in a real programming language. Tricky shell programs running as root are a real danger. It's nice to know that some of these techniques exist, but better not to use them. Many of them have to do with string manipulation, which is a sign that a proper language with data structures would be a win. Many people don't realize that perl is part of the Debian base system. If you are going to go crazy with one-liners, then that's a better tool. I would generally recommend Python, though. Avoiding bash-isms is useful so that you can run under busybox shell, reducing the size and dependencies of containers. Embedded systems often use shell. If you are tempted to install bash for more power, a better solution is probably lua. It's smaller and saner.