5 ms·

POSIX Is Not a Shell

56 points by gaigalas 4mo ago

XYen0n 4mo ago
Usually I prefer to use `#!/usr/bin/env bash`.
echoangle 4mo ago
Pretty bad argument. If it’s not defined by POSIX, it’s not POSIX compatible if you rely on a specific behavior. If you only use defined behavior and it works, it is compatible. It’s like saying C99 isn’t a compiler. True, but you can still write C99 code, right?
- smitty1e 4mo ago
  > C99 isn’t a compiler. Sure, but the pojt here is that if we say "Write in X" we generally understand it to mean "Treat X like a standard and don't get too colloquial with the stylings." Pedantry is worthwhile, but it can be a diminishing returns game.
  - eqvinox 4mo ago
    Feels like you missed the point. On the example of 'echo \n' - it's not defined in POSIX, therefore a script written in "POSIX shell" must simply never hit that case. TFA kinda implies you can't target POSIX shell. That's silly, of course you can. The question is, what tools are there to check for compliance. Whether running on 14 shells is a good such tool - idk. Something specifically searching for POSIX violations might be better.
    - Joker_vD 4mo ago
      Well, with C language it's pretty much the same. You are supposed to "just" never write (or rather, most of the time, to just not execute) anything that is UB. And lots and lots of people to this day continue to believe that can do this (most of the time, they're wrong).
      - eqvinox 4mo ago
        UB (specified to be undefined) and 'plain' unspecified are not the same thing.
      - 1718627440 4mo ago
        And when they actually write code in the C dialect of there compiler or check the assembly, this is fine.
    - gaigalas 4mo ago
      The spec is not that good. `local` for example is present in many shells (almost all of them), but they decided to leave it out uniquely because of ksh93 (scope is different). It became undefined behavior. When the spec was written, ksh was important. Since then, it has only been revised but not updated and I consider it to be obsolete. So, if you follow POSIX strictly, you then lose local scope on functions, which is more likely to cause bugs and hard to catch with a linter like you suggested. You're left with a broken feature set (on many other angles too) that is not actually practical. Even spellcheck makes concessions.
      - echoangle 4mo ago
        I don’t get this point either. If local is not in the POSIX spec, I guess you can’t use it if you want to be POSIX compatible. Just because many shells do it doesn’t mean it’s POSIX.
        gaigalas 4mo ago
        You can target POSIX if you want to, but doing that doesn't guarantee shell scripts will work. The blog post stresses this, the difference between POSIX and portability. If you want portability, testing is better for now. One of the goals of these projects is to more precisely capture a retrospec (what actually works, not what was specified), it's the same thing they did with HTML5.
        echoangle 4mo ago
        > but doing that doesn't guarantee shell scripts will work. How so? Can you give an example that should work according to the POSIX spec but doesn’t in a POSIX compliant shell? The example in the post is just some behavior the POSIX spec doesn’t specify, so obviously you wouldn’t be allowed to do that when targeting POSIX.
        rascul 4mo ago
        Not exactly what you asked for, but there's ambiguity in the spec leading to different implementations. https://stackoverflow.com/questions/11376975/is-there-a-minimally-posix-2-compliant-shell/16376043#16376043 https://stackoverflow.com/questions/11376975/is-there-a-mini...
Muhammad523 4mo ago
This post is nice: the writer first explains a problem, using a simple example. In the next section, they reflect a bit about the problem, and then they casually mention two tools they built. In my opinion, this is amazing: you sponsor you project, while also making the problem it solves clear: use their tool to test how portable your code is
- microgpt 4mo ago
  [dead]
jmclnx 4mo ago
Will not build without docker, so I am out of luck. This tells me this is not portable, even to some Linuxes.
- Joker_vD 4mo ago
  Strict POSIX conformance is arguably worse. I mean, have you seen what it advises for shebangs? First of all: The shell reads its input from a file (see sh), from the -c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-2017. If the first line of a file of shell commands starts with the characters "#!", the results are unspecified. Ah, so shebangs are not required to be supported, already a great start. Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in. [...] Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed). For example: # # Installation time script to install correct POSIX shell pathname # # Get list of paths to check # Sifs=$IFS Sifs_set=${IFS+y} IFS=: set -- $(getconf PATH) if [ "$Sifs_set" = y ] then IFS=$Sifs else unset IFS fi # # Check each path for 'sh' # for i do if [ -x "${i}"/sh ] then Pshell=${i}/sh fi done # # This is the list of scripts to update. They should be of the # form '${name}.source' and will be transformed to '${name}'. # Each script should begin: # # #!INSTALLSHELLPATH # scripts="a b c" # # Transform each script # for i in ${scripts} do sed -e "s|INSTALLSHELLPATH|${Pshell}|" < ${i}.source > ${i} done Marvelous. What a robust foundation of useful and hard-to-misuse utilities.
- gaigalas 4mo ago
  Author here. It definitely builds outside docker. It's a musl-cross-make toolchain, you can procure the dependencies locally if you don't like the Docker recipes. Feel free to open an issue if you feel like that's a challenge. Likely, you can get it to work but checksum reproducibility will be hard without a controlled environment like docker.
sdovan1 4mo ago
If your environment is POSIX, testing scripts with tool written in POSIX shell, like shellspec[1], might also be a choice. [1] https://shellspec.info/ https://shellspec.info/
paulddraper 4mo ago
> When someone says "write it in POSIX shell for portability," they mean well. > POSIX is a specification. Not a program. The thing that actually runs your script is bash, dash, ash, ksh, yash, or one of a dozen others. “When someone says ‘write it in ECMAScript,’ they mean well. “ECMAScript is specification. Not a program. The thing that actually runs your script is Node.js, Bun, Deno, Rhino, or one of a dozen others.” See how silly that sounds?
- gaigalas 4mo ago
  No one says "Let's write ECMAScript for portability". If someone did, I would probably be writing about that too.
  - hnfong 4mo ago
    Hey guys... I have a wonderful idea. ... Let's write ECMAScript for portability.
    - gaigalas 4mo ago
      Can you imagine? Using that fancy new gen JavaScript today? Oh wait, that's exactly babel's headline! It's just so old now that people don't need to say what it does. Actually funny that the spec drift got solved by polyfills and transpilers.
  - paulddraper 4mo ago
    Are you serious? People say that all the time. People use JavaScript everywhere.
    - gaigalas 4mo ago
      Nope, implementors are worried about shaping the spec (which iterates much faster than POSIX) and the overall community around JS has plenty of portability solutions. That simply doesn't happen with portable shell script. There's a lot of misconceptions (people can't distinguish between shell/terminal, shell/coreutils, etc), implementors don't talk to each other. "People use JavaScript everywhere" means nothing for this discussion, it's about whether the ecosystem grasps fundamental portability questions. The JS ecosystem does (now, after some decades), the shell one does not. We have babel, core-js and many other libs. It's just a bunch of trick compilations (Ubuntu's dash vs bash page, Rich's shell tricks), nothing actually sturdy. You can try to bend the discussion to make them closer, but it doesn't survive a critical analysis.
JdeBP 4mo ago
Stéphane Chazelas would agree, * https://unix.stackexchange.com/a/496642/5132 https://unix.stackexchange.com/a/496642/5132 because the problem here is educating people with slipshod ideas about 'sh' being 'the POSIX shell' or (worse) 'the Bourne shell'. Both M. Chazelas and M. Gaigalas are making the point that 'sh' is a language that one aims to write in, not one of the many programs that sort of, sometimes, if invoked in the right way, implement that language; and a subordinate point that people are generally very poor about doing that when yet they insist that they are writing 'POSIX shell script'. Fun facts: Standardization led by existing practice is not a simple process. * https://unix.stackexchange.com/a/493743/5132 https://unix.stackexchange.com/a/493743/5132 POSIX/SUS standardization is not a static thing. The long discussed thorny issue of echo has now subtly changed from all of those explanations given about it over the years, because in 2024 the standard was changed. * https://pubs.opengroup.org/onlinepubs/9799919799/utilities/echo.html https://pubs.opengroup.org/onlinepubs/9799919799/utilities/e... M. Chazelas's own quite famous 2013 StackExchange answer on the subject has not yet been updated with the change that now incorporates -e and -E into the rule. Amusingly, it was M. Chazelas that raised defect 1222 that caused this change. * https://unix.stackexchange.com/a/65819/5132 https://unix.stackexchange.com/a/65819/5132 * https://www.austingroupbugs.net/view.php?id=1222 https://www.austingroupbugs.net/view.php?id=1222
- wpollock 4mo ago
  Stéphane (not "M") Chazelas is a genius. I have written him several times over the past 30 years, encouraging him to write a book about shell, POSIX, and other topics that he is an expert in. He writes very well too. I fear we will all be the poorer if/when he retires.
  - rmunn 4mo ago
    "M." is the French abbreviation for "monsieur", the equivalent of "Mr." in English. So "M. Chazelas" is the correct way to politely refer to Stéphane Chazelas by his last name, since he's French. (Not that writing "Mr. Chazalas" would be wrong either when writing in English, but "M. Chazelas" is entirely correct).
kevin_thibedeau 4mo ago
Solaris most definitely has a POSIX shell at /usr/xpg4/bin/sh which behaves differently than OG Bourne at /bin/sh.
- gaigalas 4mo ago
  You can run this pup to find out what's underneath! https://www.in-ulm.de/~mascheck/various/whatshell/ https://www.in-ulm.de/~mascheck/various/whatshell/
qudat 4mo ago
This is quite a timely post compared to: https://bower.sh/posix-shell-is-all-you-need https://bower.sh/posix-shell-is-all-you-need This post was thought provoking, I wonder, is the hidden argument here that the posix spec for a shell is not well specified if there is so much variance between the implementations? Or is the fundamental issue simply a matter of history? Both?
- gaigalas 4mo ago
  Most shell families precede the spec. Both bash and ksh had features at the time the spec was written that are not in it. It's weird. My main point is that following the spec doesn't guarantee shell scripts will be portable, which is a common misconception.
  - qudat 4mo ago
    > My main point is that following the spec doesn't guarantee shell scripts will be portable, which is a common misconception. That feels like a failure in the spec. Your example illustrates it: echo has unspecified behavior that literally prevents it from being portable. Is it possible portability is just not a feature of posix?
    - gaigalas 4mo ago
      I honestly don't know! What I know is that there is an intersection of features that work with polyfills and tweaks in all shells. That intersection has a lot of the spec, but also has a lot that is not specified anywhere. I started to build something here 12 years ago: https://github.com/Mosai/workshop https://github.com/Mosai/workshop. Then here 2 years ago: https://github.com/alganet/coral https://github.com/alganet/coral Now I'm attacking it from another angle, trying to document what I learned methodically and share the tools I made for me with everyone else.
    - SAI_Peregrinus 4mo ago
      POSIX specifies a common subset of functionality of several different shells (and C libraries) that existed when it was created, and that any compliant shell (or C library) will share. Some functions like `echo` are guaranteed to be present but don't have fully-specified behavior, because the shells that POSIX was created to be the common functional subset of had different behaviors when POSIX was created. POSIX doesn't guarantee portability. POSIX only guarantees that the things it specifies will work in a POSIX-compliant environment. POSIX makes no guarantees about portability to non-POSIX-compliant environments like Windows' shells or MSVC, the Fish shell, etc. POSIX makes no guarantees about the things it doesn't specify.
croemer 4mo ago
Inconsistent: claims /bin/sh on Alpine is dash in one paragraph, BusyBox ash in another. Smells a lot of AI writing.
- gaigalas 4mo ago
  They are the same, kinda! ash and dash are from the Almquist family. https://en.wikipedia.org/wiki/Almquist_shell https://en.wikipedia.org/wiki/Almquist_shell It's just poetic license.
  - croemer 4mo ago
    They are not the same. For example, `[[` apparently works on Alpine's ash but not on Debian's dash. BusyBox ash has some bash compatibility options enabled in Alpine. It's ironic that a post about shell differences glosses over this. Maybe not surprising that slop is sloppy.
    - gaigalas 4mo ago
      The post is not about shell differences, there's too many of them. It's about portability. I want users to test instead of rote memorizing. And I'm giving them tools: docker run --rm alganet/shell-versions:all -c 'PATH=; [[ 1 == 1 ]] || echo fail' Don't trust any blogger, use science.
      - croemer 4mo ago
        The tool is handy, managed to confirm that dash and ash behave differently. But why should I trust your tool if your writing is wrong?
        gaigalas 4mo ago
        You shouldn't. You should audit the tool. It's open source, and I offer build checksums for reproducibility, it's made to be auditable.
chasil 4mo ago
The author has neglected the pdksh family of shells which includes all of Android, so this is a sizable oversight. This would include the MirBSD mksh, and oksh from OpenBSD. They are much smaller than ksh93 and bash, and I would suggest that their syntax extensions be given preference, as they originally represented ksh88 (from which the POSIX shell standard was derived).
- gaigalas 4mo ago
  These are all in the shell-versions toolkit I recommended. docker run -it --rm alganet/shell-versions:all /opt/mksh_R59c/bin/mksh -c 'echo $KSH_VERSION' docker run -it --rm alganet/shell-versions:all /opt/oksh_7.9/bin/oksh -c 'echo $KSH_VERSION' docker run -it --rm alganet/shell-versions:all /opt/loksh_7.9/bin/loksh -c 'echo $KSH_VERSION' I omitted them from the post because this is not about the history of ksh and the pdksh codebase.
  - chasil 4mo ago
    The POSIX commentary mentions the Korn shell ten times, including particular behavior of the 1988 ksh release. Bash is not mentioned. It is easier to understand the POSIX standard with a ksh focus, particularly ksh88. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
    - gaigalas 4mo ago
      I particularly like ksh93. It's a virtual machine, which makes it super fast for a lot of scenarios. The ksh family is my personal favorite, in particular mksh and ksh93. Both with excelent feature sets. I even made a platformer game that works on them https://github.com/alganet/tuish/blob/main/examples/game.sh https://github.com/alganet/tuish/blob/main/examples/game.sh (and zsh/bash/busybox too) to show how feature-complete they are. Shell interpreters are such a broad subject, I could go all day talking about cool things that can be done with them. I want to get people focusing less on the spec. It's for whoever implements interpreters, not people who write scripts. And there's a gap on that, which I'm trying to cover (with full ksh support, much more than you think).
chubot 4mo ago
FWIW Oils has an option to prevent the ambiguity: osh-0.37$ echo "c:\new" c:\new osh-0.37$ shopt --set no_parse_backslash osh-0.37$ echo "c:\new" echo "c:\new" ^ [ interactive ]:6: Invalid char escape in double quoted string (OILS-ERR-12) It really should be echo "c:\\new" # with two backslashes That is an unambiguous program that works in every shell. In a well-written shell program, the only things that should follow a single backslash in a double quoted string are \ " ` $ (Although I found that this option is only on in ysh, not in shopt --set strict:all ... arguably that should be changed) Nine Reasons to Use OSH - https://oils.pub/osh.html https://oils.pub/osh.html
- gaigalas 4mo ago
  yash also has something similar, ECHO_STYLE var. https://magicant.github.io/yash/doc/_echo.html https://magicant.github.io/yash/doc/_echo.html Additionally, the ksh family has `print`, which solves some of the issues. And the story repeats all over again, each shell solving the problem in a different way :) That's one of the main reasons I went for solving the portability problem from the scripting side, not the interpreter side.
- dzaima 4mo ago
  That seems to be be an entirely-different question - `echo "c:\\new"` still differs in behavior between bash and dash - dash parses backslashes in both the double-quoted string, and then echo does another backslash parsing pass, still printing a newline; whereas bash prints a backslash + n.
  - chubot 4mo ago
    OK interesting, yeah I think dash is just plain broken ... $ dash -c 'echo "c:\\new"' c: ew $ busybox ash -c 'echo "c:\\new"' c:\new It requires 4 backslashes: $ dash -c 'echo "c:\\\\new"' c:\new $ busybox ash -c 'echo "c:\\\\new"' c:\\new I am not sure this is a matter of "undefined behavior in POSIX" -- I think it might just be dash being wildly unconformant, which I have seen in other cases. It's one of the least POSIX compliant shells. It is derived from the same codebase as busybox ash, but busybox receives more maintenance. --- In any case, it is pretty sad that sh is in such poor shape than the default /bin/sh on Debian has different behavior in this basic case. I built OSH to be a set of semantics agreed upon by many shells. I don't think any cross-shell test suites like our spec tests had existed in the past - https://oils.pub/release/0.37.0/quality.html https://oils.pub/release/0.37.0/quality.html But there is little coordination among shell authors, and no real motivation to fix the gaps. In contrast, there is A LOT of coordination among JavaScript engine authors, mostly because there are people paid to work on them.
hulitu 4mo ago
> POSIX is a specification. Not a program. The thing that actually runs your script is bash, dash, ash, ksh, yash, or one of a dozen others. They each implement POSIX with their own gaps, extensions, and historical accidents. Blaming POSIX, because a program does not implement it, is childish.
- gaigalas 4mo ago
  Have you ever tried to implement it?

56 points by gaigalas 4mo ago

Usually I prefer to use `#!/usr/bin/env bash`.

Pretty bad argument. If it’s not defined by POSIX, it’s not POSIX compatible if you rely on a specific behavior. If you only use defined behavior and it works, it is compatible. It’s like saying C99 isn’t a compiler. True, but you can still write C99 code, right?

smitty1e 4mo ago
> C99 isn’t a compiler. Sure, but the pojt here is that if we say "Write in X" we generally understand it to mean "Treat X like a standard and don't get too colloquial with the stylings." Pedantry is worthwhile, but it can be a diminishing returns game.
- eqvinox 4mo ago
  Feels like you missed the point. On the example of 'echo \n' - it's not defined in POSIX, therefore a script written in "POSIX shell" must simply never hit that case. TFA kinda implies you can't target POSIX shell. That's silly, of course you can. The question is, what tools are there to check for compliance. Whether running on 14 shells is a good such tool - idk. Something specifically searching for POSIX violations might be better.
  - Joker_vD 4mo ago
    Well, with C language it's pretty much the same. You are supposed to "just" never write (or rather, most of the time, to just not execute) anything that is UB. And lots and lots of people to this day continue to believe that can do this (most of the time, they're wrong).
    - eqvinox 4mo ago
      UB (specified to be undefined) and 'plain' unspecified are not the same thing.
    - 1718627440 4mo ago
      And when they actually write code in the C dialect of there compiler or check the assembly, this is fine.
  - gaigalas 4mo ago
    The spec is not that good. `local` for example is present in many shells (almost all of them), but they decided to leave it out uniquely because of ksh93 (scope is different). It became undefined behavior. When the spec was written, ksh was important. Since then, it has only been revised but not updated and I consider it to be obsolete. So, if you follow POSIX strictly, you then lose local scope on functions, which is more likely to cause bugs and hard to catch with a linter like you suggested. You're left with a broken feature set (on many other angles too) that is not actually practical. Even spellcheck makes concessions.
    - echoangle 4mo ago
      I don’t get this point either. If local is not in the POSIX spec, I guess you can’t use it if you want to be POSIX compatible. Just because many shells do it doesn’t mean it’s POSIX.
      - gaigalas 4mo ago
        You can target POSIX if you want to, but doing that doesn't guarantee shell scripts will work. The blog post stresses this, the difference between POSIX and portability. If you want portability, testing is better for now. One of the goals of these projects is to more precisely capture a retrospec (what actually works, not what was specified), it's the same thing they did with HTML5.
        echoangle 4mo ago
        > but doing that doesn't guarantee shell scripts will work. How so? Can you give an example that should work according to the POSIX spec but doesn’t in a POSIX compliant shell? The example in the post is just some behavior the POSIX spec doesn’t specify, so obviously you wouldn’t be allowed to do that when targeting POSIX.
        rascul 4mo ago
        Not exactly what you asked for, but there's ambiguity in the spec leading to different implementations. https://stackoverflow.com/questions/11376975/is-there-a-minimally-posix-2-compliant-shell/16376043#16376043 https://stackoverflow.com/questions/11376975/is-there-a-mini...

Muhammad523 4mo ago

This post is nice: the writer first explains a problem, using a simple example. In the next section, they reflect a bit about the problem, and then they casually mention two tools they built. In my opinion, this is amazing: you sponsor you project, while also making the problem it solves clear: use their tool to test how portable your code is

microgpt 4mo ago
[dead]

jmclnx 4mo ago

Will not build without docker, so I am out of luck. This tells me this is not portable, even to some Linuxes.

Joker_vD 4mo ago
Strict POSIX conformance is arguably worse. I mean, have you seen what it advises for shebangs? First of all: The shell reads its input from a file (see sh), from the -c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-2017. If the first line of a file of shell commands starts with the characters "#!", the results are unspecified. Ah, so shebangs are not required to be supported, already a great start. Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in. [...] Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed). For example: # # Installation time script to install correct POSIX shell pathname # # Get list of paths to check # Sifs=$IFS Sifs_set=${IFS+y} IFS=: set -- $(getconf PATH) if [ "$Sifs_set" = y ] then IFS=$Sifs else unset IFS fi # # Check each path for 'sh' # for i do if [ -x "${i}"/sh ] then Pshell=${i}/sh fi done # # This is the list of scripts to update. They should be of the # form '${name}.source' and will be transformed to '${name}'. # Each script should begin: # # #!INSTALLSHELLPATH # scripts="a b c" # # Transform each script # for i in ${scripts} do sed -e "s|INSTALLSHELLPATH|${Pshell}|" < ${i}.source > ${i} done Marvelous. What a robust foundation of useful and hard-to-misuse utilities.
gaigalas 4mo ago
Author here. It definitely builds outside docker. It's a musl-cross-make toolchain, you can procure the dependencies locally if you don't like the Docker recipes. Feel free to open an issue if you feel like that's a challenge. Likely, you can get it to work but checksum reproducibility will be hard without a controlled environment like docker.

sdovan1 4mo ago

If your environment is POSIX, testing scripts with tool written in POSIX shell, like shellspec[1], might also be a choice. [1] https://shellspec.info/ https://shellspec.info/

paulddraper 4mo ago

> When someone says "write it in POSIX shell for portability," they mean well. > POSIX is a specification. Not a program. The thing that actually runs your script is bash, dash, ash, ksh, yash, or one of a dozen others. “When someone says ‘write it in ECMAScript,’ they mean well. “ECMAScript is specification. Not a program. The thing that actually runs your script is Node.js, Bun, Deno, Rhino, or one of a dozen others.” See how silly that sounds?

gaigalas 4mo ago
No one says "Let's write ECMAScript for portability". If someone did, I would probably be writing about that too.
- hnfong 4mo ago
  Hey guys... I have a wonderful idea. ... Let's write ECMAScript for portability.
  - gaigalas 4mo ago
    Can you imagine? Using that fancy new gen JavaScript today? Oh wait, that's exactly babel's headline! It's just so old now that people don't need to say what it does. Actually funny that the spec drift got solved by polyfills and transpilers.
- paulddraper 4mo ago
  Are you serious? People say that all the time. People use JavaScript everywhere.
  - gaigalas 4mo ago
    Nope, implementors are worried about shaping the spec (which iterates much faster than POSIX) and the overall community around JS has plenty of portability solutions. That simply doesn't happen with portable shell script. There's a lot of misconceptions (people can't distinguish between shell/terminal, shell/coreutils, etc), implementors don't talk to each other. "People use JavaScript everywhere" means nothing for this discussion, it's about whether the ecosystem grasps fundamental portability questions. The JS ecosystem does (now, after some decades), the shell one does not. We have babel, core-js and many other libs. It's just a bunch of trick compilations (Ubuntu's dash vs bash page, Rich's shell tricks), nothing actually sturdy. You can try to bend the discussion to make them closer, but it doesn't survive a critical analysis.

JdeBP 4mo ago

Stéphane Chazelas would agree, * https://unix.stackexchange.com/a/496642/5132 https://unix.stackexchange.com/a/496642/5132 because the problem here is educating people with slipshod ideas about 'sh' being 'the POSIX shell' or (worse) 'the Bourne shell'. Both M. Chazelas and M. Gaigalas are making the point that 'sh' is a language that one aims to write in, not one of the many programs that sort of, sometimes, if invoked in the right way, implement that language; and a subordinate point that people are generally very poor about doing that when yet they insist that they are writing 'POSIX shell script'. Fun facts: Standardization led by existing practice is not a simple process. * https://unix.stackexchange.com/a/493743/5132 https://unix.stackexchange.com/a/493743/5132 POSIX/SUS standardization is not a static thing. The long discussed thorny issue of echo has now subtly changed from all of those explanations given about it over the years, because in 2024 the standard was changed. * https://pubs.opengroup.org/onlinepubs/9799919799/utilities/echo.html https://pubs.opengroup.org/onlinepubs/9799919799/utilities/e... M. Chazelas's own quite famous 2013 StackExchange answer on the subject has not yet been updated with the change that now incorporates -e and -E into the rule. Amusingly, it was M. Chazelas that raised defect 1222 that caused this change. * https://unix.stackexchange.com/a/65819/5132 https://unix.stackexchange.com/a/65819/5132 * https://www.austingroupbugs.net/view.php?id=1222 https://www.austingroupbugs.net/view.php?id=1222

wpollock 4mo ago
Stéphane (not "M") Chazelas is a genius. I have written him several times over the past 30 years, encouraging him to write a book about shell, POSIX, and other topics that he is an expert in. He writes very well too. I fear we will all be the poorer if/when he retires.
- rmunn 4mo ago
  "M." is the French abbreviation for "monsieur", the equivalent of "Mr." in English. So "M. Chazelas" is the correct way to politely refer to Stéphane Chazelas by his last name, since he's French. (Not that writing "Mr. Chazalas" would be wrong either when writing in English, but "M. Chazelas" is entirely correct).

kevin_thibedeau 4mo ago

Solaris most definitely has a POSIX shell at /usr/xpg4/bin/sh which behaves differently than OG Bourne at /bin/sh.

gaigalas 4mo ago
You can run this pup to find out what's underneath! https://www.in-ulm.de/~mascheck/various/whatshell/ https://www.in-ulm.de/~mascheck/various/whatshell/

qudat 4mo ago

This is quite a timely post compared to: https://bower.sh/posix-shell-is-all-you-need https://bower.sh/posix-shell-is-all-you-need This post was thought provoking, I wonder, is the hidden argument here that the posix spec for a shell is not well specified if there is so much variance between the implementations? Or is the fundamental issue simply a matter of history? Both?

gaigalas 4mo ago
Most shell families precede the spec. Both bash and ksh had features at the time the spec was written that are not in it. It's weird. My main point is that following the spec doesn't guarantee shell scripts will be portable, which is a common misconception.
- qudat 4mo ago
  > My main point is that following the spec doesn't guarantee shell scripts will be portable, which is a common misconception. That feels like a failure in the spec. Your example illustrates it: echo has unspecified behavior that literally prevents it from being portable. Is it possible portability is just not a feature of posix?
  - gaigalas 4mo ago
    I honestly don't know! What I know is that there is an intersection of features that work with polyfills and tweaks in all shells. That intersection has a lot of the spec, but also has a lot that is not specified anywhere. I started to build something here 12 years ago: https://github.com/Mosai/workshop https://github.com/Mosai/workshop. Then here 2 years ago: https://github.com/alganet/coral https://github.com/alganet/coral Now I'm attacking it from another angle, trying to document what I learned methodically and share the tools I made for me with everyone else.
  - SAI_Peregrinus 4mo ago
    POSIX specifies a common subset of functionality of several different shells (and C libraries) that existed when it was created, and that any compliant shell (or C library) will share. Some functions like `echo` are guaranteed to be present but don't have fully-specified behavior, because the shells that POSIX was created to be the common functional subset of had different behaviors when POSIX was created. POSIX doesn't guarantee portability. POSIX only guarantees that the things it specifies will work in a POSIX-compliant environment. POSIX makes no guarantees about portability to non-POSIX-compliant environments like Windows' shells or MSVC, the Fish shell, etc. POSIX makes no guarantees about the things it doesn't specify.

croemer 4mo ago

Inconsistent: claims /bin/sh on Alpine is dash in one paragraph, BusyBox ash in another. Smells a lot of AI writing.

gaigalas 4mo ago
They are the same, kinda! ash and dash are from the Almquist family. https://en.wikipedia.org/wiki/Almquist_shell https://en.wikipedia.org/wiki/Almquist_shell It's just poetic license.
- croemer 4mo ago
  They are not the same. For example, `[[` apparently works on Alpine's ash but not on Debian's dash. BusyBox ash has some bash compatibility options enabled in Alpine. It's ironic that a post about shell differences glosses over this. Maybe not surprising that slop is sloppy.
  - gaigalas 4mo ago
    The post is not about shell differences, there's too many of them. It's about portability. I want users to test instead of rote memorizing. And I'm giving them tools: docker run --rm alganet/shell-versions:all -c 'PATH=; [[ 1 == 1 ]] || echo fail' Don't trust any blogger, use science.
    - croemer 4mo ago
      The tool is handy, managed to confirm that dash and ash behave differently. But why should I trust your tool if your writing is wrong?
      - gaigalas 4mo ago
        You shouldn't. You should audit the tool. It's open source, and I offer build checksums for reproducibility, it's made to be auditable.

chasil 4mo ago

The author has neglected the pdksh family of shells which includes all of Android, so this is a sizable oversight. This would include the MirBSD mksh, and oksh from OpenBSD. They are much smaller than ksh93 and bash, and I would suggest that their syntax extensions be given preference, as they originally represented ksh88 (from which the POSIX shell standard was derived).

gaigalas 4mo ago
These are all in the shell-versions toolkit I recommended. docker run -it --rm alganet/shell-versions:all /opt/mksh_R59c/bin/mksh -c 'echo $KSH_VERSION' docker run -it --rm alganet/shell-versions:all /opt/oksh_7.9/bin/oksh -c 'echo $KSH_VERSION' docker run -it --rm alganet/shell-versions:all /opt/loksh_7.9/bin/loksh -c 'echo $KSH_VERSION' I omitted them from the post because this is not about the history of ksh and the pdksh codebase.
- chasil 4mo ago
  The POSIX commentary mentions the Korn shell ten times, including particular behavior of the 1988 ksh release. Bash is not mentioned. It is easier to understand the POSIX standard with a ksh focus, particularly ksh88. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
  - gaigalas 4mo ago
    I particularly like ksh93. It's a virtual machine, which makes it super fast for a lot of scenarios. The ksh family is my personal favorite, in particular mksh and ksh93. Both with excelent feature sets. I even made a platformer game that works on them https://github.com/alganet/tuish/blob/main/examples/game.sh https://github.com/alganet/tuish/blob/main/examples/game.sh (and zsh/bash/busybox too) to show how feature-complete they are. Shell interpreters are such a broad subject, I could go all day talking about cool things that can be done with them. I want to get people focusing less on the spec. It's for whoever implements interpreters, not people who write scripts. And there's a gap on that, which I'm trying to cover (with full ksh support, much more than you think).

chubot 4mo ago

FWIW Oils has an option to prevent the ambiguity: osh-0.37$ echo "c:\new" c:\new osh-0.37$ shopt --set no_parse_backslash osh-0.37$ echo "c:\new" echo "c:\new" ^ [ interactive ]:6: Invalid char escape in double quoted string (OILS-ERR-12) It really should be echo "c:\\new" # with two backslashes That is an unambiguous program that works in every shell. In a well-written shell program, the only things that should follow a single backslash in a double quoted string are \ " ` $ (Although I found that this option is only on in ysh, not in shopt --set strict:all ... arguably that should be changed) Nine Reasons to Use OSH - https://oils.pub/osh.html https://oils.pub/osh.html

gaigalas 4mo ago
yash also has something similar, ECHO_STYLE var. https://magicant.github.io/yash/doc/_echo.html https://magicant.github.io/yash/doc/_echo.html Additionally, the ksh family has `print`, which solves some of the issues. And the story repeats all over again, each shell solving the problem in a different way :) That's one of the main reasons I went for solving the portability problem from the scripting side, not the interpreter side.
dzaima 4mo ago
That seems to be be an entirely-different question - `echo "c:\\new"` still differs in behavior between bash and dash - dash parses backslashes in both the double-quoted string, and then echo does another backslash parsing pass, still printing a newline; whereas bash prints a backslash + n.
- chubot 4mo ago
  OK interesting, yeah I think dash is just plain broken ... $ dash -c 'echo "c:\\new"' c: ew $ busybox ash -c 'echo "c:\\new"' c:\new It requires 4 backslashes: $ dash -c 'echo "c:\\\\new"' c:\new $ busybox ash -c 'echo "c:\\\\new"' c:\\new I am not sure this is a matter of "undefined behavior in POSIX" -- I think it might just be dash being wildly unconformant, which I have seen in other cases. It's one of the least POSIX compliant shells. It is derived from the same codebase as busybox ash, but busybox receives more maintenance. --- In any case, it is pretty sad that sh is in such poor shape than the default /bin/sh on Debian has different behavior in this basic case. I built OSH to be a set of semantics agreed upon by many shells. I don't think any cross-shell test suites like our spec tests had existed in the past - https://oils.pub/release/0.37.0/quality.html https://oils.pub/release/0.37.0/quality.html But there is little coordination among shell authors, and no real motivation to fix the gaps. In contrast, there is A LOT of coordination among JavaScript engine authors, mostly because there are people paid to work on them.

hulitu 4mo ago

> POSIX is a specification. Not a program. The thing that actually runs your script is bash, dash, ash, ksh, yash, or one of a dozen others. They each implement POSIX with their own gaps, extensions, and historical accidents. Blaming POSIX, because a program does not implement it, is childish.

gaigalas 4mo ago
Have you ever tried to implement it?