4 ms·

The README for this project still says "POSIX-approved" but there is nothing in POSIX that allows mixing commands and binary data in a shell script. It does al

The README for this project still says "POSIX-approved" but there is nothing in POSIX that allows mixing commands and binary data in a shell script. It does allow appending binary data to the end of a shell script* as long as it is guarded from being parsed as shell code by an exit, exec, etc. but that's it. * https://pubs.opengroup.org/onlinepubs/9799919799/utilities/sh.html#tag_20_110_07 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/s...

yawpitch 2y ago
Definitely not my area of either pedantry or expertise, but if I read this correctly: > The input file can be of any type, but the initial portion of the file intended to be parsed according to the shell grammar […] shall consist of characters and shall not contain the NUL character. Then POSIX doesn’t proscribe against appending binary data at all, it just requires that binary data appears after whatever portion of the file is intended to be parsed according to the grammar… there’s nothing stating that intention can’t apply to a zero characters. > The shell shall not enforce any line length limits. That seems to unambiguously allow a zero-length initial portion. > If the input file consists solely of zero or more blank lines and comments, sh shall exit with a zero exit status. And so does that, while also making an enormous loophole for any input file that is not zero or more blank lines. In other words it seems to me on a quick reading that a decent rules-lawyer could drive a planet through that part of the spec, as written.
imiric 2y ago
According to Justine herself[1], POSIX specifically changed their rules to allow this type of functionality[2]. The paragraph you linked to was part of it, but also this paragraph under RATIONALE[3]: > Earlier versions of this standard required that input files to the shell be text files except that line lengths were unlimited. However, that was overly restrictive in relation to the fact that shells can parse a script without a trailing newline, and in relation to a common practice of concatenating a shell script ending with an exit or exec $command with a binary data payload to form a single-file self-extracting archive. So Cosmopolitan indeed seems to be POSIX-approved as far as this behavior goes. [1]: https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ [2]: https://austingroupbugs.net/view.php?id=1250 https://austingroupbugs.net/view.php?id=1250 [3]: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/sh.html#tag_20_110_18 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/s...
- oguz-ismail 2y ago
  [flagged]
  - Y_Y 2y ago
    I think this isn't an entirely worthless comment, but you start off by insisting that Justine Tunney is referred to as "himself". This is either a silly mistake, or a misguided political statement. Either way I don't think it adds to the discussion, and it's certainly going to distract people from your more interesting point that mixing binary data throughout the shell script is not permitted by the standard.
    - sgammon 2y ago
      They made this comment as a correction. It is intentional. When somebody shows you who they are, believe them.
      - AlexeyBelov 2y ago
        And it's always the fresh accounts.
    - jart 2y ago
      It's not an interesting point, it's a willful misinterpretation of the intent of the standard made by someone who wasn't there. The worst thing about bigots is they usually hide it and grasp for other more socially acceptable reasons to make you miserable when you're someone like me. I'm astonished you would still continue to take them seriously after they've revealed their intent and motivation.
      - Y_Y 2y ago
        I apologise, my interpretation was naïve. Not being familiar with the intent of the person I was responding to (directly or by experience) and not knowing the how to correctly interpret the POSIX standard, I was trying to politely re-focus the thread. I'm lucky enough not to have to spend much time thinking about identity issues or being the target of bigotry and I suppose that was reflected in my previous comment. Maybe it's telling that you're here making fantastic contributions for FOSS and people who are trying to bring you down are only making invalid claims about standards compliance. I'm a big fan of yours and I'm really sorry that I contributed to that incident.
Cloudef 2y ago
POSIX is one of those standards that don't really define anything at all. Microsoft POSIX subsystem is one of the nicest examples of something that lets you tick the POSIX checkbox but does almost nothing. One of the recent posix annoyances I stumbled upon was the clock_gettime where every platform has slightly different idea how the different CLOCKs are supposed to be interpreted :) Personally I don't find the APE part of cosmopolitan that interesting, but their own POSIX API <https://justine.lol/cosmopolitan/functions.html https://justine.lol/cosmopolitan/functions.html> which also covers windows is interesting for sure. Zig's std also offers something similar, but perhaps not as good coverage. Getting fork, mmap, and poll work on windows is not small feat.

yawpitch 2y ago

Definitely not my area of either pedantry or expertise, but if I read this correctly: > The input file can be of any type, but the initial portion of the file intended to be parsed according to the shell grammar […] shall consist of characters and shall not contain the NUL character. Then POSIX doesn’t proscribe against appending binary data at all, it just requires that binary data appears after whatever portion of the file is intended to be parsed according to the grammar… there’s nothing stating that intention can’t apply to a zero characters. > The shell shall not enforce any line length limits. That seems to unambiguously allow a zero-length initial portion. > If the input file consists solely of zero or more blank lines and comments, sh shall exit with a zero exit status. And so does that, while also making an enormous loophole for any input file that is not zero or more blank lines. In other words it seems to me on a quick reading that a decent rules-lawyer could drive a planet through that part of the spec, as written.

imiric 2y ago

According to Justine herself[1], POSIX specifically changed their rules to allow this type of functionality[2]. The paragraph you linked to was part of it, but also this paragraph under RATIONALE[3]: > Earlier versions of this standard required that input files to the shell be text files except that line lengths were unlimited. However, that was overly restrictive in relation to the fact that shells can parse a script without a trailing newline, and in relation to a common practice of concatenating a shell script ending with an exit or exec $command with a binary data payload to form a single-file self-extracting archive. So Cosmopolitan indeed seems to be POSIX-approved as far as this behavior goes. [1]: https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ [2]: https://austingroupbugs.net/view.php?id=1250 https://austingroupbugs.net/view.php?id=1250 [3]: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/sh.html#tag_20_110_18 https://pubs.opengroup.org/onlinepubs/9799919799/utilities/s...

oguz-ismail 2y ago
[flagged]
- Y_Y 2y ago
  I think this isn't an entirely worthless comment, but you start off by insisting that Justine Tunney is referred to as "himself". This is either a silly mistake, or a misguided political statement. Either way I don't think it adds to the discussion, and it's certainly going to distract people from your more interesting point that mixing binary data throughout the shell script is not permitted by the standard.
  - sgammon 2y ago
    They made this comment as a correction. It is intentional. When somebody shows you who they are, believe them.
    - AlexeyBelov 2y ago
      And it's always the fresh accounts.
  - jart 2y ago
    It's not an interesting point, it's a willful misinterpretation of the intent of the standard made by someone who wasn't there. The worst thing about bigots is they usually hide it and grasp for other more socially acceptable reasons to make you miserable when you're someone like me. I'm astonished you would still continue to take them seriously after they've revealed their intent and motivation.
    - Y_Y 2y ago
      I apologise, my interpretation was naïve. Not being familiar with the intent of the person I was responding to (directly or by experience) and not knowing the how to correctly interpret the POSIX standard, I was trying to politely re-focus the thread. I'm lucky enough not to have to spend much time thinking about identity issues or being the target of bigotry and I suppose that was reflected in my previous comment. Maybe it's telling that you're here making fantastic contributions for FOSS and people who are trying to bring you down are only making invalid claims about standards compliance. I'm a big fan of yours and I'm really sorry that I contributed to that incident.

Cloudef 2y ago

POSIX is one of those standards that don't really define anything at all. Microsoft POSIX subsystem is one of the nicest examples of something that lets you tick the POSIX checkbox but does almost nothing. One of the recent posix annoyances I stumbled upon was the clock_gettime where every platform has slightly different idea how the different CLOCKs are supposed to be interpreted :) Personally I don't find the APE part of cosmopolitan that interesting, but their own POSIX API <https://justine.lol/cosmopolitan/functions.html https://justine.lol/cosmopolitan/functions.html> which also covers windows is interesting for sure. Zig's std also offers something similar, but perhaps not as good coverage. Getting fork, mmap, and poll work on windows is not small feat.