6 ms·

Sbang lets you run scripts with long shebang lines

30 points by tgamblin 6y ago

wheybags 6y ago
I've never had this problem, what is the use case? I've only ever used /usr/bin/env or /bin/bash, and can't think of a reason to do anything fancier. You would be hardcoding the full (presumably non standard) path to the interpreter right? Why not just add it to PATH and use env?
- jokethrowaway 6y ago
  See shortenPerlShebang in https://nixos.org/nixpkgs/manual/#sec-language-perl https://nixos.org/nixpkgs/manual/#sec-language-perl
  - rhizome 6y ago
    Darwin-only by my read.
- tgamblin 6y ago
  In Spack https://github.com/spack/spack https://github.com/spack/spack), we encounter this when users install packages in deep paths within their home directories, or in deep paths within shared project directories (e.g., in NFS or Lustre on HPC machines). So we patch installed scripts that have long shebangs. We want installed packages to work exactly as built, with the right versions of dependencies, and we don't want to rely on the user getting their environment right to do that. Spack users may install several versions of python or other interpreters. This ensures that scripts work without a special environment. There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this: #!/usr/bin/perl arg1 arg2 arg3 At least on Linux, you'll get one argument: "arg1 arg2 arg3". I think perl gets around this by parsing the shebang line itself, but sbang is more general. You can use it with /usr/bin/env to pass more arguments than would otherwise be allowed. See https://www.in-ulm.de/~mascheck/various/shebang/ https://www.in-ulm.de/~mascheck/various/shebang/ for an extremely comprehensive list of limitations on shebangs.
  - toomuchtodo 6y ago
    Thanks for sharing this. Wish I had this years ago at an HPC gig at FNAL (similar use case, with users having deep path home directories on Lustre and pNFS).
  - shakna 6y ago
    > There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this: Most often when calling programs with a shell script it is recommended to make use of env to find it. Like... #!/usr/bin/env python3 As in the limitations link above, env is pretty much everywhere, and when it isn't, you're probably going to struggle anyway to make anything portable to those systems. env actually has an in-built flag for ensuring you're passing arguments. #!/usr/bin/env -S python3 -m http.server 8010
    - tgamblin 6y ago
      True. The two disadvantages are: - you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter) - The -S argument to env is not POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/env.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...
      - shakna 6y ago
        > you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter) env lets you solve that one too. You can control PATH. From the FreeBSD manpages: #!/usr/bin/env -S -P/usr/local/bin:/usr/bin perl And if you lack the -P flag... env controls the environment variables... So, you could extend (or replace) it like so: #!/usr/bin/env PATH=/home/my/loc:${PATH} perl > The -S argument to env is not POSIX You're right. POSIX only guarantees the -i flag, and the ability to set variables. It hasn't been modified in the standard since 1991, despite other updates to the standard. OpenBSD and Busybox only supports the crippled POSIX version. It isn't completely portable. Versions with -S and are available on Linux, FreeBSD (also has -P), macOS (also has -P).
- mplanchard 6y ago
  I remember running into this a few years ago with a python virtual environment created in a deeply nested directory, where (IIRC) scripts installed by pip would wind up with the virtual environment’s python path as the shebang. These failed on some of but not all of our boxes. I forget how we resolved it, but I do remember being shocked to discover that there was a length limit.
- OJFord 6y ago
  Argument cause must be pretty rare, but I can imagine it happening with long paths to a virtual environment's interpreter nested deep somewhere in a home directory.
datasmurf 6y ago
I do not think that _this_ was or will be ever an issue to myself or any system that i have to deal with and i do not see any use for it but i guess somebody else did. Anyway I believe this page is a very nice read on the background and a little history https://www.in-ulm.de/~mascheck/various/shebang/ https://www.in-ulm.de/~mascheck/various/shebang/
jepler 6y ago
This reminds me of one of the _very few_ bits of freebsd userspace that are better than gnu and was only adopted by gnu much later (8.30, the version first in debian buster, I think): `env -S`. `env -S` introduces an option parser/splitter for #!-lines which can do just about whatever you need (however, it will not work around over-long lines like sbang proposes to do)
- hobby-coder-guy 6y ago
  What can it do?
- chrismorgan 6y ago
  So, taking the example from the GNU man page: #!/usr/bin/env -S perl -w -T Why is the `/usr/bin/env -S` even in there? Why not rather #!perl -w -T I know that not all environments support non-absolute paths to interpreters like this, but it feels to me that adding -S to env is a worse hack than allowing non-absolute paths to interpreters. I’m not sure which platforms support resolving the interpreter by PATH (actually, is supporting it a shell thing?), but it may even be more portable than -S at present, GNU coreutils env only having had it for two years. And even /usr/bin/env itself isn’t fully portable, as there are (old) platforms that have env at /bin/env and not /usr/bin/env.
  - tgamblin 6y ago
    It's there because on some systems all the args are passed as one. -S splits them out. You can try this on Linux and env even warns you about it: # cat test.py #!/usr/bin/python3 import sys print(sys.argv) $ cat test #!/usr/bin/env test.py foo bar baz $ PATH=. ./test /usr/bin/env: 'test.py foo bar baz': No such file or directory /usr/bin/env: use -[v]S to pass options in shebang lines with -S, env does the right thing, and you get this: $ cat test #!/usr/bin/env -S test.py foo bar baz $ PATH=. ./test ['./test.py', 'foo', 'bar', 'baz', './test']
    - chrismorgan 6y ago
      You completely missed the point of what I was writing. I’m not talking about why -S, but why /usr/bin/env -S.
strags 6y ago
The problem with shebangs that I really want to solve is that I often need to edit files on a Windows machine, and then try to run the resulting CR/LF scripts on a Linux machine using a shared filesystem. (Docker, WSL, Vagrant etc...). I'd like to invoke them with just (eg.) ./dostuff.py - Python, Ruby etc... have absolutely no problem running files containing Win-style line-endings. The only issue is that /usr/bin/env complains that it can't find (eg.) "python3\n". Yes, I know I could convert these files with dos2unix, and I also know that I can just invoke the interpreter explicitly - but I'm lazy, and this seems like such a trivial thing to solve that I can't believe it's not been done already. I've taken to actually recompiling /usr/bin/env to strip trailing whitespace from the executable name - but there must be a better solution.
- AnonC 6y ago
  There are editors that allow preserving the line endings (be they LF, CRLF or CR) from the source file. That could be a solution since you already seem to have control over your environments.
  - strags 6y ago
    Yeah, this is absolutely true, but we don't really like to impose a rule regarding which editor(s) people use. Most of our workflow is Windows-based.
eyelidlessness 6y ago
Wouldn’t it be better to standardize format and baseline argument spec, and let the evaluating environment figure out the rest? Formalizing the entry path is something that is now becoming two lines of indirection just to lie to whatever is actually interpreting it. It’s another line of implicit code that’s up to interpretation and can easily diverge.
- dspillett 6y ago
  > Wouldn’t it be better to standardize format and baseline argument spec Better, yes. Easy, no. Impossible is a sort timeframe, probably (given how many systems are out there not confirming to this as yet undecided standard). Maybe that will actually happen long-term, with this being essentially a polyfil.
caymanjim 6y ago
Why not just do this: #!/bin/bash /some/long/path with lots of args
- yjftsjthsd-h 6y ago
  You probably want an exec in there (but I can't see a problem with this approach)
- tgamblin 6y ago
  I'm not sure how this solves the problem. The goal is to execute an interpreter in a long path via the shebang line, but this is a regular script that executes a program in a long path. You can't use this to make a file invoke /some/long/path when executed. Or am I missing something?
  - caymanjim 6y ago
    Yeah, I see now what it's doing (and how it's implemented), so the use case makes more sense.
saagarjha 6y ago
I wonder if this could be part of binfmt_misc if Linux let you be the handler for shebangs in general?
- Spivak 6y ago
  I think the easier method would just be defining an alternative character sequence to shebang lines which binfmt_ misc is happy to do for you. You already have to change the shebang line to support long lines, what’s switching the symbol to ‘#%’ or something?

30 points by tgamblin 6y ago

I've never had this problem, what is the use case? I've only ever used /usr/bin/env or /bin/bash, and can't think of a reason to do anything fancier. You would be hardcoding the full (presumably non standard) path to the interpreter right? Why not just add it to PATH and use env?

jokethrowaway 6y ago
See shortenPerlShebang in https://nixos.org/nixpkgs/manual/#sec-language-perl https://nixos.org/nixpkgs/manual/#sec-language-perl
- rhizome 6y ago
  Darwin-only by my read.
tgamblin 6y ago
In Spack https://github.com/spack/spack https://github.com/spack/spack), we encounter this when users install packages in deep paths within their home directories, or in deep paths within shared project directories (e.g., in NFS or Lustre on HPC machines). So we patch installed scripts that have long shebangs. We want installed packages to work exactly as built, with the right versions of dependencies, and we don't want to rely on the user getting their environment right to do that. Spack users may install several versions of python or other interpreters. This ensures that scripts work without a special environment. There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this: #!/usr/bin/perl arg1 arg2 arg3 At least on Linux, you'll get one argument: "arg1 arg2 arg3". I think perl gets around this by parsing the shebang line itself, but sbang is more general. You can use it with /usr/bin/env to pass more arguments than would otherwise be allowed. See https://www.in-ulm.de/~mascheck/various/shebang/ https://www.in-ulm.de/~mascheck/various/shebang/ for an extremely comprehensive list of limitations on shebangs.
- toomuchtodo 6y ago
  Thanks for sharing this. Wish I had this years ago at an HPC gig at FNAL (similar use case, with users having deep path home directories on Lustre and pNFS).
- shakna 6y ago
  > There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this: Most often when calling programs with a shell script it is recommended to make use of env to find it. Like... #!/usr/bin/env python3 As in the limitations link above, env is pretty much everywhere, and when it isn't, you're probably going to struggle anyway to make anything portable to those systems. env actually has an in-built flag for ensuring you're passing arguments. #!/usr/bin/env -S python3 -m http.server 8010
  - tgamblin 6y ago
    True. The two disadvantages are: - you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter) - The -S argument to env is not POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/env.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...
    - shakna 6y ago
      > you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter) env lets you solve that one too. You can control PATH. From the FreeBSD manpages: #!/usr/bin/env -S -P/usr/local/bin:/usr/bin perl And if you lack the -P flag... env controls the environment variables... So, you could extend (or replace) it like so: #!/usr/bin/env PATH=/home/my/loc:${PATH} perl > The -S argument to env is not POSIX You're right. POSIX only guarantees the -i flag, and the ability to set variables. It hasn't been modified in the standard since 1991, despite other updates to the standard. OpenBSD and Busybox only supports the crippled POSIX version. It isn't completely portable. Versions with -S and are available on Linux, FreeBSD (also has -P), macOS (also has -P).
mplanchard 6y ago
I remember running into this a few years ago with a python virtual environment created in a deeply nested directory, where (IIRC) scripts installed by pip would wind up with the virtual environment’s python path as the shebang. These failed on some of but not all of our boxes. I forget how we resolved it, but I do remember being shocked to discover that there was a length limit.
OJFord 6y ago
Argument cause must be pretty rare, but I can imagine it happening with long paths to a virtual environment's interpreter nested deep somewhere in a home directory.

datasmurf 6y ago

I do not think that _this_ was or will be ever an issue to myself or any system that i have to deal with and i do not see any use for it but i guess somebody else did. Anyway I believe this page is a very nice read on the background and a little history https://www.in-ulm.de/~mascheck/various/shebang/ https://www.in-ulm.de/~mascheck/various/shebang/

jepler 6y ago

This reminds me of one of the _very few_ bits of freebsd userspace that are better than gnu and was only adopted by gnu much later (8.30, the version first in debian buster, I think): `env -S`. `env -S` introduces an option parser/splitter for #!-lines which can do just about whatever you need (however, it will not work around over-long lines like sbang proposes to do)

hobby-coder-guy 6y ago
What can it do?
chrismorgan 6y ago
So, taking the example from the GNU man page: #!/usr/bin/env -S perl -w -T Why is the `/usr/bin/env -S` even in there? Why not rather #!perl -w -T I know that not all environments support non-absolute paths to interpreters like this, but it feels to me that adding -S to env is a worse hack than allowing non-absolute paths to interpreters. I’m not sure which platforms support resolving the interpreter by PATH (actually, is supporting it a shell thing?), but it may even be more portable than -S at present, GNU coreutils env only having had it for two years. And even /usr/bin/env itself isn’t fully portable, as there are (old) platforms that have env at /bin/env and not /usr/bin/env.
- tgamblin 6y ago
  It's there because on some systems all the args are passed as one. -S splits them out. You can try this on Linux and env even warns you about it: # cat test.py #!/usr/bin/python3 import sys print(sys.argv) $ cat test #!/usr/bin/env test.py foo bar baz $ PATH=. ./test /usr/bin/env: 'test.py foo bar baz': No such file or directory /usr/bin/env: use -[v]S to pass options in shebang lines with -S, env does the right thing, and you get this: $ cat test #!/usr/bin/env -S test.py foo bar baz $ PATH=. ./test ['./test.py', 'foo', 'bar', 'baz', './test']
  - chrismorgan 6y ago
    You completely missed the point of what I was writing. I’m not talking about why -S, but why /usr/bin/env -S.

strags 6y ago

The problem with shebangs that I really want to solve is that I often need to edit files on a Windows machine, and then try to run the resulting CR/LF scripts on a Linux machine using a shared filesystem. (Docker, WSL, Vagrant etc...). I'd like to invoke them with just (eg.) ./dostuff.py - Python, Ruby etc... have absolutely no problem running files containing Win-style line-endings. The only issue is that /usr/bin/env complains that it can't find (eg.) "python3\n". Yes, I know I could convert these files with dos2unix, and I also know that I can just invoke the interpreter explicitly - but I'm lazy, and this seems like such a trivial thing to solve that I can't believe it's not been done already. I've taken to actually recompiling /usr/bin/env to strip trailing whitespace from the executable name - but there must be a better solution.

AnonC 6y ago
There are editors that allow preserving the line endings (be they LF, CRLF or CR) from the source file. That could be a solution since you already seem to have control over your environments.
- strags 6y ago
  Yeah, this is absolutely true, but we don't really like to impose a rule regarding which editor(s) people use. Most of our workflow is Windows-based.

eyelidlessness 6y ago

Wouldn’t it be better to standardize format and baseline argument spec, and let the evaluating environment figure out the rest? Formalizing the entry path is something that is now becoming two lines of indirection just to lie to whatever is actually interpreting it. It’s another line of implicit code that’s up to interpretation and can easily diverge.

dspillett 6y ago
> Wouldn’t it be better to standardize format and baseline argument spec Better, yes. Easy, no. Impossible is a sort timeframe, probably (given how many systems are out there not confirming to this as yet undecided standard). Maybe that will actually happen long-term, with this being essentially a polyfil.

caymanjim 6y ago

Why not just do this: #!/bin/bash /some/long/path with lots of args

yjftsjthsd-h 6y ago
You probably want an exec in there (but I can't see a problem with this approach)
tgamblin 6y ago
I'm not sure how this solves the problem. The goal is to execute an interpreter in a long path via the shebang line, but this is a regular script that executes a program in a long path. You can't use this to make a file invoke /some/long/path when executed. Or am I missing something?
- caymanjim 6y ago
  Yeah, I see now what it's doing (and how it's implemented), so the use case makes more sense.

saagarjha 6y ago

I wonder if this could be part of binfmt_misc if Linux let you be the handler for shebangs in general?

Spivak 6y ago
I think the easier method would just be defining an alternative character sequence to shebang lines which binfmt_ misc is happy to do for you. You already have to change the shebang line to support long lines, what’s switching the symbol to ‘#%’ or something?