5 ms·

Great resource! However, one thing left me with a question about the kernel: From [0]: > Since shebangs are handled by the kernel, and pull from buf instead o

Great resource! However, one thing left me with a question about the kernel: From [0]: > Since shebangs are handled by the kernel, and pull from buf instead of loading the whole file, they’re always truncated to the length of buf. Apparently, 4 years ago, someone got annoyed by the kernel truncating their >128-character paths, and their solution was to double the truncation point by doubling the buffer size! Today, on your very own Linux machine, if you have a shebang line more than 256 characters long, everything past 256 characters will be completely lost. What is the reason for silently truncating someone's filepath? The kernel, of all places, should not be doing such things. Granted, 256-byte long file paths don't make sense, but 256-byte long path+argument strings will almost certainly happen sometimes. To just silently break someone's script is wrong. [0]: https://cpu.land/how-to-run-a-program https://cpu.land/how-to-run-a-program

knome 3y ago
you shouldn't be putting complex arguments in a shebang. it's there to tell where the interpreter for the current file is. shebang can't even handle more than a single argument on its own. if you have `#!/bin/program -args somescript`, it will divide it into ['/bin/program','-args somescript'], which is almost never what you actually wanted. ( this is linux specific, shebangs aren't necessarily universally portable between all unices ) if you want to do something complex, make it `#!/bin/sh` and then `exec` from the current file with whatever arguments you need. one thing that link doesn't mention is that you can use env to lookup an interpreter via the current path. `#!/usr/bin/env python3` for example, will find whatever `python3` the users is using, without having to know where it is. this can be useful for scripts that you will be running from virtual environments, for example, which work by overriding the PATH to the python interpreter. you wouldn't want to do this with anything installed, however, since you wouldn't want installed programs to let the user control what interpreter to use. instead, you should just use an absolute path. there is an option in env, -S, that can be used to parse a command line from a single string specifically to handle this limitation. after years of avoiding much trickery in shebang lines, it seems like a questionable practice at best.
- fbdab103 3y ago
  I have always thought that shebangs are absolute paths to be weird. Everything else in my Linux day to day operates by traversing the PATH, why do shebangs get to be different?
  - knome 3y ago
    the PATH search is done in userspace by the exec'ing process. the kernel doesn't have or care what the processes' environment PATH says. it just gets the final location. shebang processing is the kernel peeking at the first couple hundred bytes of a file to see if it should execute it using some other file instead of running it directly. there isn't a userspace yet because it's still setting the process up. even if there was a userspace, PATH searching should be done for the `exec`ing process' path, not the PATH given to the new process. so you'd have to squirrel away the calling processes path somewhere on top of everything else. so, it would be a pain. and also incompatible with all of the previous exec system calls, since if you "fixed" it to follow the path, anything not expecting that could break.
archmaster 3y ago
(Author here:) I strongly agree! It should not require an understanding of the kernel's execution logic to be able to understand the limitations of shebangs. I think this is stupid design and I perhaps should've highlighted it even more as such. A couple years ago someone attempted a patch to the kernel that would at least fail loudly in the case of oversized shebangs. This ended up causing problems for NixOS which, being Nix, already was using oversized shebangs with lots of long nix-store paths. They were previously being silently truncated but still working, and suddenly all those scripts failing caused a lapse in backwards compatibility! So the patch had to be undone and we haven't gotten anything similar since. See https://lwn.net/Articles/779997/ https://lwn.net/Articles/779997/ for more info.
- vacuity 3y ago
  https://xkcd.com/1172/ https://xkcd.com/1172/
NikkiA 3y ago
I'm more surprised it isn't set to be whatever the disk block size is (which is often, but not always, 512 bytes).

knome 3y ago

you shouldn't be putting complex arguments in a shebang. it's there to tell where the interpreter for the current file is. shebang can't even handle more than a single argument on its own. if you have `#!/bin/program -args somescript`, it will divide it into ['/bin/program','-args somescript'], which is almost never what you actually wanted. ( this is linux specific, shebangs aren't necessarily universally portable between all unices ) if you want to do something complex, make it `#!/bin/sh` and then `exec` from the current file with whatever arguments you need. one thing that link doesn't mention is that you can use env to lookup an interpreter via the current path. `#!/usr/bin/env python3` for example, will find whatever `python3` the users is using, without having to know where it is. this can be useful for scripts that you will be running from virtual environments, for example, which work by overriding the PATH to the python interpreter. you wouldn't want to do this with anything installed, however, since you wouldn't want installed programs to let the user control what interpreter to use. instead, you should just use an absolute path. there is an option in env, -S, that can be used to parse a command line from a single string specifically to handle this limitation. after years of avoiding much trickery in shebang lines, it seems like a questionable practice at best.

fbdab103 3y ago
I have always thought that shebangs are absolute paths to be weird. Everything else in my Linux day to day operates by traversing the PATH, why do shebangs get to be different?
- knome 3y ago
  the PATH search is done in userspace by the exec'ing process. the kernel doesn't have or care what the processes' environment PATH says. it just gets the final location. shebang processing is the kernel peeking at the first couple hundred bytes of a file to see if it should execute it using some other file instead of running it directly. there isn't a userspace yet because it's still setting the process up. even if there was a userspace, PATH searching should be done for the `exec`ing process' path, not the PATH given to the new process. so you'd have to squirrel away the calling processes path somewhere on top of everything else. so, it would be a pain. and also incompatible with all of the previous exec system calls, since if you "fixed" it to follow the path, anything not expecting that could break.

archmaster 3y ago

(Author here:) I strongly agree! It should not require an understanding of the kernel's execution logic to be able to understand the limitations of shebangs. I think this is stupid design and I perhaps should've highlighted it even more as such. A couple years ago someone attempted a patch to the kernel that would at least fail loudly in the case of oversized shebangs. This ended up causing problems for NixOS which, being Nix, already was using oversized shebangs with lots of long nix-store paths. They were previously being silently truncated but still working, and suddenly all those scripts failing caused a lapse in backwards compatibility! So the patch had to be undone and we haven't gotten anything similar since. See https://lwn.net/Articles/779997/ https://lwn.net/Articles/779997/ for more info.

vacuity 3y ago
https://xkcd.com/1172/ https://xkcd.com/1172/

NikkiA 3y ago

I'm more surprised it isn't set to be whatever the disk block size is (which is often, but not always, 512 bytes).