6 ms·

Relative shell script includes with realpath on FreeBSD

83 points by signa11 3y ago

rurban 3y ago
He is still missing the double quotes, paths may contain spaces
- stouset 3y ago
  Yep. LIB="$(dirname "$(realpath "$0")")" It seems like overkill, but just get in the habit of quoting anything and everything.
  - js2 3y ago
    You can technically omit the double-quotes during a scalar variable assignment (as long as it's not preceded by a modifier such as `export`), so the outermost quotes are not needed above. https://unix.stackexchange.com/questions/68694/when-is-double-quoting-necessary https://unix.stackexchange.com/questions/68694/when-is-doubl... The extra quotes don't hurt, but also: use shellcheck. It will tell you if you've omitted necessary quotes.
    - lloeki 3y ago
      > The extra quotes don't hurt I have the habit of quoting everything, that way I don't have to think about any corner case where I can skip quotes, and thus conversely never miss anything where I should quote. I do the same with curly braces, that way I don't have to think whether "${foo_bar}" or "${foo}_bar" was meant, and makes later adding or reading string manipulation (e.g "${haystack/needle/replacement}") or arrays (e.g "${foo[42]}" or "${foo[@]}" that much more consistent: variable references are always bounded by ${} which makes code easier to scan. I apply that to "${1}" and "${@}" too whenever possible, unfortunately bash 3.x doesn't quite like that so it's not always the case (I can't decide which one I hate more: macOS for the refusal to either update bash or outright drop it† and just have a plain posix sh + zsh, or GNU for the viral license move which is highly detrimental in practice for that case). I also often quote "semantically", e.g: # instead of: PATH="/some/path/${foo}/bar:${PATH}" /some/command "--foo=${bar}" # I'd rather do: PATH="/some/path/${foo}/bar":"${PATH}" /some/command --foo="${bar}" † I guess there's a crapton of bash scripts that have been written and possibly embedded in apps or installers, so backwards compatibility won here.
      - stouset 3y ago
        Are you me? :)
    - stouset 3y ago
      I actually personally use `declare` frequently, particularly to make variables readonly. So in my case the quotes are necessary. Of course this has its own issues, in that subprocess exit codes are hidden, so you often need to assign then declare. But I quote everything, always, because then I don’t have to rely on remembering when it’s necessary and when it isn’t. Just do the safe thing every time and you don’t have to think.
  - tpoacher 3y ago
    no, you should only quote things that you want quoted. ("duh") the caveat being if you actually do want things expanded then you should not quote them. for i in $(seq ...) comes to mind...
- iforgotpassword 3y ago
  Yeah that made me really anxious while reading this. That's the first thing I hammer into fresh hires and interns who are "lucky" enough to end up dealing with shell scripts.
- enriquto 3y ago
  > paths may contain spaces what kind of savage puts spaces on their paths? Failing in such cases is a neat feature it would seem. I never understood why common filesystems allow this foolishness. There are already other semantically meaningful characters in paths that are forbidden, like '/' or '\0'. Disallowing bare spaces in filenames is such a natural feature that it makes no sense to deal with all this stupid quoting pain. Any space that is entered by a user in a filename context should be transparently translated to a unicode non-breaking space. Most users wouldn't notice, and shell scripts authors would be much happier!
  - rurban 3y ago
    > what kind of savage puts spaces on their paths? Failing in such cases is a neat feature it would seem. windows noobs (as in embedded development), but more likely grabbed music.
  - lifthrasiir 3y ago
    Failing in the way that it is virtually not fixable (e.g. Windows user names) or destructive (e.g. removing some unrelated directories) is not neat. It is even often said that Windows had a space in `C:\Program Files` (now `C:\ProgramData`) in order to make sure that app developers have to handle paths with spaces correctly. Don't know if it's indeed true, but at least it's believable.
  - CorrectHorseBat 3y ago
    It's much worse than this. On Unix '\000' (NUL) and '\057' (/) are the _only_ things that are not allowed. Anything else, whether it's valid in any encoding or not, is allowed.
    - ykonstant 3y ago
      In order to test my shell scripts, I have a directory hierarchy of filenames generated from /dev/urandom, which contains a sub-structure with filenames in unary, using only the newline character.
  - lmm 3y ago
yjftsjthsd-h 3y ago
That's a portable sh solution, isn't it? Like, yes, it does work on FreeBSD, but it isn't exclusive
- oefrha 3y ago
  POSIX has no realpath(1), so no.
  - arp242 3y ago
    *Yet. Will be added in the upcoming revision: https://www.austingroupbugs.net/view.php?id=1457 https://www.austingroupbugs.net/view.php?id=1457 20 years ago it was a "BSD thing", but it seems to be more wide-spread these days; even my Alpine busybox system has it. I'm sure there are some platforms without it, but there are also platforms without some POSIX tools – it's never a guarantee.
  - pixelbeat__ 3y ago
    All the main platforms have realpath(1). It was added to GNU Linux over 10 years ago now, to aid portability, and be a more natural command to access this functionality than readlink(1)
    - oefrha 3y ago
      Not really. macOS until version 12 doesn’t have realpath(1), it was only added this year. You can’t rely on it for portable code.
      - latexr 3y ago
        To clarify, it was added (late) last year, with macos 13 (Ventura).
    - inferiorhuman 3y ago
      Saw a comment on another site mention this, but NetBSD won't get it until 10.0 (not yet released) and OpenBSD got it in 7.1 (released last year). Of course I'm of the mindset that building complex shell scripts is simply an exercise in masochism so this all seems like a solution in search of a problem.
netcoyote 3y ago
Should use $BASH_SOURCE in place of $0 so it works in files that are sourced. The incantation at the top of all my scripts, that I learned from other HN folks, is: SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" And I guess throw a realpath in there too for symlinks, per TFA.
- js2 3y ago
  You don't need realpath. The `cd` into the directory + `pwd` is equivalent. BASH_SOURCE[0] obviously only works with bash.
  - account42 3y ago
    > You don't need realpath Yes you do (or a suitable alternative, e.g. GNU readlink). > The `cd` into the directory + `pwd` is equivalent. You won't know which directory to cd to without realpath. Just stripping off the filename does not get you the correct directory if the file itself is a symlink, which is a common use case.
    - js2 3y ago
      Doh, you're right. Thank you for the correction.
- arp242 3y ago
  /bin/sh on FreeBSD isn't bash.
- peebeebee 3y ago
  On MacOS, we had interoperability uses between different versions somehow, and some of us could not use `realpath` . The following worked for everyone: SCRIPT_DIR=$(dirname "$(readlink -f "${BASH_SOURCE[0]}")").
jchw 3y ago
In addition to other details noted about escaping/quoting and the portability of realpath, it's important to realize that those paths by default are not relative to the invocation directory, they're relative to the current working directory, which happens to be the current working directory of the parent process initially. Also, for a shell script it is relatively reliable to rely on $0, but not guaranteed by any means: for example, an obvious way to subvert this would be to pipe the content of the script into a shell, though there's surely plenty of ways you can think of that this could be broken.
AsthmaBoy 3y ago
Wouldn't readlink -f $0 work to get the actual location of the script, and from there get the base for relative scripts?
- Asooka 3y ago
  The -f flag is a GNU extension. Using realpath is more portable.
halayli 3y ago
Here's a good explanation from SO that I happen to have in my bookmarks: https://stackoverflow.com/questions/29832037/how-to-get-script-directory-in-posix-sh/29835459#29835459 https://stackoverflow.com/questions/29832037/how-to-get-scri...
- n_plus_1_acc 3y ago
  Thanks, I will use that from now on as a reference when somebody asks me why I don't like to program longer things in shell.
ChoHag 3y ago
[dead]
Y_Y 3y ago
What about hard links dude? Symlinks are fine because they're "fake", but it's not like a file must have a single canonical location.
- andsens 3y ago
  In that case whoever is linking the script is “breaking” it, I’d say. There’s stuff that is simply outside the responsibilities of the script/maintainer. If you start worrying about that for “simple” scripts, the logic for handling that is quickly going to outgrow the initial logic itself.
  - account42 3y ago
    Besides, hard links don't have a canonical target, they are all equally correct names for the file. So even if the script author wanted to support the same technique for hard links there is no reasonable way to do it.
- rollcat 3y ago
  I think symlinks (or hard links, doesn't matter) are the actual problem here. It is actually completely OK for your code to make certain assumptions about how it's going to be invoked, and otherwise break (loudly, and in an obvious way, if possible).
febusravenga 3y ago
Creating proper `npm` package that works in many popular variants of J*script environment slowly but surely gets to same level of complexity.
gkfasdfasdf 3y ago
FYI realpath was not available by default on MacOS until very recently as others point out: https://news.ycombinator.com/item?id=38514413 https://news.ycombinator.com/item?id=38514413 Prior to that you would need to do 'brew install coreutils' to get realpath.
a-french-anon 3y ago
As someone who has fought a long time against holes in POSIX sh (cf https://git.sr.ht/~q3cpma/scripts https://git.sr.ht/~q3cpma/scripts README with links to the Austin group Mantis), let me list my biggest pet peeves (including this one): * mktemp -d (mktemp itself is available via POSIX m4; the noclobber solution mentioned is moot, since FIFOs break it https://austingroupbugs.net/view.php?id=1364 https://austingroupbugs.net/view.php?id=1364), very much needed and available on GNU, *BSD and MacOS. * local or typeset: I think only ksh88/93 doesn't have it without using the non-standard "function" keyword, a runtime check with potential alias creation is my only "solution" (https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a953446908936cec25b0ca847df626/item/util.sh#L5 https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a9534469089...) * realpath/readlink -f/-e: as said by some, realpath is getting standardized in the next POSIX; can be emulated via a cumbersome function (https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a953446908936cec25b0ca847df626/item/readlinkf https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a9534469089...) but since you either put that massive block in your script or need realpath to reliably source it from another file... * All the "newline in paths" mess: I gave up, just too many blind spots.
- ykonstant 3y ago
  Indeed, even the script you linked is not good enough; it does not deal with trailing newlines in the file or directory name, and the ls parsing will fail with a file like 'my malicious -> file'.
- o11c 3y ago
  Another major hole (in all shells!): there's no way to perform quote-aware splitting of a string in a restricted manner. This is especially egregious given the lack of array variable support. `xargs` is close, but only handles a few kinds of quotes, not e.g. `$''` which is the best kind of quoting, and notably does not adapt to the current shell. `printf %q` is also pretty important to avoid injection attacks, though if you don't care about ugliness or performance you can hack it together pretty easily (but beware of trailing newlines in subshells!). (also, I am obligated to point out that BSDers are lying when they say they're using "Bourne shell". FreeBSD uses a fork of `ash`, related to `dash` and `busybox ash`, but all 3 have evolved independently. OpenBSD uses a derivative of `pdksh` (other BSDs also ship derivatives but not as /bin/sh). NetBSD has their own from-scratch shell, but NetBSD is dead and irrelevant. I have a whole tree of shells; `ash` is notable for being the most unpredictable within the tree.)
H4ZB7 3y ago
i like how thanks to un*x mentality i can just expect argv to be the most malicious possible implementation where the invoker can put anything in it and it will be there for the invokee, on the os family boasting its support for secure multi user systems: /usr/sbin/./././some_admin_tool /usr/sbin/../sbin/../sbin/some_admin_tool /my_rigged_folder/haha (using symlink) but i guess it would have required allocating a string or something which may have mattered 40 years ago even though we could have had a better OS in every way 20 years ago and the same for the process list (in multiple ways, one you can see other people's passwords if they do HISTFILE=/dev/null something --password password, two you can just modify the process list) and i get the last laugh because un*x lovers will reply to criticism such as "your CLI shit is retarded it logs the passwords to disk" with "NO BRO I USE HISTFILE=/dev/null" to which i reply "that's a lot of shit to type just to not log your password to disk" to which he replies with "NO BRO I USE SPACE PREFIX" and he's still wrong due to previous paragraph. similarily the un*x lover will boast about how he can detect malware / bad guys by using the process list yet nope it can just be bypassed like anything in un*x. and don't even get me started with how moronic it is for a CLI program to "prompt" the user for a password to avoid this issue (actually you can use patches to make CLI paramters not shared to all users of the system, which is still stupid because none of that type of information should ever be shared in he first place: no other user's process should see any attribute of mine) and the icing on the cake is the typical way people inside the industry are completely oblivious and say "no man this is actually amazing stuff here" > It's possible to write extremely powerful tools using shell scripts, and modularize your code. Bourne shell doesn't provide a clear mechanism for referencing relative files. By establishing a command file's base dir, you can reliably reference relative shell scripts. wow a turing complete language is powerful who knew now if only we had one that wasn't just a cesspit of footguns most of which are there for no real reason and even historically were still invalid
samgranieri 3y ago
I didn't know about this. I'm writing bash scripts in the scripts-to-rule-them-all fashion for work and this can help DRY things up
extraduder_ire 3y ago
Most scripts I write cd into the directory they are located in for this reason. (and so that nohup.out files don't clobber each other in my home folder) Hopefully I can keep this in mind if I ever have to symlink them.
- _ea1k 3y ago
  +1 - it really irritates me when some script insists that I change into its directory in order to work. Its just lazy script writing.
  - twic 3y ago
    It can also be annoying when a script changes directory itself. If anything the script does is relative to the current directory (interpreting a relative path, creating output files), you now can't have that be relative to your current working directory. Best practice, IMHO, is usually for scripts not to touch the current directory, and to be agnostic to it.
abathur 3y ago
One approach to this problem that’s been fruitful in Nix ecosystem is ~packaging scripts and baking in absolute paths at build time. (Generally by rewriting references in the script to absolute paths or generating wrappers to set PATH or BASE or DIR or whatnot.)
chubot 3y ago
This has bugged me for a long time too – on Linux you can use readlink -f to get realpath, but that’s not available on OS X. So I built it into Oils (both OSH and YSH): source $_this_dir/lib/util.ysh $_this_dir is a special variable that’s relative to the file it’s being sourced from. https://www.oilshell.org/release/0.19.0/doc/ref/chap-special-var.html#_this_dir https://www.oilshell.org/release/0.19.0/doc/ref/chap-special... https://www.oilshell.org/release/latest/doc/ysh-tour.html#appendix-example-of-an-ysh-module https://www.oilshell.org/release/latest/doc/ysh-tour.html#ap... It starts with a _ to let you know it’s magic, similar to _status and _process_sub_status (the latter being unavailable in any shell besides Oils BTW) So yeah this one reason I say POSIX and shell are “rotting” – they lack extremely basic functionality like this. This could have been done 20 years ago, but there’s nobody really pushing it. (We may add namespaces to shell, which will change some things in the example above, but it won’t change _this_dir. We can use feedback and help! - See recent story https://lobste.rs/s/h4egaz/oils_winter_status_update https://lobste.rs/s/h4egaz/oils_winter_status_update ) (copy of lobste.rs comment - https://lobste.rs/s/4emmdl/sh_relative_shell_script_includes_with https://lobste.rs/s/4emmdl/sh_relative_shell_script_includes...) more: Oils Winter Status Update - https://www.oilshell.org/blog/2023/11/status-update.html https://www.oilshell.org/blog/2023/11/status-update.html
twic 3y ago
I reject the symlink problem as not real, and i think the solution is harmful. Here's my expectation: if a script is linked into some location, and invoked at that location, it should behave the same as if it had been copied to that location. If it wouldn't work if it was copied, it isn't reasonable to expect that it should work if it's symlinked. There's no point catering to that use case. Moreover, let's say i symlink the including script, then copy the included script to the right relative position, then edit the included script. I would expect that running the symlinked including script should include the copied and modified included script, not the original. In general, i think programmers are far too eager to apply realpath or readlink -f to paths. If a path has symbolic elements in it, it's because someone has put them there on purpose. Don't just remove them. It's actually very rare to really need to know the canonical path to a file.
- seadan83 3y ago
  Where do you see the harm in the solution? To your other point, I think this is a valid use-case and do not agree with your rejection. I have had experience with such very examples and submitted PRs to add `realpath` to fix them. One example, a linting app I installed put itself in `/usr/local/<linter-name>/bin/<linter-executable>`. It was really painful to type this full path every time (even with tab completion). I created a symlink from `/usr/bin/` to the executable, and voila, the command is on my path now and easy to use; except the script does not expect itself to be a symlink and broke. To work around this, I created an actual script in `/usr/bin` that could invoke the utility (which then worked because it did properly detect path at least with $0). I don't think that extra script should have been necessary. Further, if I only updated my path, then `dirname $0` fails (returns '.') and that was not even a solution. Second question, as a counter-point, couldn't someone also reject the expectation that scripts should work when invoked from any path at all? That is saying, as a script writer, when you file a bug report, I can respond "closed, this script requires a `cd` to the install directory and only then can be run. RTFM!". At what point is this just ignoring UX?
  - twic 3y ago
    The harm i see is what i describe under "moreover" - inconsistency with respect to copying. In your linter example, i think the most normal solution would be to add /usr/local/<linter-name>/bin to your PATH. Your solution, of a script which just does exec /usr/local/<linter-name>/bin/<linter-name> "$@", also seems fine to me - quite simple, and less likely to cause weird behaviour than a symlink. I'm not sure what point you're making at the end. We agree that a script not working when invoked from anywhere is bad UX, because people expect to be able to do that. I think that scripts not working through a symlink is not bad UX, because people can't expect to be able to do that, because in many cases that doesn't work!

83 points by signa11 3y ago

rurban 3y ago

He is still missing the double quotes, paths may contain spaces

stouset 3y ago
Yep. LIB="$(dirname "$(realpath "$0")")" It seems like overkill, but just get in the habit of quoting anything and everything.
- js2 3y ago
  You can technically omit the double-quotes during a scalar variable assignment (as long as it's not preceded by a modifier such as `export`), so the outermost quotes are not needed above. https://unix.stackexchange.com/questions/68694/when-is-double-quoting-necessary https://unix.stackexchange.com/questions/68694/when-is-doubl... The extra quotes don't hurt, but also: use shellcheck. It will tell you if you've omitted necessary quotes.
  - lloeki 3y ago
    > The extra quotes don't hurt I have the habit of quoting everything, that way I don't have to think about any corner case where I can skip quotes, and thus conversely never miss anything where I should quote. I do the same with curly braces, that way I don't have to think whether "${foo_bar}" or "${foo}_bar" was meant, and makes later adding or reading string manipulation (e.g "${haystack/needle/replacement}") or arrays (e.g "${foo[42]}" or "${foo[@]}" that much more consistent: variable references are always bounded by ${} which makes code easier to scan. I apply that to "${1}" and "${@}" too whenever possible, unfortunately bash 3.x doesn't quite like that so it's not always the case (I can't decide which one I hate more: macOS for the refusal to either update bash or outright drop it† and just have a plain posix sh + zsh, or GNU for the viral license move which is highly detrimental in practice for that case). I also often quote "semantically", e.g: # instead of: PATH="/some/path/${foo}/bar:${PATH}" /some/command "--foo=${bar}" # I'd rather do: PATH="/some/path/${foo}/bar":"${PATH}" /some/command --foo="${bar}" † I guess there's a crapton of bash scripts that have been written and possibly embedded in apps or installers, so backwards compatibility won here.
    - stouset 3y ago
      Are you me? :)
  - stouset 3y ago
    I actually personally use `declare` frequently, particularly to make variables readonly. So in my case the quotes are necessary. Of course this has its own issues, in that subprocess exit codes are hidden, so you often need to assign then declare. But I quote everything, always, because then I don’t have to rely on remembering when it’s necessary and when it isn’t. Just do the safe thing every time and you don’t have to think.
- tpoacher 3y ago
  no, you should only quote things that you want quoted. ("duh") the caveat being if you actually do want things expanded then you should not quote them. for i in $(seq ...) comes to mind...
iforgotpassword 3y ago
Yeah that made me really anxious while reading this. That's the first thing I hammer into fresh hires and interns who are "lucky" enough to end up dealing with shell scripts.
enriquto 3y ago
> paths may contain spaces what kind of savage puts spaces on their paths? Failing in such cases is a neat feature it would seem. I never understood why common filesystems allow this foolishness. There are already other semantically meaningful characters in paths that are forbidden, like '/' or '\0'. Disallowing bare spaces in filenames is such a natural feature that it makes no sense to deal with all this stupid quoting pain. Any space that is entered by a user in a filename context should be transparently translated to a unicode non-breaking space. Most users wouldn't notice, and shell scripts authors would be much happier!
- rurban 3y ago
  > what kind of savage puts spaces on their paths? Failing in such cases is a neat feature it would seem. windows noobs (as in embedded development), but more likely grabbed music.
- lifthrasiir 3y ago
  Failing in the way that it is virtually not fixable (e.g. Windows user names) or destructive (e.g. removing some unrelated directories) is not neat. It is even often said that Windows had a space in `C:\Program Files` (now `C:\ProgramData`) in order to make sure that app developers have to handle paths with spaces correctly. Don't know if it's indeed true, but at least it's believable.
- CorrectHorseBat 3y ago
  It's much worse than this. On Unix '\000' (NUL) and '\057' (/) are the _only_ things that are not allowed. Anything else, whether it's valid in any encoding or not, is allowed.
  - ykonstant 3y ago
    In order to test my shell scripts, I have a directory hierarchy of filenames generated from /dev/urandom, which contains a sub-structure with filenames in unary, using only the newline character.
- lmm 3y ago

yjftsjthsd-h 3y ago

That's a portable sh solution, isn't it? Like, yes, it does work on FreeBSD, but it isn't exclusive

oefrha 3y ago
POSIX has no realpath(1), so no.
- arp242 3y ago
  *Yet. Will be added in the upcoming revision: https://www.austingroupbugs.net/view.php?id=1457 https://www.austingroupbugs.net/view.php?id=1457 20 years ago it was a "BSD thing", but it seems to be more wide-spread these days; even my Alpine busybox system has it. I'm sure there are some platforms without it, but there are also platforms without some POSIX tools – it's never a guarantee.
- pixelbeat__ 3y ago
  All the main platforms have realpath(1). It was added to GNU Linux over 10 years ago now, to aid portability, and be a more natural command to access this functionality than readlink(1)
  - oefrha 3y ago
    Not really. macOS until version 12 doesn’t have realpath(1), it was only added this year. You can’t rely on it for portable code.
    - latexr 3y ago
      To clarify, it was added (late) last year, with macos 13 (Ventura).
  - inferiorhuman 3y ago
    Saw a comment on another site mention this, but NetBSD won't get it until 10.0 (not yet released) and OpenBSD got it in 7.1 (released last year). Of course I'm of the mindset that building complex shell scripts is simply an exercise in masochism so this all seems like a solution in search of a problem.

netcoyote 3y ago

Should use $BASH_SOURCE in place of $0 so it works in files that are sourced. The incantation at the top of all my scripts, that I learned from other HN folks, is: SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" And I guess throw a realpath in there too for symlinks, per TFA.

js2 3y ago
You don't need realpath. The `cd` into the directory + `pwd` is equivalent. BASH_SOURCE[0] obviously only works with bash.
- account42 3y ago
  > You don't need realpath Yes you do (or a suitable alternative, e.g. GNU readlink). > The `cd` into the directory + `pwd` is equivalent. You won't know which directory to cd to without realpath. Just stripping off the filename does not get you the correct directory if the file itself is a symlink, which is a common use case.
  - js2 3y ago
    Doh, you're right. Thank you for the correction.
arp242 3y ago
/bin/sh on FreeBSD isn't bash.
peebeebee 3y ago
On MacOS, we had interoperability uses between different versions somehow, and some of us could not use `realpath` . The following worked for everyone: SCRIPT_DIR=$(dirname "$(readlink -f "${BASH_SOURCE[0]}")").

jchw 3y ago

In addition to other details noted about escaping/quoting and the portability of realpath, it's important to realize that those paths by default are not relative to the invocation directory, they're relative to the current working directory, which happens to be the current working directory of the parent process initially. Also, for a shell script it is relatively reliable to rely on $0, but not guaranteed by any means: for example, an obvious way to subvert this would be to pipe the content of the script into a shell, though there's surely plenty of ways you can think of that this could be broken.

AsthmaBoy 3y ago

Wouldn't readlink -f $0 work to get the actual location of the script, and from there get the base for relative scripts?

Asooka 3y ago
The -f flag is a GNU extension. Using realpath is more portable.

halayli 3y ago

Here's a good explanation from SO that I happen to have in my bookmarks: https://stackoverflow.com/questions/29832037/how-to-get-script-directory-in-posix-sh/29835459#29835459 https://stackoverflow.com/questions/29832037/how-to-get-scri...

n_plus_1_acc 3y ago
Thanks, I will use that from now on as a reference when somebody asks me why I don't like to program longer things in shell.

ChoHag 3y ago

[dead]

Y_Y 3y ago

What about hard links dude? Symlinks are fine because they're "fake", but it's not like a file must have a single canonical location.

andsens 3y ago
In that case whoever is linking the script is “breaking” it, I’d say. There’s stuff that is simply outside the responsibilities of the script/maintainer. If you start worrying about that for “simple” scripts, the logic for handling that is quickly going to outgrow the initial logic itself.
- account42 3y ago
  Besides, hard links don't have a canonical target, they are all equally correct names for the file. So even if the script author wanted to support the same technique for hard links there is no reasonable way to do it.
rollcat 3y ago
I think symlinks (or hard links, doesn't matter) are the actual problem here. It is actually completely OK for your code to make certain assumptions about how it's going to be invoked, and otherwise break (loudly, and in an obvious way, if possible).

febusravenga 3y ago

Creating proper `npm` package that works in many popular variants of J*script environment slowly but surely gets to same level of complexity.

gkfasdfasdf 3y ago

FYI realpath was not available by default on MacOS until very recently as others point out: https://news.ycombinator.com/item?id=38514413 https://news.ycombinator.com/item?id=38514413 Prior to that you would need to do 'brew install coreutils' to get realpath.

a-french-anon 3y ago

As someone who has fought a long time against holes in POSIX sh (cf https://git.sr.ht/~q3cpma/scripts https://git.sr.ht/~q3cpma/scripts README with links to the Austin group Mantis), let me list my biggest pet peeves (including this one): * mktemp -d (mktemp itself is available via POSIX m4; the noclobber solution mentioned is moot, since FIFOs break it https://austingroupbugs.net/view.php?id=1364 https://austingroupbugs.net/view.php?id=1364), very much needed and available on GNU, *BSD and MacOS. * local or typeset: I think only ksh88/93 doesn't have it without using the non-standard "function" keyword, a runtime check with potential alias creation is my only "solution" (https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a953446908936cec25b0ca847df626/item/util.sh#L5 https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a9534469089...) * realpath/readlink -f/-e: as said by some, realpath is getting standardized in the next POSIX; can be emulated via a cumbersome function (https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a953446908936cec25b0ca847df626/item/readlinkf https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a9534469089...) but since you either put that massive block in your script or need realpath to reliably source it from another file... * All the "newline in paths" mess: I gave up, just too many blind spots.

ykonstant 3y ago
Indeed, even the script you linked is not good enough; it does not deal with trailing newlines in the file or directory name, and the ls parsing will fail with a file like 'my malicious -> file'.
o11c 3y ago
Another major hole (in all shells!): there's no way to perform quote-aware splitting of a string in a restricted manner. This is especially egregious given the lack of array variable support. `xargs` is close, but only handles a few kinds of quotes, not e.g. `$''` which is the best kind of quoting, and notably does not adapt to the current shell. `printf %q` is also pretty important to avoid injection attacks, though if you don't care about ugliness or performance you can hack it together pretty easily (but beware of trailing newlines in subshells!). (also, I am obligated to point out that BSDers are lying when they say they're using "Bourne shell". FreeBSD uses a fork of `ash`, related to `dash` and `busybox ash`, but all 3 have evolved independently. OpenBSD uses a derivative of `pdksh` (other BSDs also ship derivatives but not as /bin/sh). NetBSD has their own from-scratch shell, but NetBSD is dead and irrelevant. I have a whole tree of shells; `ash` is notable for being the most unpredictable within the tree.)

H4ZB7 3y ago

i like how thanks to un*x mentality i can just expect argv to be the most malicious possible implementation where the invoker can put anything in it and it will be there for the invokee, on the os family boasting its support for secure multi user systems: /usr/sbin/./././some_admin_tool /usr/sbin/../sbin/../sbin/some_admin_tool /my_rigged_folder/haha (using symlink) but i guess it would have required allocating a string or something which may have mattered 40 years ago even though we could have had a better OS in every way 20 years ago and the same for the process list (in multiple ways, one you can see other people's passwords if they do HISTFILE=/dev/null something --password password, two you can just modify the process list) and i get the last laugh because un*x lovers will reply to criticism such as "your CLI shit is retarded it logs the passwords to disk" with "NO BRO I USE HISTFILE=/dev/null" to which i reply "that's a lot of shit to type just to not log your password to disk" to which he replies with "NO BRO I USE SPACE PREFIX" and he's still wrong due to previous paragraph. similarily the un*x lover will boast about how he can detect malware / bad guys by using the process list yet nope it can just be bypassed like anything in un*x. and don't even get me started with how moronic it is for a CLI program to "prompt" the user for a password to avoid this issue (actually you can use patches to make CLI paramters not shared to all users of the system, which is still stupid because none of that type of information should ever be shared in he first place: no other user's process should see any attribute of mine) and the icing on the cake is the typical way people inside the industry are completely oblivious and say "no man this is actually amazing stuff here" > It's possible to write extremely powerful tools using shell scripts, and modularize your code. Bourne shell doesn't provide a clear mechanism for referencing relative files. By establishing a command file's base dir, you can reliably reference relative shell scripts. wow a turing complete language is powerful who knew now if only we had one that wasn't just a cesspit of footguns most of which are there for no real reason and even historically were still invalid

samgranieri 3y ago

I didn't know about this. I'm writing bash scripts in the scripts-to-rule-them-all fashion for work and this can help DRY things up

extraduder_ire 3y ago

Most scripts I write cd into the directory they are located in for this reason. (and so that nohup.out files don't clobber each other in my home folder) Hopefully I can keep this in mind if I ever have to symlink them.

_ea1k 3y ago
+1 - it really irritates me when some script insists that I change into its directory in order to work. Its just lazy script writing.
- twic 3y ago
  It can also be annoying when a script changes directory itself. If anything the script does is relative to the current directory (interpreting a relative path, creating output files), you now can't have that be relative to your current working directory. Best practice, IMHO, is usually for scripts not to touch the current directory, and to be agnostic to it.

abathur 3y ago

One approach to this problem that’s been fruitful in Nix ecosystem is ~packaging scripts and baking in absolute paths at build time. (Generally by rewriting references in the script to absolute paths or generating wrappers to set PATH or BASE or DIR or whatnot.)

chubot 3y ago

This has bugged me for a long time too – on Linux you can use readlink -f to get realpath, but that’s not available on OS X. So I built it into Oils (both OSH and YSH): source $_this_dir/lib/util.ysh $_this_dir is a special variable that’s relative to the file it’s being sourced from. https://www.oilshell.org/release/0.19.0/doc/ref/chap-special-var.html#_this_dir https://www.oilshell.org/release/0.19.0/doc/ref/chap-special... https://www.oilshell.org/release/latest/doc/ysh-tour.html#appendix-example-of-an-ysh-module https://www.oilshell.org/release/latest/doc/ysh-tour.html#ap... It starts with a _ to let you know it’s magic, similar to _status and _process_sub_status (the latter being unavailable in any shell besides Oils BTW) So yeah this one reason I say POSIX and shell are “rotting” – they lack extremely basic functionality like this. This could have been done 20 years ago, but there’s nobody really pushing it. (We may add namespaces to shell, which will change some things in the example above, but it won’t change _this_dir. We can use feedback and help! - See recent story https://lobste.rs/s/h4egaz/oils_winter_status_update https://lobste.rs/s/h4egaz/oils_winter_status_update ) (copy of lobste.rs comment - https://lobste.rs/s/4emmdl/sh_relative_shell_script_includes_with https://lobste.rs/s/4emmdl/sh_relative_shell_script_includes...) more: Oils Winter Status Update - https://www.oilshell.org/blog/2023/11/status-update.html https://www.oilshell.org/blog/2023/11/status-update.html

twic 3y ago

I reject the symlink problem as not real, and i think the solution is harmful. Here's my expectation: if a script is linked into some location, and invoked at that location, it should behave the same as if it had been copied to that location. If it wouldn't work if it was copied, it isn't reasonable to expect that it should work if it's symlinked. There's no point catering to that use case. Moreover, let's say i symlink the including script, then copy the included script to the right relative position, then edit the included script. I would expect that running the symlinked including script should include the copied and modified included script, not the original. In general, i think programmers are far too eager to apply realpath or readlink -f to paths. If a path has symbolic elements in it, it's because someone has put them there on purpose. Don't just remove them. It's actually very rare to really need to know the canonical path to a file.

seadan83 3y ago
Where do you see the harm in the solution? To your other point, I think this is a valid use-case and do not agree with your rejection. I have had experience with such very examples and submitted PRs to add `realpath` to fix them. One example, a linting app I installed put itself in `/usr/local/<linter-name>/bin/<linter-executable>`. It was really painful to type this full path every time (even with tab completion). I created a symlink from `/usr/bin/` to the executable, and voila, the command is on my path now and easy to use; except the script does not expect itself to be a symlink and broke. To work around this, I created an actual script in `/usr/bin` that could invoke the utility (which then worked because it did properly detect path at least with $0). I don't think that extra script should have been necessary. Further, if I only updated my path, then `dirname $0` fails (returns '.') and that was not even a solution. Second question, as a counter-point, couldn't someone also reject the expectation that scripts should work when invoked from any path at all? That is saying, as a script writer, when you file a bug report, I can respond "closed, this script requires a `cd` to the install directory and only then can be run. RTFM!". At what point is this just ignoring UX?
- twic 3y ago
  The harm i see is what i describe under "moreover" - inconsistency with respect to copying. In your linter example, i think the most normal solution would be to add /usr/local/<linter-name>/bin to your PATH. Your solution, of a script which just does exec /usr/local/<linter-name>/bin/<linter-name> "$@", also seems fine to me - quite simple, and less likely to cause weird behaviour than a symlink. I'm not sure what point you're making at the end. We agree that a script not working when invoked from anywhere is bad UX, because people expect to be able to do that. I think that scripts not working through a symlink is not bad UX, because people can't expect to be able to do that, because in many cases that doesn't work!