6 ms·

The roots of an obscure Bourne shell error message

95 points by r4um 3y ago

bdamm 3y ago
And today I learned about shellcheck. Thank you for that fun and concise trip into shell evaluation :)
pinusc 3y ago
Excellent explanation! However, I would disagree both that there isn't an easy fix and that each of the decisions are sensible. If the syntax to temporarily set variables was different, say using : instead of =, or requiring some delimiter between assignments and operations, or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments would have been reasonable alternatives, perhaps even more readable than this. Or having the sntax disallow `foo=`, requiring, say, `foo=null` or `foo=''`, which are both more explicit, or even `unset foo`. These are more sensible individual choices, and that's partly because they're more explicit and partly because they avoid a (now) common error. I'd argue good language design needs to think slightly bigger picture than "individually not catastrophic" choices. Sh is littered with a myriad of small bad choices, which add up to a language that's hard to code or script without bugs. Nowadays with shellcheck it's easier, but it shouldn't be necessary. The consensus is: it's a scripting language, it shouldn't be used for complex stuff. But if it was a slightly better scripting language, it _could_ be used more safely for larger or more complex scripts. The truth is that it's an ancient language, its (bad) syntax is set in stone, and there's nothing we can do about it except slowly move to alternatives. Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure). I know how and love to write in bash. But oh god was it painful to learn
- __float 3y ago
  > I know how and love to write in bash. But oh god was it painful to learn This is very well said :) > Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure). Is there anything notable? Either particularly well designed, or just popular? I think I've only ever heard of Oil (https://oilshell.org https://oilshell.org)
  - gavinray 3y ago
    Ruby has very ergonomic and pleasant UX for doing the sorts of things you typically want to do in bash scripts IMO.
- Izkata 3y ago
  > or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments As long as you don't need to get anything out of it, it's a bit heavy but dropping to a subshell works like that: $ (foo=bar; echo $foo) bar $ echo $foo
- Galanwe 3y ago
  I don't think your comment fairly represents the situation. Bash is not _just_ a scripting language, it's also (and primarily) a shell. That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting. Being a shell implies that it has to be exceptionally succinct and flexible about executing programs and manipulating environment variables. In that light, your comparison with Python and Perl don't make much sense. Would you enjoy replacing: $ cd /tmp $ ls -l with: $ os.chdir('/tmp') $ subprocess.run(['ls', '-l']) I guess not. Shells will always be a different breed of languages. Bash is maybe not the best at what it could be, but it's not the end of the world either. The example in the article is a bit silly to be honest. I think the result is kind of obvious for anyone half bash litterate.
  - jhoechtl 3y ago
    > That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting. The creators of powershell obviously were of a different opinion
    - Kwpolska 3y ago
      It is fairly good for interactive usage, there are aliases for common commands, case-insensitivity, tab completion etc.
  - pinusc 3y ago
    You are right to point out that in my comment I talk about bash in terms of a language rather than just a shell; but I did have its shell-nature in mind while writing it, and in fact your example is exactly what I meant when I said that python is not good enough unix glue... there's a certain level of script complexity where python starts making more sense than bash, but because the ergonomics of calling other programs (unix glue!) or building pipelines are pretty poor, that threshold is pretty high. Some of bash syntax makes perfect sense in the context of a shell-but-also-scripting-language. Some of it makes perfect sense in the context of "everything is a string", which I think is an absolutely great abstraction---for example, unset variables being equivalent to empty strings. Other parts are just jargony, and might make the life of a few experienced shell-slingers slightly easier, while complicating many users' interactions with the shell. I suppose part of my point is that succintness here comes at the cost of readability. A shell could very well be _slightly_ more verbose than bash (say, requiring assignment operators to have two sides, so `foo=''` instead of `foo=`), still be plenty succint as a shell, but being slightly more explicit and readable in a scripting context. `foo=''` is only _two_ characters longer than `foo=`, and is already-implemented behavior. But the `foo=` syntax has certainly confused generations of unix users... that's just not a good tradeoff imo. I feel pretty comfortable calling this lack of foresight, or suboptimal design; that's not bashing (!) the language or its creators, rather, merely noting that we can do better. And if you don't think so... check out fish shell! It's a beloved and successful shell, and they forgo the use of = entirely (instead, you have to `set foo bar`, which is more verbose and a lot more explicit!) Edit: fish does allow `foo=bar echo $foo` to override variables, so they actually manage to keep bash ergonomics while avoiding mistakes while scripting.
1vuio0pswjnm7 3y ago
This does not seem obscure to me at all. Having learned about C from reading djb's programs, I write programs that read their arguments from environmental variables. (See djbdns for example.) When I run one of these programs it's common to set variables on the command line, e.g., echo https://example.com|Connection=keep-alive yy025|nc -vvn 127.127 80 yy025 sets the HTTP Connection header according to the value of the corresponding environmental variable. If I want to send a POST request I set the variable httpMethod to POST. The address 127.0.0.127 is a TLS forward proxy. Original netcat allows one to type 127.127 without the zeros. Using djb's envdir utility, one can put a selection of different HTTP headers into a directory so that there's no need to type them on the command line. cd /path/to/dir echo close > connection echo aplication/x-www-form-urlencoded > content-type echo id=123\;key=xyz > cookie echo 19 > content-length echo POST > httpMethod echo /api/whatever > path echo "{ \"key\": \"value\" }" > post-data cd echo https://example.com \ |if cd /path/to/dir;envdir . yy025;then cat post-data;fi \ |nc -vvn 127.127 80 It's strange to me to see this author use printenv instead of set (a built-in) to display Bourne shell environmental variables. As someone who uses Bourne shell every day and on a variety of computers, I never use printenv. On the smaller systems I make, I do not include printenv in the userland. The use of fgrep to match a single word in all caps is also strange. Maybe I am just missing some UNIX wisdom here. If so, please disregard this comment. From the article $ export FRED=value $ printenv | fgrep FRED FRED=value $ FRED= $ printenv | fgrep FRED FRED= $ unset FRED $ printenv | fgrep FRED $ # ie, no output from printenv Without using printenv or grep -F (in shell script /bin/fgrep) $ export FRED=value $ set|grep FRED FRED='value' FRED= $ set|grep FRED FRED='' $ unset FRED $ set|grep FRED $ # ie, no output from set
- zerocrates 3y ago
  A lot of people are probably just not familiar with the single-command envvar syntax, and of people that do know it some of them probably don't know the syntax for unsetting. So it's just not immediately obvious to many why a simple extra space is a problem at all, and why it's this specific error. It can also just generally be hard to see past what you intended to write to what you actually did and how that will be parsed.
- 3y ago
kazinator 3y ago
[flagged]
da39a3ee 3y ago
For what it’s worth, the explanation of the error was obvious to me after seeing the code for a few seconds. So maybe my point is… don’t worry if stuff like that in shell programming seems obscure to you; it becomes much less obscure with practice (and along the way you do learn genuine lessons about unix-like operating systems, not just shell programming details).
- rob74 3y ago
  The problem is that most people (me included) don't want to become proficient in writing shell scripts, but they need to write a script (or, God forbid, extend and/or debug a script someone else wrote) from time to time. At least I do it so rarely that I forget most things from one time to the next. I understand that Bash is the way it is because of historical reasons, but maybe some less obscure error messages would be nice? ShellCheck is all nice and well, but Bash itself being more helpful would be better. Or, no, please don't tell me that other tools rely on these error messages and therefore they can't be changed?
  - da39a3ee 3y ago
    > maybe some less obscure error messages would be nice? Bear with me here! The error message is exactly what we’d expect in this situation. The author literally tried to use the string “107” as an executable name, and bash says ‘I can’t find an executable named “107”’. It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming. And the absolute number one thing to understand about bash and related shell languages is that “everything is text”. I know one hears people saying that a lot, but you’ve got to really internalize what it means. bash is just constructing strings of text and doing stuff with it. It barely has any non-string data structures.
    - Joker_vD 3y ago
      > The author literally tried to use the string “107” Well, no. That string was not literally used there at all: if it was, there would have been a literal 107 in the user's input. There wasn't. > It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming > bash is just constructing strings of text and doing stuff with it It's a reassuring knowledge in 2023, after 40 years of countless bugs, exploits, vulnerabilities and accidentally rm-ed important files, the paradigm of "just constructing strings of text and (blindly) doing stuff with it" is still ubiquitously used and one can not be a good software engineer without extensive knowledge of it. Bash truly deserves description of "a mistake, carried through to perfection. It is the language of the future for the programming techniques of the past: it creates a new generation of coding bums" much more than APL.
jlv2 3y ago
Sorry, that's not obscure!
- rob74 3y ago
  ...not if you live and breathe shell scripts, no. But most people coming from other programming languages first need to understand the concept of whitespace being significant, and even if they do, it's easy to insert a space accidentally. In this case, if you get this error message, you would probably first look at the part in the "$(...)" (which, you think, is the only thing resembling a "command" on that line) and wonder why on earth it doesn't work, overlooking the problem slightly to the left of it.
  - naniwaduni 3y ago
    I would expect to look at the part inside the $(), probably attempt to run it to confirm that it's working at all, and be a bit curious as to why the error message implies I was trying to run the output of that command!
  - blueflow 3y ago
    I remember yaml and python also having syntactically relevant whitespaces.
cmeacham98 3y ago
Can I suggest a simple fix that doesn't involve changing behavior at all but probably improves the lives of the many users who hit this? Just add an additional line to the error message for lines that look like this pattern: sh: 107: command not found Did you mean "AVAR=$(... | wc -l)"?
- hnben 3y ago
  I feel like that approach would create much effort to do properly. I.e. to be consistent, you would add a lot of such messages, and that would add a lot of complexity to the overall code base.

95 points by r4um 3y ago

And today I learned about shellcheck. Thank you for that fun and concise trip into shell evaluation :)

Excellent explanation! However, I would disagree both that there isn't an easy fix and that each of the decisions are sensible. If the syntax to temporarily set variables was different, say using : instead of =, or requiring some delimiter between assignments and operations, or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments would have been reasonable alternatives, perhaps even more readable than this. Or having the sntax disallow `foo=`, requiring, say, `foo=null` or `foo=''`, which are both more explicit, or even `unset foo`. These are more sensible individual choices, and that's partly because they're more explicit and partly because they avoid a (now) common error. I'd argue good language design needs to think slightly bigger picture than "individually not catastrophic" choices. Sh is littered with a myriad of small bad choices, which add up to a language that's hard to code or script without bugs. Nowadays with shellcheck it's easier, but it shouldn't be necessary. The consensus is: it's a scripting language, it shouldn't be used for complex stuff. But if it was a slightly better scripting language, it _could_ be used more safely for larger or more complex scripts. The truth is that it's an ancient language, its (bad) syntax is set in stone, and there's nothing we can do about it except slowly move to alternatives. Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure). I know how and love to write in bash. But oh god was it painful to learn

__float 3y ago
> I know how and love to write in bash. But oh god was it painful to learn This is very well said :) > Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure). Is there anything notable? Either particularly well designed, or just popular? I think I've only ever heard of Oil (https://oilshell.org https://oilshell.org)
- gavinray 3y ago
  Ruby has very ergonomic and pleasant UX for doing the sorts of things you typically want to do in bash scripts IMO.
Izkata 3y ago
> or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments As long as you don't need to get anything out of it, it's a bit heavy but dropping to a subshell works like that: $ (foo=bar; echo $foo) bar $ echo $foo
Galanwe 3y ago
I don't think your comment fairly represents the situation. Bash is not _just_ a scripting language, it's also (and primarily) a shell. That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting. Being a shell implies that it has to be exceptionally succinct and flexible about executing programs and manipulating environment variables. In that light, your comparison with Python and Perl don't make much sense. Would you enjoy replacing: $ cd /tmp $ ls -l with: $ os.chdir('/tmp') $ subprocess.run(['ls', '-l']) I guess not. Shells will always be a different breed of languages. Bash is maybe not the best at what it could be, but it's not the end of the world either. The example in the article is a bit silly to be honest. I think the result is kind of obvious for anyone half bash litterate.
- jhoechtl 3y ago
  > That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting. The creators of powershell obviously were of a different opinion
  - Kwpolska 3y ago
    It is fairly good for interactive usage, there are aliases for common commands, case-insensitivity, tab completion etc.
- pinusc 3y ago
  You are right to point out that in my comment I talk about bash in terms of a language rather than just a shell; but I did have its shell-nature in mind while writing it, and in fact your example is exactly what I meant when I said that python is not good enough unix glue... there's a certain level of script complexity where python starts making more sense than bash, but because the ergonomics of calling other programs (unix glue!) or building pipelines are pretty poor, that threshold is pretty high. Some of bash syntax makes perfect sense in the context of a shell-but-also-scripting-language. Some of it makes perfect sense in the context of "everything is a string", which I think is an absolutely great abstraction---for example, unset variables being equivalent to empty strings. Other parts are just jargony, and might make the life of a few experienced shell-slingers slightly easier, while complicating many users' interactions with the shell. I suppose part of my point is that succintness here comes at the cost of readability. A shell could very well be _slightly_ more verbose than bash (say, requiring assignment operators to have two sides, so `foo=''` instead of `foo=`), still be plenty succint as a shell, but being slightly more explicit and readable in a scripting context. `foo=''` is only _two_ characters longer than `foo=`, and is already-implemented behavior. But the `foo=` syntax has certainly confused generations of unix users... that's just not a good tradeoff imo. I feel pretty comfortable calling this lack of foresight, or suboptimal design; that's not bashing (!) the language or its creators, rather, merely noting that we can do better. And if you don't think so... check out fish shell! It's a beloved and successful shell, and they forgo the use of = entirely (instead, you have to `set foo bar`, which is more verbose and a lot more explicit!) Edit: fish does allow `foo=bar echo $foo` to override variables, so they actually manage to keep bash ergonomics while avoiding mistakes while scripting.

1vuio0pswjnm7 3y ago

This does not seem obscure to me at all. Having learned about C from reading djb's programs, I write programs that read their arguments from environmental variables. (See djbdns for example.) When I run one of these programs it's common to set variables on the command line, e.g., echo https://example.com|Connection=keep-alive yy025|nc -vvn 127.127 80 yy025 sets the HTTP Connection header according to the value of the corresponding environmental variable. If I want to send a POST request I set the variable httpMethod to POST. The address 127.0.0.127 is a TLS forward proxy. Original netcat allows one to type 127.127 without the zeros. Using djb's envdir utility, one can put a selection of different HTTP headers into a directory so that there's no need to type them on the command line. cd /path/to/dir echo close > connection echo aplication/x-www-form-urlencoded > content-type echo id=123\;key=xyz > cookie echo 19 > content-length echo POST > httpMethod echo /api/whatever > path echo "{ \"key\": \"value\" }" > post-data cd echo https://example.com \ |if cd /path/to/dir;envdir . yy025;then cat post-data;fi \ |nc -vvn 127.127 80 It's strange to me to see this author use printenv instead of set (a built-in) to display Bourne shell environmental variables. As someone who uses Bourne shell every day and on a variety of computers, I never use printenv. On the smaller systems I make, I do not include printenv in the userland. The use of fgrep to match a single word in all caps is also strange. Maybe I am just missing some UNIX wisdom here. If so, please disregard this comment. From the article $ export FRED=value $ printenv | fgrep FRED FRED=value $ FRED= $ printenv | fgrep FRED FRED= $ unset FRED $ printenv | fgrep FRED $ # ie, no output from printenv Without using printenv or grep -F (in shell script /bin/fgrep) $ export FRED=value $ set|grep FRED FRED='value' FRED= $ set|grep FRED FRED='' $ unset FRED $ set|grep FRED $ # ie, no output from set

zerocrates 3y ago
A lot of people are probably just not familiar with the single-command envvar syntax, and of people that do know it some of them probably don't know the syntax for unsetting. So it's just not immediately obvious to many why a simple extra space is a problem at all, and why it's this specific error. It can also just generally be hard to see past what you intended to write to what you actually did and how that will be parsed.
3y ago

kazinator 3y ago

[flagged]

da39a3ee 3y ago

For what it’s worth, the explanation of the error was obvious to me after seeing the code for a few seconds. So maybe my point is… don’t worry if stuff like that in shell programming seems obscure to you; it becomes much less obscure with practice (and along the way you do learn genuine lessons about unix-like operating systems, not just shell programming details).

rob74 3y ago
The problem is that most people (me included) don't want to become proficient in writing shell scripts, but they need to write a script (or, God forbid, extend and/or debug a script someone else wrote) from time to time. At least I do it so rarely that I forget most things from one time to the next. I understand that Bash is the way it is because of historical reasons, but maybe some less obscure error messages would be nice? ShellCheck is all nice and well, but Bash itself being more helpful would be better. Or, no, please don't tell me that other tools rely on these error messages and therefore they can't be changed?
- da39a3ee 3y ago
  > maybe some less obscure error messages would be nice? Bear with me here! The error message is exactly what we’d expect in this situation. The author literally tried to use the string “107” as an executable name, and bash says ‘I can’t find an executable named “107”’. It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming. And the absolute number one thing to understand about bash and related shell languages is that “everything is text”. I know one hears people saying that a lot, but you’ve got to really internalize what it means. bash is just constructing strings of text and doing stuff with it. It barely has any non-string data structures.
  - Joker_vD 3y ago
    > The author literally tried to use the string “107” Well, no. That string was not literally used there at all: if it was, there would have been a literal 107 in the user's input. There wasn't. > It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming > bash is just constructing strings of text and doing stuff with it It's a reassuring knowledge in 2023, after 40 years of countless bugs, exploits, vulnerabilities and accidentally rm-ed important files, the paradigm of "just constructing strings of text and (blindly) doing stuff with it" is still ubiquitously used and one can not be a good software engineer without extensive knowledge of it. Bash truly deserves description of "a mistake, carried through to perfection. It is the language of the future for the programming techniques of the past: it creates a new generation of coding bums" much more than APL.

jlv2 3y ago

Sorry, that's not obscure!

rob74 3y ago
...not if you live and breathe shell scripts, no. But most people coming from other programming languages first need to understand the concept of whitespace being significant, and even if they do, it's easy to insert a space accidentally. In this case, if you get this error message, you would probably first look at the part in the "$(...)" (which, you think, is the only thing resembling a "command" on that line) and wonder why on earth it doesn't work, overlooking the problem slightly to the left of it.
- naniwaduni 3y ago
  I would expect to look at the part inside the $(), probably attempt to run it to confirm that it's working at all, and be a bit curious as to why the error message implies I was trying to run the output of that command!
- blueflow 3y ago
  I remember yaml and python also having syntactically relevant whitespaces.

cmeacham98 3y ago

Can I suggest a simple fix that doesn't involve changing behavior at all but probably improves the lives of the many users who hit this? Just add an additional line to the error message for lines that look like this pattern: sh: 107: command not found Did you mean "AVAR=$(... | wc -l)"?

hnben 3y ago
I feel like that approach would create much effort to do properly. I.e. to be consistent, you would add a lot of such messages, and that would add a lot of complexity to the overall code base.