5 ms·

The “best practice” of preferring higher level languages is questionable when it comes to System level tasks such as this one. If it’s something that needs to

by pm90 5y ago

The “best practice” of preferring higher level languages is questionable when it comes to System level tasks such as this one. If it’s something that needs to be part of a complex system, sure higher level langs can be powerful. If it’s a standalone thing, shell scripts are generally ok. In fact higher level languages add complexity: their build process, library updates, breaking changes of dependencies all make it a pain to maintain. Shell scripts? “ls” is never gonna change. IMO the biggest reason shell scripts are avoided is simply because most software developers aren’t very comfortable with the syntax (tbh it does feel archaic in certain places) or know how unixy stuff like signals, pipes etc really work. If you have to do system level tasks though it’s probably worth the effort to understand these things.

jbverschoor 5y ago
The problem with shell scripts is the same as automation a gui. It’s flakey and hard to maintain. Grepping through output No uniform handling of spaces, tabs. No data structures. Multiple dialects. Shell scripts mainly work with files and process management. So many scripts don’t handle spaces correctly. This is not a problem of the creator, but because of the language.
- zelphirkalt 5y ago
  Although not following best practices when writing shell scripts is the fault of the author. For example not thinking about spaces and using $varname directly instead of adhering to best practice and enclosing it in "${...}". It is uninformed and one should inform oneself, if one is to write a shell script. Plus there is no shame in looking up, how to do it correctly. There is little excuse for getting that one wrong. I mean, if this is like one of the first shell scripsts one writes, OK, but a seasoned engineer ... nah.
  - pm90 5y ago
    There’s also automations like shellcheck that figure this out for you.
  - jbverschoor 5y ago
    Well sure.. Then everybody, please stop blaming C, C++, php or wordpress. If it's the "wrong" way, remove it. Don't even deprecate it. Just hard throw an error. There are 10 tiny buttons next to each other. You press the wrong one. Was that your fault or the fault of the designer? Stop blaming "the user". In this case, not even a software engineer, but usually a sysop or power user.
    - zelphirkalt 5y ago
      No one (not that I have seen a comment like that anyway) is claiming, that shell scripts are not gnarly in some way or the other. However, this gnarliness of a language is of course a gradient, not an on-off, 1 or 0 kind of thing. Shell scripts are surely further on the gnarly end of things than some other things. If you ask me personally, I would not start a new project in any of the languages or frameworks you mentioned. I still do not agree though, that the user is completely not at fault, when there are easily avoided traps (if that is what you are proposing?). Of course those should ne even exist. I agree with that obviously. Blaming every single mistake on the language design, which might be gnarly, is not what I think is the correct thing to do though. We do not have the perfect language yet (or might never have it). So some gnarliness is to be expected and that means, that a realistic perspective is to educate oneself about those gnarly things and avoid mistakes. Mistakes are human, so there will be mistakes, but we should try to avoid cheap ones out of laziness or being uninformed. Also it is totally fine to criticize a language for its design flaws. I like to do that often even, with mainstream languages, because they can learn loads of stuff from less mainstream ones. This is the way to create awareness of those flaws and thus a way to improvement.
inferiorhuman 5y ago
> IMO the biggest reason shell scripts are avoided is simply because most software developers aren’t very comfortable with the syntax (tbh it does feel archaic in certain places) or know how unixy stuff like signals, pipes etc really work. If you have to do system level tasks though it’s probably worth the effort to understand these things. Hard disagree on that one. People aren't comfortable with the syntax because it's rather awkward and the subset of portable syntax is minuscule. The syntax itself has all sorts of variations (e.g. bash 3 vs 5 vs not-bash) and when you have to leverage external programs (e.g. sed or awk) things can get hairy quickly (especially if you're trying to support both BSD and GNU userland components). From a security POV I'd much rather use not-sh style stuff. Error handling is much more accessible and robust in higher level scripting languages. Sure you can run "set -e" and hope for the best... until you have to use something that tries to stuff meaning into the exit code like grep. Handling filenames with spaces is significantly easier once you leave the sh baggage behind. As for ls(1), are you talking about BSD or GNU ls?
- t3we334 5y ago
  If you don't standardize on a sane shell then you shouldn't use the unix shell, agreed. Total disagreement on security fitness of high level languages -vs- a shell script.
jseban 5y ago
> most software developers aren’t very comfortable with the syntax Yeah which is the same reason why sql is avoided, regex is avoided, javascript is avoided etc. People will go through enormous trouble to avoid learning anything other than the multi threaded c-style programming model from the 1970s, that they learned in their computer science degree.
- pm90 5y ago
  > multi threaded c-style programming model from the 1970s, that they learned in their computer science degree. Yep, most folks learn programming in high level languages, lower level and “glue” scripts feels “not right”. I can attest to this… after using one shitty orm after another, learning sql and bash made me a much better programmer.
dimitar 5y ago
It's best practice for longer scripts. I think anything over 100 lines is better off rewritten in a general purpose language as it is bound to have the complexity they are designed to solve and it will outweight the drawbacks.
GuB-42 5y ago
> “ls” is never gonna change. It has changed in the past and it will change in the future. "ls" will always list files in some way, but it is the only thing you can take for granted. You don't really know what kind of decoration you will get, how spaces and non-ascii characters will appear, the language, the order, etc... You may also find yourself with a weird environment variable your user has set that makes ls look better in his interactive shell and breaks your script. Generally, I avoid "ls" in scripts for that reason, but many commands have the same kind of problem. Your GNU script may break on busybox or BSD (happened to me many times). And good luck knowing what /bin/sh means, it may be bash, it is supposed to be POSIX, that's all we know.
- justaj 5y ago
  That's why it's a good rule of thumb to just stick with POSIX as defined by the POSIX standard [0] 0: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
  - GuB-42 5y ago
    Totally agree, tough I am not sure that all shells that pretend to be POSIX really are. But it is a minefield, it is way to easy to sneak-in a bashism because you are so sure it is POSIX that you didn't check, made worse by the fact that if you are writing shell scripts, you are probably using an much more liberal interactive shell all day long. And your script will work and no one will complain until you try to run it on another system. It is not entirely unlike client-side web technologies, where you have to check for browser compatibility. For the web, it is a recognized problem and there are plenty of solutions to deal with it: jquery, babel, etc... I am not aware of such tools for shell scripts, or maybe just analyzers that check for posix compliance.
    - justaj 5y ago
      That's easily solved with http://www.shellcheck.net/ http://www.shellcheck.net/

by pm90 5y ago

jbverschoor 5y ago

The problem with shell scripts is the same as automation a gui. It’s flakey and hard to maintain. Grepping through output No uniform handling of spaces, tabs. No data structures. Multiple dialects. Shell scripts mainly work with files and process management. So many scripts don’t handle spaces correctly. This is not a problem of the creator, but because of the language.

zelphirkalt 5y ago
Although not following best practices when writing shell scripts is the fault of the author. For example not thinking about spaces and using $varname directly instead of adhering to best practice and enclosing it in "${...}". It is uninformed and one should inform oneself, if one is to write a shell script. Plus there is no shame in looking up, how to do it correctly. There is little excuse for getting that one wrong. I mean, if this is like one of the first shell scripsts one writes, OK, but a seasoned engineer ... nah.
- pm90 5y ago
  There’s also automations like shellcheck that figure this out for you.
- jbverschoor 5y ago
  Well sure.. Then everybody, please stop blaming C, C++, php or wordpress. If it's the "wrong" way, remove it. Don't even deprecate it. Just hard throw an error. There are 10 tiny buttons next to each other. You press the wrong one. Was that your fault or the fault of the designer? Stop blaming "the user". In this case, not even a software engineer, but usually a sysop or power user.
  - zelphirkalt 5y ago
    No one (not that I have seen a comment like that anyway) is claiming, that shell scripts are not gnarly in some way or the other. However, this gnarliness of a language is of course a gradient, not an on-off, 1 or 0 kind of thing. Shell scripts are surely further on the gnarly end of things than some other things. If you ask me personally, I would not start a new project in any of the languages or frameworks you mentioned. I still do not agree though, that the user is completely not at fault, when there are easily avoided traps (if that is what you are proposing?). Of course those should ne even exist. I agree with that obviously. Blaming every single mistake on the language design, which might be gnarly, is not what I think is the correct thing to do though. We do not have the perfect language yet (or might never have it). So some gnarliness is to be expected and that means, that a realistic perspective is to educate oneself about those gnarly things and avoid mistakes. Mistakes are human, so there will be mistakes, but we should try to avoid cheap ones out of laziness or being uninformed. Also it is totally fine to criticize a language for its design flaws. I like to do that often even, with mainstream languages, because they can learn loads of stuff from less mainstream ones. This is the way to create awareness of those flaws and thus a way to improvement.

inferiorhuman 5y ago

> IMO the biggest reason shell scripts are avoided is simply because most software developers aren’t very comfortable with the syntax (tbh it does feel archaic in certain places) or know how unixy stuff like signals, pipes etc really work. If you have to do system level tasks though it’s probably worth the effort to understand these things. Hard disagree on that one. People aren't comfortable with the syntax because it's rather awkward and the subset of portable syntax is minuscule. The syntax itself has all sorts of variations (e.g. bash 3 vs 5 vs not-bash) and when you have to leverage external programs (e.g. sed or awk) things can get hairy quickly (especially if you're trying to support both BSD and GNU userland components). From a security POV I'd much rather use not-sh style stuff. Error handling is much more accessible and robust in higher level scripting languages. Sure you can run "set -e" and hope for the best... until you have to use something that tries to stuff meaning into the exit code like grep. Handling filenames with spaces is significantly easier once you leave the sh baggage behind. As for ls(1), are you talking about BSD or GNU ls?

t3we334 5y ago
If you don't standardize on a sane shell then you shouldn't use the unix shell, agreed. Total disagreement on security fitness of high level languages -vs- a shell script.

jseban 5y ago

> most software developers aren’t very comfortable with the syntax Yeah which is the same reason why sql is avoided, regex is avoided, javascript is avoided etc. People will go through enormous trouble to avoid learning anything other than the multi threaded c-style programming model from the 1970s, that they learned in their computer science degree.

pm90 5y ago
> multi threaded c-style programming model from the 1970s, that they learned in their computer science degree. Yep, most folks learn programming in high level languages, lower level and “glue” scripts feels “not right”. I can attest to this… after using one shitty orm after another, learning sql and bash made me a much better programmer.

dimitar 5y ago

It's best practice for longer scripts. I think anything over 100 lines is better off rewritten in a general purpose language as it is bound to have the complexity they are designed to solve and it will outweight the drawbacks.

GuB-42 5y ago

> “ls” is never gonna change. It has changed in the past and it will change in the future. "ls" will always list files in some way, but it is the only thing you can take for granted. You don't really know what kind of decoration you will get, how spaces and non-ascii characters will appear, the language, the order, etc... You may also find yourself with a weird environment variable your user has set that makes ls look better in his interactive shell and breaks your script. Generally, I avoid "ls" in scripts for that reason, but many commands have the same kind of problem. Your GNU script may break on busybox or BSD (happened to me many times). And good luck knowing what /bin/sh means, it may be bash, it is supposed to be POSIX, that's all we know.

justaj 5y ago
That's why it's a good rule of thumb to just stick with POSIX as defined by the POSIX standard [0] 0: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
- GuB-42 5y ago
  Totally agree, tough I am not sure that all shells that pretend to be POSIX really are. But it is a minefield, it is way to easy to sneak-in a bashism because you are so sure it is POSIX that you didn't check, made worse by the fact that if you are writing shell scripts, you are probably using an much more liberal interactive shell all day long. And your script will work and no one will complain until you try to run it on another system. It is not entirely unlike client-side web technologies, where you have to check for browser compatibility. For the web, it is a recognized problem and there are plenty of solutions to deal with it: jquery, babel, etc... I am not aware of such tools for shell scripts, or maybe just analyzers that check for posix compliance.
  - justaj 5y ago
    That's easily solved with http://www.shellcheck.net/ http://www.shellcheck.net/