5 ms·

I find that after a shell script gets to be over 3 or so lines, it's easier to switch over to python or perl. Do others feel the same?

by BCM43 13y ago

joshguthrie 13y ago
I was gonna write this. Shell script syntax has always been hard to remember to me (but I feel it to be less than intuitive compared to lisp, perl, js, c, ruby, python...) , subject to different behaviors between shells (csh vs ksh IIRC), string manipulation is close to non-existing, types are more difficult to use than in PHP,... Okay, it works, we can do stuff with it, we can hack a quick thingie there and there,... I know the main appeal is, "it just runs everywhere, no install required" (minus the csh/ksh differences), but is it really the tool we need for all we use it for?
- barrkel 13y ago
  The primary advantage of learning shell script syntax for me is that it's the same language I use on the command line. I write for and "while read" loops on the command line a dozen times a day, () subshells, process substitution, sed, awk etc. The more proficient I am on the command line, the more I am with shell scripts, and vice versa. Another advantage; the shell is a repl in which you can prototype your script. Perform the operations manually, pull the commands out of your history, and adjust as necessary.
  - joshguthrie 13y ago
    > I write for and "while read" loops on the command line a dozen times a day You earned my respect. I like REPLs for small operations. When talking about a "full" script, I'm more at ease in a text editor with a script I'm gonna adjust as I run it. Best example is a syntax error: the script just won't run. Meanwhile, in a REPL, I may get myself stuck without knowing it.
chrismonsanto 13y ago
For me, it has nothing to do with the number of lines, but the amount of actual computation I am doing. If all I need to do is wire up the inputs and outputs of X commands, then shell script is the best tool for the job. If I actually need to do something with an input or output before sending it to the next command, I'll switch to Python. For CLI tasks in Python, http://docopt.org/ http://docopt.org/ and http://shell-command.readthedocs.org/en/latest/ http://shell-command.readthedocs.org/en/latest/ are invaluable tools. Yes, I know about Envoy, but shell_command is much nicer to use in my opinion. Envoy doesn't automatically escape arguments, which is annoying for all the reasons that SQL injections are annoying.
- dllthomas 13y ago
  That's more true of python than perl, I think - my recollection is that python adds slightly more syntax on top of running external commands. That said, I happily assemble shell scripts of tens (or occasionally hundreds) of lines when it's a good fit...
- barrkel 13y ago
  If I actually need to do something with an input or output before sending it to the next command, I'll switch to Python. If it's not something that can be done trivially with sed or awk, I'll consider whether it's something that can be implemented in a generic way to be used elsewhere, in which case I'll write that specific utility, and insert it into the pipeline (with appropriate arguments). Only if it's fairly specific, or I need to package it for distribution, will I consider trying to rewrite everything in a single language script.
elwin 13y ago
It depends what it's doing. Other languages are much better at data processing. They make it more complicated to start child processes, use pipes, set environment variables, etc. If you're doing system administration tasks, you'll end up with a lot of shell command lines embedded in the program. (Try editing a crontab in Python...)
- icebraining 13y ago
  Try editing a crontab in Python... pip install python-crontab
laumars 13y ago
I don't agree with this. I love Perl, it's up there as one of my favourite languages (despite it's many short comings), but in my opinion shell scripts make more sense if you're writing scripts which depend upon a number of additional programs as the bulk of it's processing. For me, the point at which Perl makes more sense is if your script requires more internal logic than it does depends on spawning other programs. For example, if I'm writing a routine to auto snapshot ZFS / Btrfs volumes and delete any over a certain age, the script would be dependant on your file system CLI tools. So it makes more sense to have an 80+ line shell script than it does to write that in Perl / Python. However if I was writing a routine which requires users inputting details, where those details need to be sanity checked and then stored some where (such as a database), then the core logic of that program resides within your script (where you'd have to read inputs, do your sanity checks and then write to the database). So a Perl or Python script makes more sense. Obviously you can do either of those examples in each of those languages (crudely speaking as I know shell scripts aren't technically a programming language); but that's just a basic example of where I personally draw the line. I also think this is one of those occasions where it doesn't massively matter which approach you take just so long as the code works and is maintainable (though I draw the line at one maintenance script I saw last year. It was a Python script where every other line was os.system. It just struck me as rather pointless starting a Python interpreter if you're just going to use it like a shell script - you might as well do the whole lot in the shell to begin with).
- carbocation 13y ago
  Great comment. One nitpick: Bash, at least, is Turing-complete. https://en.wikibooks.org/wiki/Bash_Shell_Scripting https://en.wikibooks.org/wiki/Bash_Shell_Scripting
- jsight 13y ago
  *(though I draw the line at one maintenance script I saw last year. It was a Python script where every other line was os.system. It just struck me as rather pointless starting a Python interpreter if you're just going to use it like a shell script - you might as well do the whole lot in the shell to begin with).* FWIW, one of my favorite features of Perl syntax is that you can do things like this. A lot of my quick-and-dirty sysadmin scripts end up being Perl scripts with lots of backticks. It's handy for when they grow (as they often do) into more full-featured scripts.
emmelaich 13y ago
Yep pretty much, but it can be up to 100 lines depending on the nature of the script. To the article, I've never used a shell that doesn't understand [[ (guaranteed to be builtin, not sensitive to the hyphen issue) and {,} expansion. For the "for file in *; ... " that is also susceptible to the shell arg limits and is fairly hairy. Lastly if you're concerned about performance -- well, it's definitely time to leave shell for something better.
- nandhp 13y ago
  > I've never used a shell that doesn't understand [[ That may be, but that doesn't mean they aren't pretty widespread. [[ and {,} are bashisms. However, they still work when bash is executed as /bin/sh, as is the case on RedHat and some other Linux distributions. However, other distributions, including Debian, Ubuntu, and any embedded system using Busybox, point the /bin/sh symlink to a version of ash, the Almquist shell. It is used for scripts instead of bash because it is much smaller and faster (reportedly, Debian/Ubuntu boot speed improved significantly -- leaving the shell isn't a very good option for sysvinit).
  - emmelaich 13y ago
    [[ is from ksh88 or earlier; {,} is apparently from csh. (Funny; I could've sworn I used {} in ksh under Solaris but apparently not!) Your point stands though, they are bashisms for all intents and purposes on Linux. As to speed -- well, again, I think if you want speed don't use shell. For startup you're better off redesigning the whole damn thing (cue systemd). Changing shell is like overclocking your cpu when you should be choosing a better algorithm.
jmount 13y ago
Yes. I just promised myself yesterday to stop writing bash scripts for things other than setting up an environment and launching another program. What pushed me over the edge is most of the script was incorporating the kind of fixes from the article (force variable to be treated as int, add dollars signs to access as variable, and so on).
nilved 13y ago
It depends on the goal of the script. I tend to rewrite complicated shell scripts in Ruby, but working with subprocesses in Ruby is terrible, so those scripts are destined to end in `.sh` forever. bash has a very specific purpose where it excels over scripting languages like Python and Ruby: wiring input and output and communicating with subprocesses.
barrkel 13y ago
It entirely depends on what the script is doing. If it's orchestrating a pipeline of processes, shell script is ideal. If it needs much in the way of complicated temporary data structures, shell is the worst way to go.
ams6110 13y ago
Depends on the target. Python, and even perl is not always available. /bin/sh is.
- jes5199 13y ago
  What systems don't have an interpreter installed these days?
  - fragsworth 13y ago
    Well, cygwin won't have it by default
coolsunglasses 13y ago
I've gotten in the habit of using Python or Haskell for my automation. Pretty happy with it.

I find that after a shell script gets to be over 3 or so lines, it's easier to switch over to python or perl. Do others feel the same?

by BCM43 13y ago

I find that after a shell script gets to be over 3 or so lines, it's easier to switch over to python or perl. Do others feel the same?

joshguthrie 13y ago

I was gonna write this. Shell script syntax has always been hard to remember to me (but I feel it to be less than intuitive compared to lisp, perl, js, c, ruby, python...) , subject to different behaviors between shells (csh vs ksh IIRC), string manipulation is close to non-existing, types are more difficult to use than in PHP,... Okay, it works, we can do stuff with it, we can hack a quick thingie there and there,... I know the main appeal is, "it just runs everywhere, no install required" (minus the csh/ksh differences), but is it really the tool we need for all we use it for?

barrkel 13y ago
The primary advantage of learning shell script syntax for me is that it's the same language I use on the command line. I write for and "while read" loops on the command line a dozen times a day, () subshells, process substitution, sed, awk etc. The more proficient I am on the command line, the more I am with shell scripts, and vice versa. Another advantage; the shell is a repl in which you can prototype your script. Perform the operations manually, pull the commands out of your history, and adjust as necessary.
- joshguthrie 13y ago
  > I write for and "while read" loops on the command line a dozen times a day You earned my respect. I like REPLs for small operations. When talking about a "full" script, I'm more at ease in a text editor with a script I'm gonna adjust as I run it. Best example is a syntax error: the script just won't run. Meanwhile, in a REPL, I may get myself stuck without knowing it.

chrismonsanto 13y ago

For me, it has nothing to do with the number of lines, but the amount of actual computation I am doing. If all I need to do is wire up the inputs and outputs of X commands, then shell script is the best tool for the job. If I actually need to do something with an input or output before sending it to the next command, I'll switch to Python. For CLI tasks in Python, http://docopt.org/ http://docopt.org/ and http://shell-command.readthedocs.org/en/latest/ http://shell-command.readthedocs.org/en/latest/ are invaluable tools. Yes, I know about Envoy, but shell_command is much nicer to use in my opinion. Envoy doesn't automatically escape arguments, which is annoying for all the reasons that SQL injections are annoying.

dllthomas 13y ago
That's more true of python than perl, I think - my recollection is that python adds slightly more syntax on top of running external commands. That said, I happily assemble shell scripts of tens (or occasionally hundreds) of lines when it's a good fit...
barrkel 13y ago
If I actually need to do something with an input or output before sending it to the next command, I'll switch to Python. If it's not something that can be done trivially with sed or awk, I'll consider whether it's something that can be implemented in a generic way to be used elsewhere, in which case I'll write that specific utility, and insert it into the pipeline (with appropriate arguments). Only if it's fairly specific, or I need to package it for distribution, will I consider trying to rewrite everything in a single language script.

elwin 13y ago

It depends what it's doing. Other languages are much better at data processing. They make it more complicated to start child processes, use pipes, set environment variables, etc. If you're doing system administration tasks, you'll end up with a lot of shell command lines embedded in the program. (Try editing a crontab in Python...)

icebraining 13y ago
Try editing a crontab in Python... pip install python-crontab

laumars 13y ago

I don't agree with this. I love Perl, it's up there as one of my favourite languages (despite it's many short comings), but in my opinion shell scripts make more sense if you're writing scripts which depend upon a number of additional programs as the bulk of it's processing. For me, the point at which Perl makes more sense is if your script requires more internal logic than it does depends on spawning other programs. For example, if I'm writing a routine to auto snapshot ZFS / Btrfs volumes and delete any over a certain age, the script would be dependant on your file system CLI tools. So it makes more sense to have an 80+ line shell script than it does to write that in Perl / Python. However if I was writing a routine which requires users inputting details, where those details need to be sanity checked and then stored some where (such as a database), then the core logic of that program resides within your script (where you'd have to read inputs, do your sanity checks and then write to the database). So a Perl or Python script makes more sense. Obviously you can do either of those examples in each of those languages (crudely speaking as I know shell scripts aren't technically a programming language); but that's just a basic example of where I personally draw the line. I also think this is one of those occasions where it doesn't massively matter which approach you take just so long as the code works and is maintainable (though I draw the line at one maintenance script I saw last year. It was a Python script where every other line was os.system. It just struck me as rather pointless starting a Python interpreter if you're just going to use it like a shell script - you might as well do the whole lot in the shell to begin with).

carbocation 13y ago
Great comment. One nitpick: Bash, at least, is Turing-complete. https://en.wikibooks.org/wiki/Bash_Shell_Scripting https://en.wikibooks.org/wiki/Bash_Shell_Scripting
jsight 13y ago
*(though I draw the line at one maintenance script I saw last year. It was a Python script where every other line was os.system. It just struck me as rather pointless starting a Python interpreter if you're just going to use it like a shell script - you might as well do the whole lot in the shell to begin with).* FWIW, one of my favorite features of Perl syntax is that you can do things like this. A lot of my quick-and-dirty sysadmin scripts end up being Perl scripts with lots of backticks. It's handy for when they grow (as they often do) into more full-featured scripts.

emmelaich 13y ago

Yep pretty much, but it can be up to 100 lines depending on the nature of the script. To the article, I've never used a shell that doesn't understand [[ (guaranteed to be builtin, not sensitive to the hyphen issue) and {,} expansion. For the "for file in *; ... " that is also susceptible to the shell arg limits and is fairly hairy. Lastly if you're concerned about performance -- well, it's definitely time to leave shell for something better.

nandhp 13y ago
> I've never used a shell that doesn't understand [[ That may be, but that doesn't mean they aren't pretty widespread. [[ and {,} are bashisms. However, they still work when bash is executed as /bin/sh, as is the case on RedHat and some other Linux distributions. However, other distributions, including Debian, Ubuntu, and any embedded system using Busybox, point the /bin/sh symlink to a version of ash, the Almquist shell. It is used for scripts instead of bash because it is much smaller and faster (reportedly, Debian/Ubuntu boot speed improved significantly -- leaving the shell isn't a very good option for sysvinit).
- emmelaich 13y ago
  [[ is from ksh88 or earlier; {,} is apparently from csh. (Funny; I could've sworn I used {} in ksh under Solaris but apparently not!) Your point stands though, they are bashisms for all intents and purposes on Linux. As to speed -- well, again, I think if you want speed don't use shell. For startup you're better off redesigning the whole damn thing (cue systemd). Changing shell is like overclocking your cpu when you should be choosing a better algorithm.

jmount 13y ago

Yes. I just promised myself yesterday to stop writing bash scripts for things other than setting up an environment and launching another program. What pushed me over the edge is most of the script was incorporating the kind of fixes from the article (force variable to be treated as int, add dollars signs to access as variable, and so on).

nilved 13y ago

It depends on the goal of the script. I tend to rewrite complicated shell scripts in Ruby, but working with subprocesses in Ruby is terrible, so those scripts are destined to end in `.sh` forever. bash has a very specific purpose where it excels over scripting languages like Python and Ruby: wiring input and output and communicating with subprocesses.

barrkel 13y ago

It entirely depends on what the script is doing. If it's orchestrating a pipeline of processes, shell script is ideal. If it needs much in the way of complicated temporary data structures, shell is the worst way to go.

ams6110 13y ago

Depends on the target. Python, and even perl is not always available. /bin/sh is.

jes5199 13y ago
What systems don't have an interpreter installed these days?
- fragsworth 13y ago
  Well, cygwin won't have it by default

coolsunglasses 13y ago

I've gotten in the habit of using Python or Haskell for my automation. Pretty happy with it.