6 ms·

While this is interesting I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh. Run shellcheck on some

While this is interesting I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh. Run shellcheck on some of your own code, or the code of even a simple project to see how hard it is to really use bash. Why I think people gravitate towards it is because languages such as python add too much pomp to launching a shell process. A language like perl is usually easier to use but everyone hates it now.

masklinn 7y ago
> I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh I would heartily agree. If your shell script grows beyond half a dozen commands or so, you're probably better off rewriting it in just about anything. Python, ruby, go (gorun), rust (cargo-script), or whatever else, doesn't really matter as long as it's not shell.
- dvfjsdhgfv 7y ago
  Or D via rdmd: https://dlang.org/rdmd.html https://dlang.org/rdmd.html
danielecook 7y ago
I don’t think it’s pomp. Once I learned Unix pipes and the tools for manipulating data (sed awk cut etc), it just became much faster and easier than writing python scripts that do the same thing. You can literally connect the output of one process to the input of another with a single character. It’s much more complex in python. It also tends to be very portable.
- jigglesniggle 7y ago
  I think you're reading my comment the wrong way: I meant to say that doing e.g. piping in Python is a lot of pointless work (pomp), as you agree. This is perhaps one big benefit but not one that is exclusive to a sh-like language. Instead I would like to see a language with strong flow control or metaprogramming capabilities take on processes as a first class citizen. Perl is probably the closest but still has some warts related to redirection. The best pattern I have seen is encapsulating business logic into Python or Go and then if really necessary piping it to another script. But, often, if you do this you can just keep the piping internal using data structures. Unix-pattern facilities work very well for interactive use, which tends to be simple, exploratory, and trial-and-error. But a project's build script may not be simple.
  - danielecook 7y ago
    My mistake - I managed to completely misinterpret what you meant. I agree with your comment!
  - kamaal 7y ago
    Perl was literally born because Larry Wall reached the limits of what could be done with a combination of Shell + C + Unix utils. In fact this is how Perl 1 looks: https://st.aticpan.org/source/RCLAMP/perl-1.0_16/ https://st.aticpan.org/source/RCLAMP/perl-1.0_16/ https://github.com/Perl/perl5/commit/8d063cd8450e59ea1c611a2f4f5a21059a2804f1 https://github.com/Perl/perl5/commit/8d063cd8450e59ea1c611a2...
  - lostmsu 7y ago
    So PowerShell then?
    - iamnotacrook 7y ago
      Isn't that Windows only?
      - meloentje 7y ago
        No, but I haven't seen many Linux users (admitting to) using it.
      - cannam 7y ago
        Not any more: https://github.com/PowerShell/PowerShell/releases/ https://github.com/PowerShell/PowerShell/releases/ I quite like Powershell on Windows, but I'm not sure I dare try it on Linux. I think it might make my head explode.
        iamnotacrook 7y ago
        It's quite clear Microsoft are only a couple of years from either an official linux distro or some halfway house of using a linux kernel internally for major os functionality, so it makes little sense learning some new(ish) shell when you could just use bash (via wsl or whatever).
al_form2000 7y ago
Yes - and no. I find the shell is a useful glue language for procedural tasks (do this, run P1, then P2, and if blah) which can/had better be logically simple but rather long and heavy on the system interaction side. For these, being able to avoid the various $(echo... |sed) can be refreshing. Beside, the book makes for a nice repository of techniques.
dyanaraps 7y ago
The shell is my favorite language and I really don't know why. It's definitely possible to write shell code which properly passes shellcheck's linter though it's an uphill battle to learn the ins and outs and _why_ X is wrong when Y is right. I even managed to write a full TUI file manager in bash! https://github.com/dylanaraps/fff https://github.com/dylanaraps/fff I full understand that there are times when the shell should not be used and when other languages are a better way to solve a specific problem, however I love pushing the shell beyond its supposed limits! :)
- jgtrosh 7y ago
  Have you read Bash Pitfalls[1]? Do you write truly correct Bash/POSIX code? Do you still love it? [1]: https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls
  - dyanaraps 7y ago
    > Have you read Bash Pitfalls[1]? I've read pretty much everything I could get my hands on regarding the shell (including the mentioned link) and I still love it. > Do you write truly correct Bash/POSIX code? If we define correct as passing shellcheck, avoiding all pitfalls and maintaining compatibility (POSIX sh not bash), then yes, I like to think so. :) > Do you still love it? Oh yeah! I've been writing a ton of POSIX sh as of late. My latest project being a Linux distribution: https://getkiss.org/ https://getkiss.org/ (hello from Firefox in KISS!)
    - jgtrosh 7y ago
      Wow, very impressive. Do you also believe it's a viable language with which newcomers should start writing scripts? Edit: follow up question is Do you believe bash / POSIX shells actually follow KISS principles? Not questioning whether your OS is KISS, but I don't think that necessarily reflects the KISS-ness of the underlying language. My questions clearly reflect my current impression that in the long term, shell pitfalls largely undermine the benefits of its apparent simplicity. The gist would be for you to provide some way to change my mind. I guess the codebase you provide is a strong counter example; but you'll agree it doesn't reflect general usage of shell in the wild. Edit 2: You know what, I just read your original comment again. I kind of retract my question since you do concede that it's an uphill battle and you love it in spite of its flaws. I guess that's cool (and I agree it's fun trying to write correct bash as a challenge) as long as you're in control of the code being produced, but my main impression remains that it's a bad language to publicize and its presence in most codebases inherently bears a strong cost.
harry8 7y ago
Yeah perl was nice when we were allowed to use that. Why doesn't python have something like named pipes? f = open("ls|", "r) f.read() f.close()
- masklinn 7y ago
  I'm not sure what that's supposed to do, but subprocess provides these facilities, although definitely in a more verbose way. f = Popen('ls', stdout=PIPE).stdout f.read() f.close() alternatively given this exact behaviour: run('ls', stdout=PIPE).stdout
  - kamaal 7y ago
    Doesn't scale well when you have to stitch more than two commands with pipes, which is a very common use case for Unix CLI utilities.
    - masklinn 7y ago
      Don't see why it "doesn't scale well", the overhead is roughly constant: use the stdout of one program as the input of the next.
      - kamaal 7y ago
        That's not as easy as cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' You end up writing too much code, it's very verbose. Some times symbols are what you want. In fact the biggest progress in the growth of Math happened when they tossed out doing math with words and bought in symbols.
        Too 7y ago
        You forgot setting -e and -o pipefail. In case "something" is missing or can't be read for some other reason, your script will just continue happily without warning. But oh, if you do set -o pipefail the grep will stop the whole pipeline when none of the lines matches "this". So you have to keep fiddling with ${PIPESTATUS[0]}. And none of pipefail or PIPESTATUS are really portable. Not so simple after all.
kamaal 7y ago
Inability to use tools like Perl and Emacs is the biggest reason why I have to often break the bad news to people that their week to month long projects(Typically in Python and Java) can likely be done if a few minutes to a day or two, if they knew how to Emacs or Perl well. Programmers take great pride in freeing accountants and ware house workers from drudgery. But seldom do we look at our work in the same way. In the real world, most software work is done very similar to digging coal mines with shovels. Laborious manual hand typing jobs.
- unionpivo 7y ago
  ehh, I agree that its usually faster do do stuff in perl than in python, but if someone spent weeks doing something in python, that could be done in few minutes of perl, python is not the problem. And would probebly take them weeks to do it in perl as well. There are things you can do in line of perl that will take you 5 -10 min in python, but once program goes above one liners, difference is not that big. And of course, if you learned vim instead of emacs, you would be even faster :))
autoexec 7y ago
I started out with bash scripts and .bat files and eventually found perl to be the ideal solution to 99% of what I needed as well. I'm still using it for log file parsing and day to day administrative tasks, but I'm starting to pick up python because that seems to be where everyone has agreed to move to.
daitangio 7y ago
I am slowing moving from bash to python for my utility script. I was unable to love Perl, I think this weired syntax is a very bad choice. Anyway bash is still faster to use, and a lot of Unix services are based on it. The book is well written and have a great added value. Thank you for sharing!!
- kamaal 7y ago
  >>I think this weired syntax is a very bad choice. Ah, the luxury the modern breed of programmers enjoy today makes me feel jealous. In these days of Splunk and Document databases(returning jsons and xmls) its hard to understand why so many things in the past were the way they were. Apart from DBMS interaction(Perl had DBI/x modules for that), pretty much every thing in the decade of 80's even upto late 2000's(tons of legacy systems) was so non standardized that people were literally parsing through log files and non standard data formats to store/exchange a lot of things, this was when the internet was growing crazy year over year, systems needed to be built and put in place. This means you really need to have first class facilities to manipulate text. You needed regexes baked neatly into the language. You needed qw, you needed ``, you needed while<FILEHANDLE>, you needed binary file handling features, you needed string manipulation facilities that could help you drill through any text file you could imagine, you needed powerful functional programming features, you needed OO etc etc. And you needed to get this done under tough deadlines on slow machines. Remember Java being cross platform compliant was one of the biggest selling point of the day. Perl had this before Java. I personally worked on building a store using rcs and perl, that could store versioned config files, almost like a document data base. The parsing facilities required for the application we did demanded nothing short of a tool like Perl. Python, and also Java growing rapidly once the data exchange formats were reduced to mark up languages and JSON. Suddenly you could with a library what most Perl programmers were doing using their language powers. Also look at the Human Genome Project and Perl usage there.
  - collyw 7y ago
    Perl's Tie::DBFile is still the lowest effort data persistence I have seen. https://perldoc.perl.org/DB_File.html#A-Simple-Example https://perldoc.perl.org/DB_File.html#A-Simple-Example
grewil2 7y ago
Yes, Python is the Cobol of script languages.
- hosteur 7y ago
  What? Why?
davedx 7y ago
I use nodejs. Sometimes it's a case of "go with what you know" rather than "use the best tool for the job". I have too many other things to do for me to bother learning yet another language just for command line scripting.
sambe 7y ago
The overhead of repeatedly launching subshells/processes to do simple operations can add up quickly, especially if it is happening in loops or in parallel. Yes, you shouldn't be using bash for performance, we all know that. But scripts often grow over time and suddenly are found to be slow/resource hogs. I have seen people demand that proper logging be added to a bash program and it then get minutes behind because of the overhead of all the processes called to do the logging. I was able to make that 10000x faster using pure bash. As to why people like it: it often feels more natural when you are automating what you would type interactively. Unix pipes/coreutils/etc. also feel like a better fit when that automation is mostly about connecting other programs (it's the auxiliary stuff that you'd maybe want in pure bash). Reading the subprocess Python documentation does not exactly fill me with joy. I've heard libraries like Plumbum make it a bit neater - but then you have to ask why learn a bunch of new libraries when I already know bash? In the end it's about the best tool for the job. The danger with bash is going too far, especially if you don't actually know it very well.
- sumosudo 7y ago
  My favourite way to log.. Redirect stdout and stderr ( &> ) into a named pipe ( >() ) running "tee" And get the redirect into the log file as well. `exec &> >(tee ${__DIR}/${DOC_LOCAL}/${LOG_LOCAL})`
  - lordfoo 7y ago
    Would you mind expanding on this with an example? I'm trying to improve performance of my bash logging
    - sumosudo 7y ago
      So this line only redirects all script output to a file as well as ensure that that output makes it onto your screen at the same time. It is unstructured only in the way you allow any running command within your script to dump their output. I run most commands inside my scripts with `> /dev/null 2>&1` and then rely on exit codes to wrap structured information to be echoed out with this function: echo_l(){echo;echo '--------';echo "${1}";echo '--------';echo} Or functions such as this to populate a log: log_cat(){echo "${1}" >> ${__LOG} } But the tee named pipe is the winner. PS. The __DOC_LOCAL and __DIR variables start with these magic variables below. These variables are a life saver and allow easy directory and file manipulation, they kind of setup a top-level context: # Set magic variables for current file & dir __DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" __FILE="${__DIR}/$(basename "${BASH_SOURCE[0]}")" __SCRIPT="$(basename ${__FILE})" __BASE="$(basename ${__FILE} .sh)" __ROOT="$(cd "$(dirname "${__DIR}")" && pwd)"
Crinus 7y ago
One reason i prefer Bash (and i mean Bash not just any shell) is that it tends to stay stable - scripts written years ago work just fine today. These days i mainly use Bash on Windows (via MSYS2) and really it is available pretty much everywhere, either out of the box or via installation. If anything code that i used to write in Python at the past i write it in Bash nowadays, exactly because Bash has a better record when it comes to not breaking stuff. Though it helps that my Python use is also mostly scripts meant to run from the shell.
letusnot 7y ago
I recently started working a more devops role at a company (I've been a ruby dev for most of my career) and one of the things that I've noticed is how _insane_ scripting by ops people is. I'll start poking around a build pipeline and there will all sorts of tortured usages of sed and jq and bash and coreutils to get things done that would be _really_ simple in ruby. I routinely see people using wget and curl in the same script. I'll see people pipe sed text replacement into perl for more text replacement. I'm constantly aware of ruby three liners that are fully portable, even to Windows, that could replace a dozen lines of potentially subtly buggy shell scripting. And honestly I can't see any good argument for this patchwork approach to gluing things together. I guess some ops people might argue that you'd have to have ruby everywhere but the counter argument would be that we use docker images for everything and adding ruby as a dependency isn't any worse than all the insane dependency gymnastics it takes to get our node apps working. And all of this applies equally for any language with a reasonable standard library (python, perl). I think people have weird feelings about using bash or make or whatever to accomplish things, like they are riding closer to the metal or that they are living some deeply pragmatic zen Unix philosophy, but mostly they are making an un-testable mess until it works once and then, if they are lucky, they don't have to touch it again.

masklinn 7y ago

> I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh I would heartily agree. If your shell script grows beyond half a dozen commands or so, you're probably better off rewriting it in just about anything. Python, ruby, go (gorun), rust (cargo-script), or whatever else, doesn't really matter as long as it's not shell.

dvfjsdhgfv 7y ago
Or D via rdmd: https://dlang.org/rdmd.html https://dlang.org/rdmd.html

danielecook 7y ago

I don’t think it’s pomp. Once I learned Unix pipes and the tools for manipulating data (sed awk cut etc), it just became much faster and easier than writing python scripts that do the same thing. You can literally connect the output of one process to the input of another with a single character. It’s much more complex in python. It also tends to be very portable.

jigglesniggle 7y ago
I think you're reading my comment the wrong way: I meant to say that doing e.g. piping in Python is a lot of pointless work (pomp), as you agree. This is perhaps one big benefit but not one that is exclusive to a sh-like language. Instead I would like to see a language with strong flow control or metaprogramming capabilities take on processes as a first class citizen. Perl is probably the closest but still has some warts related to redirection. The best pattern I have seen is encapsulating business logic into Python or Go and then if really necessary piping it to another script. But, often, if you do this you can just keep the piping internal using data structures. Unix-pattern facilities work very well for interactive use, which tends to be simple, exploratory, and trial-and-error. But a project's build script may not be simple.
- danielecook 7y ago
  My mistake - I managed to completely misinterpret what you meant. I agree with your comment!
- kamaal 7y ago
  Perl was literally born because Larry Wall reached the limits of what could be done with a combination of Shell + C + Unix utils. In fact this is how Perl 1 looks: https://st.aticpan.org/source/RCLAMP/perl-1.0_16/ https://st.aticpan.org/source/RCLAMP/perl-1.0_16/ https://github.com/Perl/perl5/commit/8d063cd8450e59ea1c611a2f4f5a21059a2804f1 https://github.com/Perl/perl5/commit/8d063cd8450e59ea1c611a2...
- lostmsu 7y ago
  So PowerShell then?
  - iamnotacrook 7y ago
    Isn't that Windows only?
    - meloentje 7y ago
      No, but I haven't seen many Linux users (admitting to) using it.
    - cannam 7y ago
      Not any more: https://github.com/PowerShell/PowerShell/releases/ https://github.com/PowerShell/PowerShell/releases/ I quite like Powershell on Windows, but I'm not sure I dare try it on Linux. I think it might make my head explode.
      - iamnotacrook 7y ago
        It's quite clear Microsoft are only a couple of years from either an official linux distro or some halfway house of using a linux kernel internally for major os functionality, so it makes little sense learning some new(ish) shell when you could just use bash (via wsl or whatever).

al_form2000 7y ago

Yes - and no. I find the shell is a useful glue language for procedural tasks (do this, run P1, then P2, and if blah) which can/had better be logically simple but rather long and heavy on the system interaction side. For these, being able to avoid the various $(echo... |sed) can be refreshing. Beside, the book makes for a nice repository of techniques.

dyanaraps 7y ago

The shell is my favorite language and I really don't know why. It's definitely possible to write shell code which properly passes shellcheck's linter though it's an uphill battle to learn the ins and outs and _why_ X is wrong when Y is right. I even managed to write a full TUI file manager in bash! https://github.com/dylanaraps/fff https://github.com/dylanaraps/fff I full understand that there are times when the shell should not be used and when other languages are a better way to solve a specific problem, however I love pushing the shell beyond its supposed limits! :)

jgtrosh 7y ago
Have you read Bash Pitfalls[1]? Do you write truly correct Bash/POSIX code? Do you still love it? [1]: https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls
- dyanaraps 7y ago
  > Have you read Bash Pitfalls[1]? I've read pretty much everything I could get my hands on regarding the shell (including the mentioned link) and I still love it. > Do you write truly correct Bash/POSIX code? If we define correct as passing shellcheck, avoiding all pitfalls and maintaining compatibility (POSIX sh not bash), then yes, I like to think so. :) > Do you still love it? Oh yeah! I've been writing a ton of POSIX sh as of late. My latest project being a Linux distribution: https://getkiss.org/ https://getkiss.org/ (hello from Firefox in KISS!)
  - jgtrosh 7y ago
    Wow, very impressive. Do you also believe it's a viable language with which newcomers should start writing scripts? Edit: follow up question is Do you believe bash / POSIX shells actually follow KISS principles? Not questioning whether your OS is KISS, but I don't think that necessarily reflects the KISS-ness of the underlying language. My questions clearly reflect my current impression that in the long term, shell pitfalls largely undermine the benefits of its apparent simplicity. The gist would be for you to provide some way to change my mind. I guess the codebase you provide is a strong counter example; but you'll agree it doesn't reflect general usage of shell in the wild. Edit 2: You know what, I just read your original comment again. I kind of retract my question since you do concede that it's an uphill battle and you love it in spite of its flaws. I guess that's cool (and I agree it's fun trying to write correct bash as a challenge) as long as you're in control of the code being produced, but my main impression remains that it's a bad language to publicize and its presence in most codebases inherently bears a strong cost.

harry8 7y ago

Yeah perl was nice when we were allowed to use that. Why doesn't python have something like named pipes? f = open("ls|", "r) f.read() f.close()

masklinn 7y ago
I'm not sure what that's supposed to do, but subprocess provides these facilities, although definitely in a more verbose way. f = Popen('ls', stdout=PIPE).stdout f.read() f.close() alternatively given this exact behaviour: run('ls', stdout=PIPE).stdout
- kamaal 7y ago
  Doesn't scale well when you have to stitch more than two commands with pipes, which is a very common use case for Unix CLI utilities.
  - masklinn 7y ago
    Don't see why it "doesn't scale well", the overhead is roughly constant: use the stdout of one program as the input of the next.
    - kamaal 7y ago
      That's not as easy as cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' You end up writing too much code, it's very verbose. Some times symbols are what you want. In fact the biggest progress in the growth of Math happened when they tossed out doing math with words and bought in symbols.
      - Too 7y ago
        You forgot setting -e and -o pipefail. In case "something" is missing or can't be read for some other reason, your script will just continue happily without warning. But oh, if you do set -o pipefail the grep will stop the whole pipeline when none of the lines matches "this". So you have to keep fiddling with ${PIPESTATUS[0]}. And none of pipefail or PIPESTATUS are really portable. Not so simple after all.

kamaal 7y ago

Inability to use tools like Perl and Emacs is the biggest reason why I have to often break the bad news to people that their week to month long projects(Typically in Python and Java) can likely be done if a few minutes to a day or two, if they knew how to Emacs or Perl well. Programmers take great pride in freeing accountants and ware house workers from drudgery. But seldom do we look at our work in the same way. In the real world, most software work is done very similar to digging coal mines with shovels. Laborious manual hand typing jobs.

unionpivo 7y ago
ehh, I agree that its usually faster do do stuff in perl than in python, but if someone spent weeks doing something in python, that could be done in few minutes of perl, python is not the problem. And would probebly take them weeks to do it in perl as well. There are things you can do in line of perl that will take you 5 -10 min in python, but once program goes above one liners, difference is not that big. And of course, if you learned vim instead of emacs, you would be even faster :))

autoexec 7y ago

I started out with bash scripts and .bat files and eventually found perl to be the ideal solution to 99% of what I needed as well. I'm still using it for log file parsing and day to day administrative tasks, but I'm starting to pick up python because that seems to be where everyone has agreed to move to.

daitangio 7y ago

I am slowing moving from bash to python for my utility script. I was unable to love Perl, I think this weired syntax is a very bad choice. Anyway bash is still faster to use, and a lot of Unix services are based on it. The book is well written and have a great added value. Thank you for sharing!!

kamaal 7y ago
>>I think this weired syntax is a very bad choice. Ah, the luxury the modern breed of programmers enjoy today makes me feel jealous. In these days of Splunk and Document databases(returning jsons and xmls) its hard to understand why so many things in the past were the way they were. Apart from DBMS interaction(Perl had DBI/x modules for that), pretty much every thing in the decade of 80's even upto late 2000's(tons of legacy systems) was so non standardized that people were literally parsing through log files and non standard data formats to store/exchange a lot of things, this was when the internet was growing crazy year over year, systems needed to be built and put in place. This means you really need to have first class facilities to manipulate text. You needed regexes baked neatly into the language. You needed qw, you needed ``, you needed while<FILEHANDLE>, you needed binary file handling features, you needed string manipulation facilities that could help you drill through any text file you could imagine, you needed powerful functional programming features, you needed OO etc etc. And you needed to get this done under tough deadlines on slow machines. Remember Java being cross platform compliant was one of the biggest selling point of the day. Perl had this before Java. I personally worked on building a store using rcs and perl, that could store versioned config files, almost like a document data base. The parsing facilities required for the application we did demanded nothing short of a tool like Perl. Python, and also Java growing rapidly once the data exchange formats were reduced to mark up languages and JSON. Suddenly you could with a library what most Perl programmers were doing using their language powers. Also look at the Human Genome Project and Perl usage there.
- collyw 7y ago
  Perl's Tie::DBFile is still the lowest effort data persistence I have seen. https://perldoc.perl.org/DB_File.html#A-Simple-Example https://perldoc.perl.org/DB_File.html#A-Simple-Example

grewil2 7y ago

Yes, Python is the Cobol of script languages.

hosteur 7y ago
What? Why?

davedx 7y ago

I use nodejs. Sometimes it's a case of "go with what you know" rather than "use the best tool for the job". I have too many other things to do for me to bother learning yet another language just for command line scripting.

sambe 7y ago

The overhead of repeatedly launching subshells/processes to do simple operations can add up quickly, especially if it is happening in loops or in parallel. Yes, you shouldn't be using bash for performance, we all know that. But scripts often grow over time and suddenly are found to be slow/resource hogs. I have seen people demand that proper logging be added to a bash program and it then get minutes behind because of the overhead of all the processes called to do the logging. I was able to make that 10000x faster using pure bash. As to why people like it: it often feels more natural when you are automating what you would type interactively. Unix pipes/coreutils/etc. also feel like a better fit when that automation is mostly about connecting other programs (it's the auxiliary stuff that you'd maybe want in pure bash). Reading the subprocess Python documentation does not exactly fill me with joy. I've heard libraries like Plumbum make it a bit neater - but then you have to ask why learn a bunch of new libraries when I already know bash? In the end it's about the best tool for the job. The danger with bash is going too far, especially if you don't actually know it very well.

sumosudo 7y ago
My favourite way to log.. Redirect stdout and stderr ( &> ) into a named pipe ( >() ) running "tee" And get the redirect into the log file as well. `exec &> >(tee ${__DIR}/${DOC_LOCAL}/${LOG_LOCAL})`
- lordfoo 7y ago
  Would you mind expanding on this with an example? I'm trying to improve performance of my bash logging
  - sumosudo 7y ago
    So this line only redirects all script output to a file as well as ensure that that output makes it onto your screen at the same time. It is unstructured only in the way you allow any running command within your script to dump their output. I run most commands inside my scripts with `> /dev/null 2>&1` and then rely on exit codes to wrap structured information to be echoed out with this function: echo_l(){echo;echo '--------';echo "${1}";echo '--------';echo} Or functions such as this to populate a log: log_cat(){echo "${1}" >> ${__LOG} } But the tee named pipe is the winner. PS. The __DOC_LOCAL and __DIR variables start with these magic variables below. These variables are a life saver and allow easy directory and file manipulation, they kind of setup a top-level context: # Set magic variables for current file & dir __DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" __FILE="${__DIR}/$(basename "${BASH_SOURCE[0]}")" __SCRIPT="$(basename ${__FILE})" __BASE="$(basename ${__FILE} .sh)" __ROOT="$(cd "$(dirname "${__DIR}")" && pwd)"

Crinus 7y ago

One reason i prefer Bash (and i mean Bash not just any shell) is that it tends to stay stable - scripts written years ago work just fine today. These days i mainly use Bash on Windows (via MSYS2) and really it is available pretty much everywhere, either out of the box or via installation. If anything code that i used to write in Python at the past i write it in Bash nowadays, exactly because Bash has a better record when it comes to not breaking stuff. Though it helps that my Python use is also mostly scripts meant to run from the shell.

letusnot 7y ago

I recently started working a more devops role at a company (I've been a ruby dev for most of my career) and one of the things that I've noticed is how _insane_ scripting by ops people is. I'll start poking around a build pipeline and there will all sorts of tortured usages of sed and jq and bash and coreutils to get things done that would be _really_ simple in ruby. I routinely see people using wget and curl in the same script. I'll see people pipe sed text replacement into perl for more text replacement. I'm constantly aware of ruby three liners that are fully portable, even to Windows, that could replace a dozen lines of potentially subtly buggy shell scripting. And honestly I can't see any good argument for this patchwork approach to gluing things together. I guess some ops people might argue that you'd have to have ruby everywhere but the counter argument would be that we use docker images for everything and adding ruby as a dependency isn't any worse than all the insane dependency gymnastics it takes to get our node apps working. And all of this applies equally for any language with a reasonable standard library (python, perl). I think people have weird feelings about using bash or make or whatever to accomplish things, like they are riding closer to the metal or that they are living some deeply pragmatic zen Unix philosophy, but mostly they are making an un-testable mess until it works once and then, if they are lucky, they don't have to touch it again.