9 ms·
I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in al
by freedomben 9y ago
I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion.
1. This article contains excellent advice and should be starred for later retrieval.
2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors and needle-in-hay-stack searches using only shell commands.
3. Shell is the universal language.
4. Bash isn't that bad once you get used to it (seriously.
I'll grant you tho that arrays are still nasty ;-) ).
5. Bash is not that dangerous if you follow best practices. Don't be lazy!
6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing.
- digitalsushi 9y agoRe 6: I am an /extremely/ mediocre developer but I have crafted some real 99th percentile bash skills (within my company, not globally) and I completely self-sustain myself on just that. (You're wondering how many people write bash where I am, and the answer is somewhere between 8 and 15 thousand people). It's funny how many things that seem kind of incredible for a pure bash solution, looking back over the past 21 years doing this drivel. This year is the first I had to learn how to export all of the shell's variables and function definitions for a forked clone to run asynchronous background processing of the main thread. Bash! Bash of all things. Probably 10 seconds in any other language, but here is bash kind of hopping alongside. Callbacks, reflective programming, there's always a really awkward way in bash to do what's happening in the popular languages.
- koolba 9y ago> It's funny how many things that seem kind of incredible for a pure bash solution, ... My go to example of “Bash can do whaaat?!” is the source for xip.io. A custom DNS server written in a handful of lines of Bash! https://github.com/basecamp/xip-pdns/blob/master/bin/xip-pdns https://github.com/basecamp/xip-pdns/blob/master/bin/xip-pdn...
- ddtaylor 9y agoI have made a lot of DNS products and doing the same thing happening here is a few lines of code in most any language. This code, while entertaining, cannot handle very simple DNS packets because of its "compression encoding.
- kiallmacinnes 9y agoThis is ran behind PowerDNS, which handles the compression prior to handing off to this code, so that shouldn't be a problem.
- brazzledazzle 9y agoWhich means it’s not really doing any of the heavy lifting.
- xyproto 9y agoWhich is also the case for most bash scripts, the heaving lifting is done by all the executables that are called from bash, not in native bash.
- opinionator1 9y agoNobody said it couldn't be done, just that it shouldn't. Fun stunt but ultimately suitable only for toys or as a gimmick.
- digitalsushi 9y agoI've developed on consumer grade routers from big vendors - Asus, linksys, netgear etc, and they are all using identical quality scripts for a large majority of the router's data processing. These scripts may not be pristine, but they are sure not always just a toy.
- saltcured 9y agoOver 20 years ago, I was home on a sick day in the middle of refactoring a build system based on Bourne shell. We could not use any of the new features of BASH but had to stick to what worked as /bin/sh across a bunch of supercomputer vendors. I was finding a lot of arcane tricks to optimize execution on those systems, some of which had improbably expensive fork-exec. In my fevered state, I wrote some arithmetic functions to do the equivalent of `expr $a + $b` and `expr $a \* $b` without any fork-exec to external commands, just to prove to myself that it could be done. I think I abused things like $IFS, for loops, recursive function calls, and case statements to make an arbitrary-precision decimal adder function.
- themodelplumber 9y ago> 6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing. Stories! C'mon, it was a long Monday :-)
- freedomben 9y agoHaha. Mostly it is dozens, probably hundreds of little things over time when turning to the shell proved highly productive and effective. The shell is surprisingly really, really good at processing text (using the good 'ol unix tools), and it's amazing how often "text processing" type problems come up. Whether you are doing find/replace on source code, looking for where certain strings are used/defined, or curling some page and extracting data out of it, the shell can often provide very quickly and effectively. The bonus feature is that people are often very impressed when you whip out the shell skills and nail something they thought would be difficult. I also started capturing most tasks as bash functions or aliases in my `~/.bashrc` file. This doubles as both a handy reference for me to look at later to remember how to do something, and automating simple things.
- themodelplumber 9y agoThat's really cool, thanks. It seems that whenever I come across an annoying text-processing problem, it's always really hard or too generic-sounding to get search engine help with potential bash solutions. However I'd somehow like to improve in that area.
- wool_gather 9y agoCheck out the sed and awk tutorials here: http://www.grymoire.com/Unix/Sed.html http://www.grymoire.com/Unix/Sed.html sed in particular is really amazing in what it can do, very simply. (Note, depending on your system, you might want to get GNU sed, for "extended" regex support.) Those pages do a great job breaking down the basic behavior and explaining pitfalls. They're skim-able, and pretty easily searchable after than, so you can come back and remind yourself of the thing you learned last week.
- antris 9y agoWhat is your opinion on using other languages to augment bash? Awk, perl etc.
- freedomben 9y agoI personally consider awk, sed, grep, cut, etc. to be "part" of bash. I don't think I ever write a script without invoking those venerable tools at least once. I often have many pipes, such as some_command \ | grep -E 'some.*regex$' \ | sed -e 's/erase_text//g' \ | sed -e 's/erase_more//g' \ | awk '{ print $2 }' \ | cut -d ':' -f 1 I used to reach for Perl one-liners a lot, and still do sometimes when I need a gross regex that `sed -E` can't handle (See Perl Pie[1]), but Ruby has been making an appearance more and more often for handy one-liners. Roblog has a great post on using Ruby [2]. If the script is complex enough and I'm willing to take a dependency on Ruby, that is usually where I turn. Where that line is does change from time to time. Ruby makes it super trivial to call shell commands and process them easily as well, which makes it easy to integrate into existing scripting [3]. Hopefully that answered your question :-) [1] http://technosophos.com/2009/05/21/perl-pie-if-you-only-learn-how-do-one-thing-perl-it.html http://technosophos.com/2009/05/21/perl-pie-if-you-only-lear... [2] https://robm.me.uk/ruby/2013/11/20/ruby-enp.html https://robm.me.uk/ruby/2013/11/20/ruby-enp.html [3] https://stackoverflow.com/a/2400/2062384 https://stackoverflow.com/a/2400/2062384
- rdtsc 9y ago> sed -e 's/erase_text//g' \ I like the pattern. Copied it to my notes to try next time I am grepping through logs. I usually just string a few 'grep -v ignore_this' with pipes but then it ignores the whole line. But I can see how erasing some parts of the line would be very helpful sometimes.
- laumars 9y agoYou might already be aware of this, but you can chain multiple sed functions together into one command. eg sed -e 's/erase_text//g; s/erase_more//g' But I appreciate your example there is more for illustrative purposes rather than an actual pipeline you have in production.
- freedomben 9y agoCan't edit original, but to answer the question regarding good sources for learning, I highly recommend The Linux Command Line . Great book: http://linuxcommand.org/tlcl.php http://linuxcommand.org/tlcl.php
- zwischenzug 9y agoI did a talk on this a while ago: https://www.youtube.com/watch?v=pb3k0sGKrjQ&t=457s https://www.youtube.com/watch?v=pb3k0sGKrjQ&t=457s 'Take bash seriously'
- h1d 9y agoIt's time we need a better shell than bash instead of thinking it's great after bleeding with it for years and gets used to it.
- brazzledazzle 9y agoI’d vote for PowerShell in a heartbeat. It passes real objects instead of strings. It’s cross platform and open source. It’s imperative but borrows some functional concepts.
- agumonkey 9y agoPowerShell is good. I don't like some of the hardened conventions .. but it's highly convenient without being a mess. Nice output, regular sort/filter functions. I like sed but it goes old quickly.
- zwischenzug 9y agoAs I say in the talk (and get a big laugh) 'Powershell is boring. It's too well-designed.'
- freedomben 9y agoI really wonder if PowerShell wouldn't have been much bigger had it been cross-platform from the start. Definitely better late than never (:clap: Microsoft) but at least in my sphere we never even thought seriously about PowerShell because we had too many linux machines in our environment.
- JdeBP 9y agoWe have just had the 30th anniversary of someone acting upon that very thought. > Perl is a interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information. It's also a good language for many system management tasks. The language is intended to be practical (easy to use, efficient, complete) rather than beautiful (tiny, elegant, minimal). It combines (in the author's opinion, anyway) some of the best features of C, sed, awk, and sh, so people familiar with those languages should have little difficulty with it. (Language historians will also note some vestiges of csh, Pascal, and even BASIC-PLUS.) Expression syntax corresponds quite closely to C expression syntax. If you have a problem that would ordinarily use sed or awk or sh, but it exceeds their capabilities or must run a little faster, and you don't want to write the silly thing in C, then perl may be for you. There are also translators to turn your sed and awk scripts into perl scripts. -- Larry Wall, perl 1.0 release announcement, comp.sources.unix v13i001, 1988-01-02.
- deleted 9y ago[deleted]
- Groxx 9y agoRe 4: bash is an acceptably-good domain-specific language. Which leads right into 5: that domain is unsafe-by-design and failure-tolerant. Which is phenomenally useful and efficient for one-off tasks, and phenomenally dangerous for something you allow others to inject behavior into.[1] There are a fair number of things which bash could do better, and probably should. But many of the gotchas like "if you don't quote it, it'll expand into multiple arguments" are features when you want to throw things around efficiently. It wouldn't be as efficient for plugging so many disparate tools together if it weren't such an efficient footgun. [1]: If you need anything even remotely fault-resistant, `set -euo pipefail` and use `shellcheck` (and the rest of this resource, it is indeed great) and then consider using a "normal" programming language instead. But for behavior you fully control it's not too bad if you're careful.
- emmelaich 9y agoI probably have a similar history but I've come to the belief that you should not use Bash if you're doing anything fancy or long (>10 lines). Just use Perl, Python or Ruby. One or all of them is installed on every machine you are likely to use.
- davedx 9y agoYes, but which one, and which version? This is especially painful with Python ;)
- h1d 9y agoWhat's painful about Python but not so for others? Writing anything complicated in bash/zsh is painful enough I wouldn't even think about it.
- CJefferson 9y agoI currently find some users have python2, some have python3, some both. It's fine in a controlled environment, or if your users are Devs, but a pain with normal users who I just want to send a script to.
- Groxx 9y ago2 vs 3 is not a small change. and tools for helping you do things (like a 3+2 compatibility lib) get installed with pip... which is horrifyingly fragile. `pip install your-lib` -> you have no idea if you have the dependencies you require, because pip doesn't behave rationally. it often Just Works™, but when it doesn't there's no help and a massive minefield of potential problems.
- vinceguidry 9y agoUse Ruby. It doesn't change much from version to version, thank God. I curse python every time I have to install a package. Nothing Just Works. If you install just one gem, bundler, you can get dependency management in a one-file script. Checkout Bundler Inline: https://github.com/bundler/bundler/blob/master/lib/bundler/inline.rb https://github.com/bundler/bundler/blob/master/lib/bundler/i...
- h1d 9y agoI think it's time people start using something better than bash/zsh that is decades old, like fish or even come up with a more modern shell. Even by looking at these examples, you see it has less verbosity like "then" and "do", you can reference arguments as $argv instead of cryptic $@ and exit status code as $status instead of $? which is confusing with $! and the likes. https://blog.codeship.com/lets-talk-about-shell-scripting/ https://blog.codeship.com/lets-talk-about-shell-scripting/ https://fishshell.com/docs/current/tutorial.html https://fishshell.com/docs/current/tutorial.html Shell is such an integral part of admins and programmers workflow yet I find it hard to believe this field has been so slow at improving. Even the fish site jokingly states "Finally, a command line shell for the 90s" implying the others are even older.
- zingmars 9y agoI think a big issue is that bash is available everywhere, while fish might not be. There's also the fact that a lot of us have fancy dotfiles for our work/home computers, and switching to another shell would mean having to rewrite them in the target shell language.
- yani 9y agoAvailability is a big issue. It is nearly impossible to get a seasoned sysadmin to install fish for you when bash is available.
- raboukhalil 9y ago>> I'll grant you tho that arrays are still nasty ;-) If you're interested, here's an article I wrote recently that attempts to explain the method in the madness: https://medium.com/p/the-weird-wondrous-world-of-bash-arrays-a86e5adf2c69 https://medium.com/p/the-weird-wondrous-world-of-bash-arrays...