15 ms·
Bash Pitfalls
- mlacitation 13y agoI have a mirror of this site. wooledge.org runs off of greycat's (#bash on freenode) home DSL connection: http://bash.cumulonim.biz/BashPitfalls.html http://bash.cumulonim.biz/BashPitfalls.html
- wyclif 13y agoThanks. I don't understand why people submit links with content that can't be accessed by more than a few users simultaneously.
- joshbaptiste 13y agoWell it is the main reference but it won't sustain the "HN effect", thanks for the mirror link. I typically use http://wiki.bash-hackers.org/doku.php http://wiki.bash-hackers.org/doku.php which also provides a mirror and some extras.
- jamesbritt 13y agoHow would one know this in advance?
- wyclif 13y agoHow would one know in advance that submitting to HN would result in traffic consisting of more than a handful of users?
- jamesbritt 13y agoHow would one know in advance whether a site can handle high traffic? And did I really have to spell this out?
- Sivart13 13y agoSeriously? They can't find a better place to host a site like this here in 2013?
- clarry 13y agoIf you already have a machine serving stuff at home, adding a remote server is only time, money, and effort spent. Is it worth it? Not all sites are high in volume, and even well prepared sites on decent lines can go down if it hits the HN front page. People do occasionally bitch and moan about slow downloads from my home host, but nobody ever offered money for a pain-free alternative.
- jjwiseman 13y agoGoogle doc, gist, tumblr, any free blog engine.
- clarry 13y agoNone of which are pain-free to me.
- Dylan16807 13y agoDropbox? Unless you're pushing gigabytes per day it should make a perfect static host with urls no more ugly than previously mentioned services.
- joshbaptiste 13y agoAh yes, the ultimate reference from Freenode #bash, after you learn from Greycat's wiki you won't simply Google/DDG "bash tutorial" again and just head straight here.
- comex 13y agoThe Unix shell may be a highly powerful interactive programming environment, but it's sure hard to think of anything that comes anywhere close to sucking as badly. With the shell and the standard Unix commands, some things that are hard in other languages are easy, and most of the things that are easy in other languages are hard to impossible... I'd love to see a clean slate replacement for the shell that still feels Unix-like and retains most of its existing benefits. (I suspect PowerShell would be a good environment to take design cues from or even port, but I've never used it so I can't say for sure.)
- sigil 13y ago> I'd love to see a clean slate replacement for the shell that still feels Unix-like and retains most of its existing benefits. Have you looked at scsh, the Scheme Configurable Shell? It's a nice clean language with a REPL (well, it's just scheme) that has syntax for all the usual things you'd want in a unix shell -- pipelines, redirections, environment, signals, etc. http://www.scsh.net/docu/html/man-Z-H-1.html http://www.scsh.net/docu/html/man-Z-H-1.html
- thristian 13y agoHave you looked at "rc", the shell from Plan 9? It's very similar in spirit to the Bourne shell, but it's fundamentally better thought-out. http://plan9.bell-labs.com/sys/doc/rc.html http://plan9.bell-labs.com/sys/doc/rc.html
- mixmastamyk 13y agoHmm, sounds like you've never written batch files, the ultimate in "suck", but then you mention PowerShell, so I'm not so sure. Anyway, I enjoy using the fish shell, and you may too.
- jfb 13y agoI think the problem is that the terrible shell semantics are inextricable from the larger semantics of using Unix. So you could replace sh with something less dumb-stupid, but you'd still be interacting with the garbage that is the shell environment. You could then replace the latter, but at that point, well, you're off into the weeds. I hold my nose and write shell, even as I look over at, for instance, scsh, and think ... yeah.
- Sprint 13y agoApparently a mod changed the URL (to relieve a not so powerful host). The original URL (and thus the one you should be bookmarking/remember) was http://mywiki.wooledge.org/BashPitfalls http://mywiki.wooledge.org/BashPitfalls http://bash.cumulonim.biz/BashPitfalls.html http://bash.cumulonim.biz/BashPitfalls.html is a mirror, see mlacitation's comment.
- druiid 13y agoBash is for most things, one of the easier languages I have dealt with, being even beyond python, etc. That is though, for shell type scripting. There's many problems with it, but the only one I've run into that keeps it from being more useful is that there are no multi dimensional arrays built-in. There are super hacky ways I have seen them implemented, but by default it's something I basically never am able to turn to when scripting in bash and have to turn to other languages, even when the particular task I was working with would be mostly simpler in bash. That said, there are associative arrays in bash these days.
- brandonbloom 13y agoMaybe some controversial advice: Go ahead, fall in these pits. I write my fair share of shell scripts and I've hit practically every one of these snags in the past. However, for the majority of tasks I perform with bash, I genuinely don't care if I support spaces in filenames, or if I throw away a little efficiency with a few extra sub-shells, or if I can't test numbers vs strings or have a weird notion of booleans. Your scripts are going to have bugs. The important question is: What happens when they fail? Are your scripts idempotent? Are they audit-able? Interruptible? Do you have backups before performing destructive operations? How do you verify that they did the right job? For example, if your shell scripts operate only on files under version control, you can simply run a diff before committing. Rather than spent a bunch of time tracking down a word expansion bug, you can simply rename that one file that failed to not include a space in its name.
- druiid 13y agoI totally agree. Sometimes the strength of making a 'quick bash script', is that you are making a quick bash script. I have made some pretty strong, well tested projects in bash before including one that was a big part of an open-source qmail project. Sometimes though you just need to get stuff done with the least amount of fuss, without worrying about the extreme edge-cases which the majority of the webpage attached to this story talk about. Heck, it's probably the majority of bash work I'd say that ends up like that.
- clarry 13y agoOn the other hand, if you examine the given examples, you'll see that very often the "correct" way isn't really longer or harder to type. If you make a point of sticking to the correct way, it will eventually become automatic, and there'll be no fuss and worry. This might end up saving some sorry ass later on. And having learned the correct way, you'll instantly see it when a script you review is doing something in a way that will eventually bite someone. There's no downsides to learning and doing things right. Of course it does take some extra time and effort at the start, as everything.. Sure, there are extreme examples that can be really hard to handle portably and safely if you're doing something more complicated (embedded newlines in filenames come to mind). So in the end some corner cutting is often inevitable :-)
- ak217 13y agoI find it sad and amusing that we're writing a ton of mission-critical code in this language that has an incredible number of obscure quirks. Yes, most of these pitfalls are directly connected to the semantics of Unix, but I wish someone made a concerted effort to get rid of them in an otherwise evolutionary way.
- Aardwolf 13y agoThe following does not work on files with spaces according to the article: for i in $(ls *.mp3); do some command $i done So does that mean, that "for" will do something per word of the output of $, rather than per line of output of it? What to do if I want to do something for every line? What for example if I really want the output of ls, find (or any other command you can put int he $()) and loop through that line per line, even if some output has spaces? Thanks.
- LukeShu 13y agoThere are a couple of ways you can do it. Don't do this with `ls` though; there are better ways to get that info in scripts. # Loops over lines, in a subshell prints_lines | while read -r line; do some_command "$line" done # Loops over lines, in the current shell while read -r line; do some_command "$line" done < <(prints_lines) # If for some reason you really want a for loop oldIFS=$IFS IFS=$'\n' lines=($(prints_lines)) IFS=$oldIFS for line in "${lines[@]}"; do some_command "$line" done
- sigil 13y ago> So does that mean, that "for" will do something per word of the output of $, rather than per line of output of it? Correct. The argument to "for" is a list of words. > What to do if I want to do something for every line? Use a while loop. find /some/dir/ -type f | while read -r line; do ; # something with $line done PS. You should almost always use `find` instead of `ls` in shell scripts. Given a pattern, `ls` will exit non-zero if nothing matches it, and you should be treating non-zero exits like you would exceptions in other languages.
- joseph 13y agoOne thing to be careful of when doing "while read..." is that a new shell is started on each iteration, so you cannot for example set a variable within the loop that you can use later in the script, as its value will be lost when the shell process exits.
- 13y ago
- dj-wonk 13y agoWould someone recommend an automatic bash style checker, such as a 'linter'? Perhaps something along the lines of Chef's Food Critic? http://acrmp.github.io/foodcritic/ http://acrmp.github.io/foodcritic/
- zwp 13y agoShellcheck recently found a bug in one of my old scripts. You can use the web interface or compile it for local use: http://www.shellcheck.net/ http://www.shellcheck.net/ https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- secure 13y agoHere’s my personal favorite shell pitfall, which was the last drop to make me start recommending _against_ using shell except for very very narrow niche use cases: https://plus.google.com/+MichaelStapelberg/posts/YLarC7WPVQB https://plus.google.com/+MichaelStapelberg/posts/YLarC7WPVQB
- PhasmaFelis 13y agoSo why do we put up with classic command line tools in general that are so full of horrible, counterintuitive pitfalls? Is it just tradition? Backwards compatibility? The "Unix should be hard" crew has gotten a lot quieter in the last ten years with the rise of Ubuntu and other relatively user-friendly distros, but I feel like there's still an underlying current of elitism there; people are proud of mastering these bizarre, arcane methods, and they're offended that someone else might be able to accomplish just as much without doing half as much work.
- mixmastamyk 13y agoThere are alternatives. I use the fish shell for interactive work, and python when a bash script surpasses a certain complexity.
- Timmmmbob 13y ago1. Using bash.
- anon4 13y agoI know someone will sooner or later propose that we ban spaces and special characters in names. Let me just put my two cents forward. We should absolutely ban special characters from names. Specifically, all whitespace, the colon, semicolon, forward slash, backward slash, question mark, star, ampersand, and whatever else I'm missing that will confuse the shell. Also files cannot start with a dash. However, people should be able to name files with these characters. So I propose that these characters in filenames be percent-encoded like they would be in a URL. Specifically, the algorithm should be 1. Take the file name and encode it as UTF-8. Enforce some sort of normalization. 2. Substitute each problematic byte with equivalent percent-encoded form. This does not touch bytes over 0x80 - they are assumed non-problematic. 3. Write the file in the file system under that name. 4. When displaying files, run the algorithm in reverse. In the general case files like "01 - Don't Eat the Yellow Snow.mp3" would simply become 01%20-%20Don't%20Eat%20the%20Yellow%20Snow.mp3 in the filesystem and cause absolutely no further problems. To make it completely backwards-compatible we should also add the following rule: If a filename includes a problematic byte or a percent-encoded byte higher than 0x80, then it is assumed to be raw and will not undergo percent decoding. Basically, I propose that every program which receives free text input for a file name percent-encode the filenames before writing them to the filesystem and decode them for display. Everything else remains unchanged. Why this will not work: Requiring programmers to keep track of two filenames instead of just one is rather a lot of work. File APIs will have to take both encoded and non-encoded forms and encode the non-encoded form, creating problems when people inadvertently use the wrong function with a name, either double-encoding it or not encoding it and leading to "this file does not exist" errors. It will be possible to create two files with different names on disk which are nonetheless shown with the same name to the user. Why it is ugly: We're taping over a deficiency of an ancient language by inflicting pain on programmers. Double-encoded filenames? MADNESS. Why I like it: I'll be able to have ?, * and : in filenames in windows. My shell scripts will be much simpler. What do you guys think?
- derefr 13y ago> Substitute each problematic byte with equivalent percent-encoded form. This does not touch bytes over 0x80 - they are assumed non-problematic. You know what's crazy? Currently, in Unix, control characters are allowed in filenames. Like, \t and \n and \b and even \[. Those shouldn't be allowed, percent-escaped or not. Everything else you said is sensible.
- Nick_C 13y agofor arg instead of for arg in "$@" is gold. That is going straight to the pool room.