44 ms·
Shell Style Guide
- empath75 8y ago> Shell should only be used for small utilities or simple wrapper scripts. Oh you mean we weren’t supposed to write an etl solution in bash?
- jrs95 8y agoI’ve actually seen this happen. It’s terrifying.
- enriquto 8y agoSure, but not as terrifying as people rewriting shell one-liners in python or (gasp) java.
- pjmlp 8y agoI have been there, porting ksh scripts to Java. As much as I might like Java, this surely did not make any sense, but when people want to pay you for doing it, oh well.
- hrktb 8y agoThen a lot of people don’t understand the limits of what they’re doing. For instance piping though commands works fine until one character sequence is interpreted as EOF. The fun part is it will work most of the time, and when it fails nobody will understand why (“we didn’t touch anything”), and rewriting the thing will be a political nightmare (“it was working before and was only 3 lines, why you need so much time to redo it ?”) I think most people (me included) don’t do enough shell programming to really know the trade-off of the one-liner vs doing it in groovy, so the latter becomes the safer option (rightly so IMO)
- icebraining 8y agoone character sequence is interpreted as EOF I know this is just an example, but how would that happen?
- hrktb 8y agoYou can hit EOF or EOT in a stream of UTF-8 or other multi-byte encoding as part of a character. There must be ways to have it handled through something that understands character encodings and workaround the issue, but I know that dumb shell piping will fumble on those.
- JdeBP 8y ago% printf '\x04\necho EOT is not EOF in a pipe.\n' | bash bash: line 1: $'\004': command not found EOT is not EOF in a pipe. % Pipes do not contain line disciplines and thus do not do special character processing.
- vorg 8y agoIf you can do it as a one-liner in Bash, then you're not using the JVM. The benefit of using Apache Groovy for scripting is access to the JVM (and of course defining Gradle builds). If you don't need the JVM, then do it in Python or Ruby.
- jrs95 8y agoWell, rewriting one liners is kind of stupid, but it's probably not going to result in a nearly incomprehensible mess running in production.
- tigershark 8y agoI hope there was a reason for rewriting them.. Anyway Java is for sure NOT a scripting language. F# is much much better from that point of view, with the right tools, but sadly it is not famous enough.
- reificator 8y agoYou don't know true terror. I've seen an "ETL" solution (minus the "T") written entirely in SQL but using XML instead of tables and columns. After traveling through multiple servers via OpenQuery it eventually gets loaded into a domain-specific piece. That piece passes the data into a frontend that can only run in quirks mode. That frontend has 60-80 separate projects that all parse out basically the same data but from different fields and with different parsing/mapping logic. There is no ability to load shared libraries or otherwise de-duplicate code and no authorization to call out to any internal web services that could be written to handle it. Nor is there any form of templating available, it's all static files with no nesting of folders allowed. And the whole thing is managed in a domain-specific editor that effectively precludes the use of version control thanks to the way it stores files. Two people are responsible for managing that frontend piece and more.
- nurettin 8y agoSounds very much like IBM AS400
- deleted 8y ago[deleted]
- tigershark 8y agoI’m not sure when you worked on AS/400, but from what I remember there was no xml involved at all, although you had to deal with DB2 and the almost comical table and column names...
- nurettin 8y agoI meant the difficulty in storing source files in version control, the data input separated to 60-70 projects with very little code shared between them and the number of people responsible for understanding the system being very low, sounds very much like an AS400+rpg environment created for parsing XML files.
- tigershark 8y ago
- empath75 8y agoAs have I. It actually worked for a one time migration, but it required a lot of handholding.
- bryanlarsen 8y ago90% of the time ETL is trivial and not system-critical and can be done with a 3 line bash script. And in the other 10% a simple typo can cost your company millions of dollars.
- tigershark 8y agoIf it wasn’t a toy project in which the transformation was elementary then no, you probably shouldn’t have. If it went well for your company in case it wasn’t a toy project then I would say that it was despite bash, not thanks to bash.
- Sir_Cmpwn 8y ago>Bash is the only shell scripting language permitted for executables. Whelp, wrong right off the bat. I'm gonna get down my my knees here and beg everyone reading: use POSIX shell. Do not write scripts with bash. Do not write scripts with zsh. Do not write scripts with fish. Use POSIX shell. sh has a really bad interactive mode (so does bash), so I'm not gonna give anyone a hard time for using another shell as their day-to-day interactive shell. But, for the love of god, write your scripts with POSIX sh.
- jon-wood 8y agoI'm genuinely curious, why? If you're writing generic scripts designed to run across multiple platforms then fine, but in the context of Google (or most other companies) they'll have a standard set of tooling available on all their machines, which presumably includes Bash if they're making that statement. What about POSIX shell makes it superior for scripting?
- JepZ 8y agoMaybe one day there will be another cool and dominating shell. That shell is more likely to be POSIX compliant than bash compliant.
- deleted 8y ago[deleted]
- oblio 8y agoAll the cool new shells I've seen are either completely POSIX incompatible (powershell, xonsh, etc.) or they try to be bash supersets (zsh, osh, etc.). Bash is a de facto modern standard. The only new purely POSIX shells I know are for embedded stuff (dash) and that space is neither fast moving nor in dire need of a better shell, since most modern shell features target interactive use. Or they make coding easier and/or safer, in which case we're back to point 1, with the POSIX incompatibility.
- JepZ 8y ago
- geofft 8y agoThe only thing I'd quibble with is the recommendation of [[ ... ]] over [ ... ], because shell programmers used to [ will be surprised that [[ "foo" == "f*" ]] does pattern matching. But that is more or less an arbitrary style preference. One thing I'm curious about is whether Google machines have followed Debian etc. in making /bin/sh a faster non-bash shell, or if /bin/sh is bash.
- JepZ 8y agoSame here. Dropping non-bash support just for [[ ... ]] over [ ... ] seems a little odd.
- dualbus 8y ago> will be surprised that [[ "foo" == "f*" ]] does pattern matching It will not do pattern matching, because you quoted the right-hand side.
- your-nanny 8y agoWhich is totally fucking indane
- geofft 8y ago... ... okay, now I extremely object to [[ "foo" == f* ]] not doing globbing and doing pattern matching. But, I guess Google folks have experience that this is non-confusing?
- mrdmnd 8y agoI'm on a google machine now and it's a symlink to /bin/bash.
- kevincox 8y ago> One thing I'm curious about is whether Google machines have followed Debian etc. in making /bin/sh a faster non-bash shell, or if /bin/sh is bash. Since the guide says "Executables must start with #!/bin/bash" it doesn't really matter.
- koolba 8y agoI've never seen this guide before today but skimming through I absolutely love it. It aligns with my exact bash scripting style. Everything from the variable and function naming rules, stderr logging, pipeline indentation, and even having a "main () { ...}" wrapper and invocation. It's almost creepy!
- enriquto 8y agoIt's not creepy if you wrote it while working at google (as a mathematician, i couldn't help but notice that you are not excluding this possibility).
- deleted 8y ago[deleted]
- nileshtrivedi 8y agoS/he did exclude this possibility by saying "I've never seen this guide before today" :-)
- JepZ 8y agoMaybe koolba was blind and has been cured.
- raverbashing 8y ago"One example of this is Solaris SVR4 packages which require plain Bourne shell for any scripts." Such as?
- tootie 8y agoI absolutely hate coding in bash or looking at bash or suddenly being in Windows and being up a tree because bash isn't supported well (it is now if you install WSL). I only use bash to set environment variables and maybe string together 2 or 3 build commands. If I need so much as an if statement, I'm going to switch to a real programming language.
- cellard00r 8y agoOften times it's easier to use binaries and manage their input/output through shell code. Having a good understanding of shell code will also allow you to easily write slick one-liners that will save you lots of time.
- SOLAR_FIELDS 8y agoFor basic file system operations it’s a necessity. Write 10-20 line python script or type a one line grep/awk/sed piipeline to spit out the contents of some files?
- chrisfinazzo 8y agoThere's a great talk by bwk where he complains about exactly this kind of behavior...before admitting that anything useful in a language will be abused in order to do silly things. https://www.youtube.com/watch?v=Sg4U4r_AgJU https://www.youtube.com/watch?v=Sg4U4r_AgJU I'll admit, I'm no fan of Python, but this has little to do with the language itself. I found Bash and Ruby first, so Python just seems like too much work to get right. Requisite xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/
- jerf 8y agoIt's a 10-20 line Python script if you write it without any support code. If you're writing any amount of Python code that does shell things, you should either write or get an existing library. Also, it really isn't 10-20 lines to spit out the contents of some files in Python: >>> filelist = ["tmp.pl", "tmp.go", "Tmp.hs"] >>> for f in filelist: ... print(open(f).read()) It's trivial to code golf that down to 1 line, but if we're writing in Python at any sort of scale, this is more realistic. Easy things are still generally easy. They just aren't optimized down to shell level in terms of keystroke count. They're often more optimized than bash in terms of conceptual complexity, though. For instance, if one of my files has a space in it, shell becomes a conceptual minefield, requiring you to understand the half-a-dozen ways it might process strings and how that interacts with filenames and arguments passed to commands, whereas the Python just keeps doing what you probably meant, because it knows what's a string and what's something else. Shell makes a lot of sacrifices for that concision, and there's a lot of "Well, this will probably be OK on anything I run it on...". I never put spaces in my file name, but use periods or underscores instead, precisely so I don't screw myself over with shell. I shouldn't have to do that. Personally I think the crossover point is in the high single digit number of bash lines. The only advantage bash still holds over Python at that point is large pipelines, and 80% of those can be replaced by Python functions. For the remainder, use a pipelining library.
- dkns 8y ago> If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python. This is exactly why I wrote my last script in Python. I started googling for how to do some simple stuff with arrays in Bash and after 10 minutes decided "I'll write it in Python".
- benmmurphy 8y agoi'm surprised it doesn't recommend using set -e /other flags. [http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/] maybe it is not covered because it is not considered 'style' but i think those flags are some of the most important things you can set when writing a bash script. i guess if you are writing scripts that start other things, and then check their error codes it can be annoying because you have turn them off/on.
- falsedan 8y ago`set -e` is basically a way to find where you need better error-handling, not a way to make scripts safe. I'd much rather a crummy script run into trouble calling some new featuref, shrug, and move on to the core business-critical code, instead of erroring out and triggering a bunch of pages.
- ianbicking 8y agoDo you normally protect every command with || exit? cd foobar || exit It's those really small failures that cause shell scripts to do crazy things without set -e.
- falsedan 8y agoWhy wouldn't I rewrite the script to avoid the cd instead? The point of `set -e` isn't to insert all those pointless guards, it's to point out "hey this might fail, so think of a way to do this that doesn't rely on cd". For your extremely thin example, if it (say) removed some files, I would only remove the files from the current directory or an absolute path.
- quotemstr 8y agoPlease don't blindly follow style guides. Use what works right for you. I generally strongly dislike the decions codified in Google "style guides"
- dvfjsdhgfv 8y ago> Indent 2 spaces It's interesting how quickly the 2-space indent became the new norm
- ma2rten 8y agoAll Google style guides use two spaces.
- fernandotakai 8y agonot the python one! https://google.github.io/styleguide/pyguide.html https://google.github.io/styleguide/pyguide.html anything other than 4 spaces for python look out of place for me.
- bru 8y agoThis is either outdated or only for the public version of the guide: the internal guide, and all our python code, has 2-spaces indents.
- teddyh 8y agoMeanwhile, the rest of the Python world probably follows PEP8, which uses 4 spaces. https://www.python.org/dev/peps/pep-0008/#indentation https://www.python.org/dev/peps/pep-0008/#indentation
- GavinAnderegg 8y agoI've seen a lot of Python from Google using two spaces, which I find very hard to read. Many of the samples here use two-space indentation: https://github.com/google/google-api-python-client/blob/master/samples/README.md https://github.com/google/google-api-python-client/blob/mast... Here's a specific example: https://github.com/google/google-api-python-client/blob/master/samples/coordinate/coordinate.py https://github.com/google/google-api-python-client/blob/mast...
- andrewmcwatters 8y ago
- _arvin 8y agoI'm really surprised they went with: #!/bin/bash as opposed to: #!/usr/bin/env bash the latter feels more flexible and dependable for a script to be passed around.
- moviuro 8y agoIt is, but don't forget it's a guide for Google engineers on the Google machines. People who share code for the world to use should use: - `#!/usr/bin/env bash` - or `#!/bin/sh` for POSIX shell scripting
- peterwwillis 8y ago/bin/sh is most often either the C shell, Dash, or Ash, not the Bourne shell. Bourne shell extensions will cause the script to fail. edit: bourne again shell
- moviuro 8y agoYou certainly meant /bin/sh is not the Bourne again shell. I updated my comment to clear any possible misunderstanding https://en.wikipedia.org/wiki/Bourne_shell https://en.wikipedia.org/wiki/Bourne_shell https://en.wikipedia.org/wiki/Bourne-again_shell https://en.wikipedia.org/wiki/Bourne-again_shell
- floatboth 8y ago/bin/sh shouldn't be a C shell, it's always a POSIX-compatible shell IIRC
- peterwwillis 8y agoI suppose you're right, though the main point is it may not have Bourne-again extensions
- scruple 8y agoIn the wild, hopefully not, but I do remember coming across /bin/sh executing csh in an obscure RTOS-ish / proprietary flavor of Linux that we were prototyping at a previous employer. Would've been around 2008. I can't remember the variant, but I remember being caught off-guard by it.
- LinuxBender 8y agoI write long bash and perl scripts. I keep things neat and tidy most of the time. My "temporary workarounds" are still in use today, in production, so I must have written them well enough for people to understand. I see a lot of debate around preferences at the risk of becoming a funny title on n-gate. Just use whatever languages you know best. If it follows the best practices of that language, someone can port it to their preferred language. Maybe it is just my browser, but that xml file should be renamed to .txt or have the mime type set differently. There is no formatting for me. It is just one really long line.
- occams_chainsaw 8y agoMy "temporary workarounds" are still in use today, in production, so I must have written them well enough for people to understand. Maybe the reason the temporary workarounds persist is because nobody understands them well enough to replace them ;)
- LinuxBender 8y agoI completely understand. In my case, every one of my scripts has my name and email in it. They won't hesitate to reach out to me if something is broken. :-)
- BlackLotus89 8y agoNothing against you or your long perl scripts (wrote perl for a few years as well), but at my current job there are perl scripts as well, for legacy reasons. The scripts where written 20+ years ago to compliment our mail systems and identy management. Today nobody fixes those scripts or replaces them and finding a replacement solution for the underlying systems is nearly impossible, because nobody understands all the perl scripts or dares to touch them. Our dns management can't handle wildcards and sub-sub domains (easily) and every time something like this is needed there is one person who needs to manually add this. Anyway the point I was trying to make is that temporary soluyions often stay because of other reasons than perfection. On the flip side I solve every problem at first through bash or python scripts as well and aolutions tend to be permanent, so we are in the same boat.
- deeg 8y agoThis quote on whether to always use braces with variable names (e.g. "${var}" or "$var"): > These are meant to be guidelines, as the topic seems too controversial for a mandatory regulation. It's nice to see that even Google has bike-shedding arguments.
- gre 8y ago> If performance matters, use something other than shell. I wrote a script that wrapped compiler calls to make dummy output files to debug a huge compilation. Since startup time was the most important metric, bash actually had by far the best performance.
- nameless912 8y agoPeople rag on bash a lot, but it now forms the core of a major deployment system I built at work. I'm not happy that it's in bash, but every other solution was worse, and it calls out to other tools (e.g. Ansible and Terraform) when they're appropriate. People have forgotten the subtle art of realizing what tool is best for the job. If all you're doing is stitching together other tools and doing some logging, bash is hard to beat.
- adrianhel 8y ago^ this! Bash is the best stitching together language you'll get. Sure it sucks at scripting.. So stitch a script in there for God's sake!
- h1d 8y agoFish shell?
- nameless912 8y agoNot portable enough. The point of bash (well, really sh if we want to be pedantic) is that it's a stitching tool available on just about every conceivable platform.
- no_wizard 8y agoI don’t understand all the hate for bash or python in this thread. These are programming languages. Like all programming languages one must understand how to deploy them and use them properly. Yes bash has faults absolutely. It can be very arcane and esoteric and I think this due to the fact that it’s creators are very much of the old 1970s and 1980s Unix mindset and largely bash tools still feel like they work the way they did back then Python is also a tool. I have written scripts in python to manage processes do forking etc and it too has some warts. It is also used by companies tohat deploy thousands of lines of code or more. Yes it isn’t statically typed and yes you can get into a box but guys no single language is objectively superior to another. So learn them like you would any other tool and move on.
- pstuart 8y agoI like bash, but I think I understand the hate -- it's the slippery slope of "gluing real applications" to "I can write this all in bash!". Just because you can, doesn't mean you should.
- s3m4j 8y agoIf the application can be reduced to some form of simple piping of existing programs...
- amptorn 8y agoI enjoy the section of the Advanced Bash-Scripting Guide which explains how Bash has sparse arrays which "are useful in spreadsheet-processing software".[0] Because someone out there is apparently writing spreadsheet-processing software in Bash? [0] http://tldp.org/LDP/abs/html/arrays.html http://tldp.org/LDP/abs/html/arrays.html
- kazinator 8y ago> due to the fact that it’s [sic] creators are very much of the old 1970s and 1980s Unix mindset Firstly, Bash is relied upon to conform to a standard; it's not an exploratory project which expresses the maintainers' views about what a shell should look like. Secondly, Stallman didn't start Bash because he loved Unix. It was a pragmatic choice. Stallman really wanted a free platform based on Lisp. He went with cloning the Unix user space because that was becoming a dominant OS. It was proprietary software, and he wanted to replace that directly with workalike free software, rather than to provide incompatible free software and then have to evangelize not only freedom, but also a different platform at the same time. There were some technical reasons. Stallman rejected the idea of copying the Lisp Machine approach: https://www.gnu.org/gnu/rms-lisp.en.html https://www.gnu.org/gnu/rms-lisp.en.html At first, I thought of making a Lisp-based system, but I realized that wouldn't be a good idea technically. To have something like the Lisp machine system, you needed special purpose microcode. That's what made it possible to run programs as fast as other computers would run their programs and still get the benefit of typechecking. Without that, you would be reduced to something like the Lisp compilers for other machines. The programs would be faster, but unstable. Now that's okay if you're running one program on a timesharing system — if one program crashes, that's not a disaster, that's something your program occasionally does. But that didn't make it good for writing the operating system in, so I rejected the idea of making a system like the Lisp machine. I decided instead to make a Unix-like operating system that would have Lisp implementations to run as user programs. The kernel wouldn't be written in Lisp, but we'd have Lisp. https://www.gnu.org/gnu/thegnuproject.en.html https://www.gnu.org/gnu/thegnuproject.en.html For example, we developed the GNU C library because a Unix-like system needs a C library, BASH because a Unix-like system needs a shell, and GNU tar because a Unix-like system needs a tar program. The same is true for my own programs—the GNU C compiler, GNU Emacs, GDB and GNU Make.
- jroseattle 8y ago" Indent 2 spaces. No tabs." Oh dear, here we go....
- assafmo 8y agoWhen breaking a pipe into two lines, I find it cleaner to end the first line with pipe instead of backslash. Then I indent the second line. Bash knows a command cannot end with pipe, therefore it's syntactically ok. E.g. cmd1 | cmd2 Instead of: cmd1 \ | cmd2
- deleted 8y ago[deleted]
- hi41 8y agovery nice!
- textmode 8y agoOriginal HN title had the word "Google" in it. According to this document, Google requires Bash. I do not use Bash. I use Almquist shell. I try to avoid using uncommon "features" or utilities. I am not a frequent Linux user but whenever I have to use it, all of the hundreds of scripts I wrote using another OS and Almquist shell still work. They all work in Bash. Over the years, I used this site as a reference: https://www.in-ulm.de/~mascheck/ https://www.in-ulm.de/~mascheck/
- kevincox 8y agoThey require bash for scripts. You can use whatever shell you like on your machines.
- textmode 8y ago"They require Bash for scripts." I do not write scripts to run with Bash. I write them to run with an Almquist-derived shell, which also means I can run them with Bash and other shells without any problems. However a script written in Bash may not run correctly under Almquist shell and other shells.
- amptorn 8y ago# Not this: if [[ "${my_var}X" = "some_stringX" ]]; then do_something fi Why would anybody write this?
- tobylane 8y agoThey learned that $null = test is bad, that $a = $b is bad, but didn't learn that you do either x$null = xtest or "$a" = "$b", not both.