11 ms·
If you’re writing shell scripts you should have https://www.shellcheck.net/ https://www.shellcheck.net/ in your editor and pre-commit hooks to catch common foot
by acdha 7y ago
If you’re writing shell scripts you should have https://www.shellcheck.net/ https://www.shellcheck.net/ in your editor and pre-commit hooks to catch common footguns.
Even then, my threshold for “this should be Python” has shrunk over the years: I used to say “greater than one screen of code” but now it’s more like “>1 branch point or any non-trivial scalar variable”.
- mysterydip 7y agoA common problem that happened to a coworker was he made a quick bash script for something simple, then kept adding "just one more" thing with sunk cost fallacy not wanting to take the time to rewrite it. Eventually the monstrosity created was too difficult to debug and it had to be rewritten in a different language.
- Jade_Jet 7y agoI’ve seen the same thing happen with any language. Generally tends to happen when a dev hasn’t thought through the scope of what they are doing beforehand. I’ve written some ugly python in my earlier days due to this as well. My point here is it is less to due with the language and more to due with the mindset when solving a problem. The main issue I see with more inexperienced devs with bash is that they tend to think it’s okay to be lazy with the code because it’s just “bash”. If you would write safety checks and comments in your python you should be doing the same in bash really.
- scbrg 7y agoAnother rule of thumb would be: "How many years from now do you still want this to work?" If I run a shell script I wrote ten years ago, it works. If I run a Python script I wrote ten years ago, it's quite likely to fail with a SyntaxError. This, I say as someone who loves Python and I use it as my primary language both privately and at work. But I have to admit, Python scripts do not really age well.
- np_tedious 7y agoI'm really struggling to think of an example that doesn't involve py2 --> py3. Can you share a few?
- scbrg 7y agoWell, py2 -> py3 is obviously one. Before that, well it's been a while. But with and yield were added as a keywords (breaking any code using them as a variables). The xreadlines module disappeared at some point (I'm sure a few other modules have died along the way). At some point you couldn't raise anything you wanted as exceptions any more. Oh, and the source encoding declaration. That's what I remember off the top of my head. Minor things that are handled in things that are maintained, sure. But many things are also built once and then expected to run for a long time - especially in the role shell scripts usually fill. Note, that I don't argue that any of these changes were bad. They were all good changes that made the language better. But, they resulted in old code breaking. One extra headache to deal with for the poor schmuck that was responsible for upgrading this server or that to the latest OS release.
- edoceo 7y agoI had one like that, years ago, in a 2.3 to 2.7 environment update. Had to just update a few lines. But, we knew to watch for it in test, cause we knew of the update. We test our she'll scripts too, when the env changes. Anecdotally the Py is slightly more touchy than Bash
- bsder 7y ago> If I run a shell script I wrote ten years ago, it works. I'll provide the counterexample. Any scripts I wrote more than 10 years ago for actual "Bourne shell" /bin/sh often fail in mysterious (and sometimes silent) ways when run under bash. Bash is just as guilty of backward incompatibility as Python is. Python at least generally has the decency to squawk about it.
- scbrg 7y agoWell, I won't argue with that. When I say shell script, I mean POSIX. I never write bash scripts (although admittedly, I do miss pipefail).
- npmaile 7y agosecond this. We recently added it to a project at my company (https://github.com/homedepot/spingo https://github.com/homedepot/spingo) as part of a github action and it is awesome. A quick search for the specific code in the shellcheck wiki reveals the problem and a solution. I've had no real issues with it yet.
- kylek 7y agoMy rule of thumb is- if I think I'm going to need to use an array in bash, I'm probably doing something wrong and better just use python.
- pinopinopino 7y agoI don't know, I often miss the pipe operator in python, which makes certain things much easier. I once created a small class to simulate this a bit, it allows you to use + as pipe operator. Wouldn't use it in production though, was just fun to write it. https://pastebin.com/eQacwLj7 https://pastebin.com/eQacwLj7
- ehsankia 7y agoWhy not use | as the pipe operator? Just use __or__ instead of __add__ I have done something similar too once in a shell-like Python CLI I worked on. Can also use __ror__ to be able to pass primitives into your commands like `[1, 2, 3] | SumCmd() | PrintCmd()` I do agree that typing something like that, especially when working in a shell, is much nicer than having to go back and forth all the time to type `print(sum([1,2,3]))` Here it is mentioned in a talk about Python Aesthetics by Brandon Rhodes: https://www.youtube.com/watch?v=x-kB2o8sd5c&t=8m24s https://www.youtube.com/watch?v=x-kB2o8sd5c&t=8m24s
- pinopinopino 7y agoOh that is possible? I am not that deep into python honestly. Looks much neater to use |.
- swiley 7y agoDoesn’t python have channels? The pipe is essentially a channel combined with a close message (a null object I guess) Otherwise you can do something similar with iterators and function composition. I guess the syntax isn’t quite as easy to read if you’re not used to function composition?(which seems surprisingly difficult for some beginners)
- 7y ago
- sverhagen 7y agoI suppose I had a more forgiving day, the other day, and told someone the threshold was hundred lines. Anyway, some value. It just challenged my colleague to make the code denser, to stay within the given limit. Sigh.
- michaelmrose 7y agoWhat about some sort of complexity checking more precise than lines of code? What about trying to write one thing you would normally write in shell with something else per unit of time.
- sverhagen 7y agoWill, this is more about giving someone some quick guidance in passing, that they can apply without too much, um, complexity.
- lukeschlather 7y agoI would simply say if you're planning on making changes to the script you probably shouldn't write it in bash. Or to be a little more rigorous, don't use bash if you couldn't rewrite the entire script, correctly, from scratch, in an hour. Although I think all of these things are just complicated ways of saying "please seriously reconsider writing it in bash." But I write bash scripts all the time, I just try to keep them as short and simple as possible.
- BurningFrog 7y agoAs someone who likes small fonts and big screens, for me a screen is often a lot more than 100 lines.
- kjeetgill 7y agoI don't think this is like the 80 character line length that's about screen size. This 100 line limit is framed as a quick and dirty for heuristic for script complexity.
- acemarke 7y agoI'm a big fan of using Plumbum [0] for writing more complex shell-script-like logic as Python. [0] https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/
- moopling 7y agoWow this looks great, thanks!
- koala_man 7y agoI keep posting this, but my favorite rule of thumb came from a Google dev infra engineer who said that "every Python script over 100 lines should be rewritten in bash, because at least then you won't fool yourself into thinking it's production quality"
- yodsanklai 7y agoThe Google shell style guide [1] says this: "If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date." [1] https://google.github.io/styleguide/shell.xml https://google.github.io/styleguide/shell.xml
- a3n 7y agoRelated, the longer your shell script gets, the greater potential for accumulated foot-guns. When you finally do the rewrite, will you be bug-compatible? And once discovered, will you be confident on which side the bug lies?
- blaser-waffle 7y agoAn aside: when did the phrase "foot-guns" become a thing. Cuz I've never heard it until this thread, where it pops up multiple times. Great phrase but went 0-60 in usage.
- mrguyorama 7y agoI'm pretty sure it's at least as old as usenet and mail lists
- HocusLocus 7y ago"If you are writing a script that is more than 100 lines long, you should probably be writing it in Perl instead." This is true, especially of Python scripts.
- 3xblah 7y agoshellcheck /scriptsdir/script I noticed this in the output. ^-- SC2148: Tips depend on target shell and yours is unknown. Add a shebang. Being new to shellcheck, not familar with options or what it does, so I hastily and erroneously typed: shellcheck -shell=bash script Note I learned UNIX via NetBSD. I prefer and use their version of ash for both interactive and scripting use.1 I never got used to "--" GNU-style long options. I sometimes type a single "-" out of habit. Anyway, here is the output I got from shellcheck: Unknown shell: hell=bash I agree with shellcheck. Although there may be some irony in the fact it cannot sort out it own argument parsing. 1. I do not use other scripting languages such as Python, Perl, Ruby, etc. That means, e.g., for quick and dirty one-offs and prototyping, I can omit the shebang. Debian's "dash" scripting shell is derived from NetBSD's ash, the one I choose for interactive use.
- inetknght 7y agoThe error message is quite clear: SC2148: Tips depend on target shell and yours is unknown. Add a shebang. If you google what a shebang is, the top link for me is a Wikipedia article on the subject [0]. A shebang is basically just a line (always the first line) in a file which tells the operating system what program to invoke to execute the script. There are different shells beyond just bash, so shellcheck wants to know which flavor the shell is written for and uses the shebang to figure it out. I always have the top of my shell scripts with a shebang, even if the script isn't intended to be directly executed. Pick the user's bash from PATH environment: #!/usr/bin/env bash Or specify a specific bash: #!/bin/bash Or use whatever plain-shell is installed: #!/bin/sh Or maybe it's a Python script: #!/usr/bin/env python3 Or it's a text file: #!/usr/bin/env vi If you're not using shebangs then you're probably writing your scripts wrongly. [0] https://en.wikipedia.org/wiki/Shebang_(Unix) https://en.wikipedia.org/wiki/Shebang_(Unix)
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- 7y ago
- gdevenyi 7y agoI still find python incredibly inconvenient for pipelining a ton of unix style command line tools together. Bash still wins there.
- znpy 7y agoUh.. you just reminded me of a python syntax hack that might come in handy... I'll give it a look again today at work, maybe I'll write some PoC and link it back here :)
- skocznymroczny 7y agoTo be honest, for most of my scripting needs I usually start with Python directly and just go mass os.system() or commands.get_output() calls. Later on I refactor into subprocess.Popen as needed
- JdeBP 7y agoOr parsing the output of the ls command. (-: * https://unix.stackexchange.com/a/129120/5132 https://unix.stackexchange.com/a/129120/5132
- de_watcher 7y agoOh no, Python feels like an application code. When you hit the bug tangled in Python you know it's going to be tedious.