12 ms·

Use Long Options in Scripts

• gabrielsroka 2y ago
  Related 2013/2020 https://news.ycombinator.com/item?id=24518682 https://news.ycombinator.com/item?id=24518682
• starkparker 2y ago
  And put them on separate lines so you can track and git blame them more easily.
  • ivanjermakov 2y ago
    Same line git blame is not that hard, just list commits affecting specific file or even specific line span: https://git-scm.com/docs/git-log#Documentation/git-log.txt--Lltstartgtltendgtltfilegt https://git-scm.com/docs/git-log#Documentation/git-log.txt--...
• ratrocket 2y ago
  I agree with this practice. Another benefit is it makes it easier (slightly, but still) to grep the man page for what the options do. The corollary must be "write programs that take long options".
• wahern 2y ago
  Please DO NOT mix string interpolation and command execution, especially when a command is processed through the shell. Whatever your language, use a list-based or array-based execution API that passes arguments straight through to execv(2), execvp(2), etc, bypassing the shell.
  • echelon 2y ago
    SQL injection on steroids.
    • rat87 2y ago
      Only if you are getting input from untrusted users
      • remus 2y ago
        imo it's best to just avoid it altogether. Requirements change, and what was once a trusted input can become untrusted input.
        • rat87 2y ago
          Generally you shouldn't be passing random data from the web to shell scripts. Maybe I haven't done the right type of work but having to deal with fidlg bits it's much more likely not passing it to be shell will cause issues (with stuff like executable paths)
      • brookst 2y ago
        Or if your trusted users are fallible and could be tricked into providing unsafe inputs.
  • paulddraper 2y ago
    Tbf this input does not need escaping. But at the very least the shell is unnecessary here.
  • tasuki 2y ago
    Why not?
    • bulatb 2y ago
      Any time you send commands and data down a single channel, user input that's intended to be data can be misinterpreted as a command. For example, if your program wants to: run("program --option '{user_input}' > file") to save some input to a file, and the user's input is: '; bad_command # then when run() sends that string to the shell, the shell will run: program --option ''; bad_command #' > file Most languages have something like a safe_exec() that separates the shape of the command from the values of the options, executing "program" with the options and the user_input in the arguments array as data. Skipping the shell step, which would just be building an exec call anyway, removes the opportunity for users to confuse it into doing something else. The list-based API alternative they recommend might look like this: safe_exec(["program", "--option", user_input], stdout="file") and it would always exec "program" with argv[1] == "--option" && argv[2] == user_input. If the user_input happens to be: '; bad_command # ...well, then, the user can enjoy the contents of their file.
      • tasuki 2y ago
        Yes of course. But why would you expect me to run shell commands with random person's input? Also: safe_exec(["rm", user_input]) This isn't safe either! Despite clearly saying "safe_exec"!
        • bulatb 2y ago
          Yeah, "safe_exec" is a useless name without context. But the context was you need to call a program from another program. Many people would call system() or whatever because usually it's obvious and easy, and the pitfalls are less so. Shelling out is not the only option. People are just saying not to use that option. Better ones won't save you if you purposely do something stupid. They will save you if the user wants to trick you into doing something else.
          • 2y ago
  • gray_-_wolf 2y ago
    Another approach is to have powerful enough language that allows you to guard against the shell injection. I wrote a syntax form allowing to do this: (sh "cat " file " >" output) With file being bound to "foo'bar" and output to "x", it is automatically translated into cat 'foo'\''bar' >'x' This gives you the flexibility to use shell (sometimes it just is the most concise way) while being safe against injection. I believe for example in rust you should be able to do the same.
    • delusional 2y ago
      How do you know which shell you're escaping for? You could query the system, but now you end up implementing escaping for every shell out there.
      • gray_-_wolf 2y ago
        Good question. I care only about POSIX compatible shells, so the escaping just follows the POSIX rules. In practice that means it works on any actually used system except windows, which is fine with me.
  • crazygringo 2y ago
    For anything involving file paths, user input, etc. -- yes of course. It's not even a question because they would need to be escaped otherwise which nobody wants to do. But for a simple example like this where it's inserting a date which has known properties, it seems fine, and is much more readable.
  • latexr 2y ago
    Why would they even change the language and the commands in the example? It confuses and undermines the point. Just say “use `git switch -c my-new-branch` for interactive usage and `git switch --create my-new-branch` in scripts”. It makes no sense to introduce other unexplained information.
  • matklad 2y ago
    Was waiting for this comment :P The API used handles string interpolation correctly: the string literal is parsed at compile time, and the interpolated arguments are never concatenated or escaped, and end up directly as an element of arv array passed to a child. See https://github.com/tigerbeetle/tigerbeetle/blob/7053ecd2137ace2a14069cc212b0502fe06d3d24/src/shell.zig#L679-L774 https://github.com/tigerbeetle/tigerbeetle/blob/7053ecd2137a...
    • chubot 2y ago
      This approach creates an odd mini language, which is incomplete: comptime assert(std.mem.indexOfScalar(u8, cmd, '\'') == null); // Quoting isn't supported yet. comptime assert(std.mem.indexOfScalar(u8, cmd, '"') == null); But you can do correct interpolation with simple shell variables, rather than generating shell code strings: $ today='foo; bar' sh -c 'argv git switch --create "release-$today" origin/main' ['git', 'switch', '--create', 'release-foo; bar', 'origin/main'] So that is a test that we can use a plain shell string, without any shell injection bug. (argv is my command to test quoting: python -c 'import sys; print(sys.argv)' "$@" ) Note that there's no escaping function needed, because we're not generating any shell code. We're generating an argv array for `/bin/sh` instead. --- So by invoking with an env var, you can easily create a correct API that uses plain shell git switch --create "release-$today" rather than git switch --create release-{today} # what language is this? It's not obvious If you don't want to use the env var, you can also use git switch --create "release-$1" And invoke with ['sh', '-c', shell_string, 'unused-arg0', today_string] With this approach, you don't need 1. any kind of shell escaping 2. any analyzing of pseudo-shell strings, which can't contain quotes Because you are not generating any shell code. The shell code is constant.
  • gorgoiler 2y ago
    Miniature, in-line sh scripts are also fine as long as you use the provided parameter substitution. If you’re averse to this: q(“select x where y = ‘“ + v + “‘“) And instead do this: q(“select x where y = %s”, v) Then you should be averse to this: x(“foo --option ‘“ + v + “‘“) And instead do this: x(‘foo --option “$1”’, v) This is particularly useful when it’s expedient to have one thing piping into another. Like it or not the sh DSL for pipes is excellent compared to doing things natively with execve() and pipe(), just as doing group by and count is far more concise in SQL than doing so natively. Most SQL libraries give you something like q. Writing your own x is as simple as calling sh correctly. In Python, for example: def x(script, *args): run([“sh”, “-c”, script, “--“, *args])
    • pwdisswordfishz 2y ago
      SyntaxError: invalid character '“' (U+201C)
    • kbenson 2y ago
      Neither of those are equivalent to variable binding, which is what most SQL libraries provide, specifically because they don't actually solve the problem since they're still doing string substitution. Putting a double quotes in $1 in your "good" execute example will allow you break out of what's expected and then you're Bobby Tables. Your python example at the bottom is correct, in that each separate element is more correct in that it allows each arg to be passed as an element, so there's no option to break out through quoting characters. SQL binds are like that in most ljbraries, even if they don't look like it. The parser knows a single item below there so if it passes it along as such. You cannot escape it in the same way.
      • gorgoiler 2y ago
        I don’t really follow. My “good” example and the code at the bottom are the same. sh is smarter than just doing string interpolation and ”$1” is passed on as a single argument, no matter what: > run(["sh", "-c", 'echo "$1"', "--", 'a"']) a” Whereas if it were simple string interpolation, you’d see this: > run(["sh", "-c", 'echo "a""') --: 1: Syntax error: Unterminated quoted string It’s the same special casing that gets "$@" right.
• ndegruchy 2y ago
  This is one of my default rules for writing scripts. If the long option is available, use it. It makes too much sense to do so.
• saagarjha 2y ago
  Unfortunately, if you want your scripts to be portable to other POSIX systems you might have to use the short options, as the long ones are not standardized. You have to decide the tradeoff for yourself.
  • pcwalton 2y ago
    What POSIX systems in actual use (not historical Unixes) don't have the long options? macOS' BSD utilities I guess?
    • saagarjha 2y ago
      Yes or like BusyBox
    • schneems 2y ago
      You can also brew install tools like gnused which have the same arguments. Not a viable option for all situations but if you just need to execute it on Linux and your local machine for dev you can use those.
    • yjftsjthsd-h 2y ago
      > What POSIX systems in actual use (not historical Unixes) don't have the long options? All of them except for GNU, AFAICT? (That is, only GNU seems to have long options.) Checking manpages for rm(1) as a simple reference, I can't see long options in any of the 3 major BSDs or illumos, and checking Alpine Linux seems to show busybox also only doing short options (sorry, can't find an online doc for this, though it's easy to check in docker if you don't have a machine running Alpine handy). OpenWRT also uses busybox and has the same (lack of) options. https://man.netbsd.org/rm.1 https://man.netbsd.org/rm.1 https://man.openbsd.org/rm.1 https://man.openbsd.org/rm.1 https://man.freebsd.org/cgi/man.cgi?query=rm&apropos=0&sektion=1&manpath=FreeBSD+14.2-RELEASE+and+Ports&arch=default&format=html https://man.freebsd.org/cgi/man.cgi?query=rm&apropos=0&sekti... https://illumos.org/man/1/rm https://illumos.org/man/1/rm
      • SoftTalker 2y ago
• croes 2y ago
  > Long form options are much more self-explanatory for the reader. And less prone to typos
  • dosourcenotcode 2y ago
    And not just options but base command names too. I wrote a tool to partially mitigate this in some cases: https://github.com/makesourcenotcode/name-safe-in-bash https://github.com/makesourcenotcode/name-safe-in-bash
    • sebastianmestre 2y ago
      You have a nice ternary counter going in the version numbers :)
• dosourcenotcode 2y ago
  Agree that long options should be used. But there is one caveat to consider: portability. Sadly to this day not all BSD distributions have GNU style long options. And the ones that now do only got them fairly recently. So if you want portability you have to use short options as you weep with a bottle of vodka in hand.
  • mplanchard 2y ago
    Not trying to spam this thread with praises of nix, because it does have its own problems, but it certainly solves the portability problem. Four years in to using it at work for dev environments across mac (x86 & ARM) and various linuxes and can’t imagine going back. I also always make dev environment definitions for my open source projects, so even if people aren’t using nix, there is at least a record of what tools they will need to install to run scripts, tests, etc.
    • nine_k 2y ago
      Does nix work well on BSD-derived Unices? In particular, the most widespread of them, macOS?
      • mplanchard 2y ago
        Yes, works great on Mac. About half our engineers us Macs, the other half Linux. We have one nix configuration for the dev environment, which works for everyone.
  • saghm 2y ago
    This surprises me because the first case I remember ever coming across where short versus long options impacted portability across GNU and BSD was _fixed_ by using long options. Maybe six years ago or so I had an issue porting a script someone else had written for use in CI that happened to decode some base64 data that failed when I tried to use it on a different platform. I forget which one it was originally written for and which one I was trying to use it on, but the issue boiled down to the MacOS version of base64 using the BSD short option for decode and Linux using the GNU one, and they each used a different capitalization; one used `-d` and the other used `-D` (although I also can't remember which used which honestly). My solution was to use the long option `--decode`, which was the same on both of them, and since then the times I've needed to decode base64 I've always used the long option out of habit, which probably explains why I can't remember what option Linux uses despite it being the one I've used far more over the years since then.
• userbinator 2y ago
  Strongly, strongly disagree. This is a GNU-ism, and needlessly verbose. What's with people refusing to use the vast amount of memory in their brains and actually learning, instead lazily going for the lowest-common-denominator approach relying on "loanwords" from English?
  • mplanchard 2y ago
    Because not everyone has 20 or more years of flag memory from using Linux, and they’ve also got to maintain the scripts. If you’re not familiar with every arcane invocation of find or tar or whatever, or even if it’s just been a while, the long options are a godsend when you’re skimming a script trying to figure out what it’s doing.
    • userbinator 2y ago
      Because not everyone has 20 or more years of flag memory from using Linux It's called learning, apparently something that is now being eschewed in favour of quick superficiality (and making developers more replaceable --- with AI or unskilled offshore labour.) No wonder "modern" software is almost always total shit. "But I don't have the time to learn," you complain, but ask yourself this instead: Why do you never have the time to do it right, but always the time to do it twice (or however many times is necessary to get a barely-working product)?
      • 9dev 2y ago
        [flagged]
        • userbinator 2y ago
          [flagged]
          • 9dev 2y ago
            Nobody here was advocating against learning and memorising, but for writing code that is as self-evident and unambiguous as possible. Talk to any electrician which kinds of diagrams they prefer; those were a fellow was clever, trying to compress stuff as much as possible, or those that are clearly readable and easy to understand. Pilots carry heaps of documentation with them on the plane, walk through checklists instead of trying to memorise what to do. Your argument doesn’t hold up the slightest scrutiny. I’m advocating for a culture of helping people hone their skills by learning from others instead of being turned off by their arrogance. That’s a very different thing from encouraging them to avoid thinking.
• teddyh 2y ago
  Also, do not forget using “--” after all options, but before any dynamic arguments, just to be safe.
  • arcanemachiner 2y ago
    I know to do this intuitively, but I have no idea why.
    • less_less 2y ago
      It tells the shell utility that any remaining arguments are not options, but instead files or whatever the script might process. You know, in case someone makes a file called -rf.
      • pletnes 2y ago
        But not all shell utilities follow this particular convention
        • ribcage 2y ago
          Yes. The famous echo on Linux systems does not have it and therefore it's impossible to print the string "-n o p e", because -n will be interpreted as an option.
          • bonzini 2y ago
            echo is not portable anyway, use "printf %s STRING" or "printf '%s\n' STRING".
            • ribcage 2y ago
              Yes, that's what I use. Sometimes I still get tempted to use echo because there's less typing...
          • ezequiel-garzon 2y ago
• ofrzeta 2y ago
  and while you're at it, please alias --help to -h :-)
  • jiehong 2y ago
    Some cli are event worse at this when you try -h or —help and they tell you that no, you must run ‘cli help’ instead of showing the help already.
• vivzkestrel 2y ago
  what if I or someone wrote a bot / script that searches across github for every shell script file that it can find and converts all short options into long options and opens a PR? Think dependabot but lets call it longabot or readabot?
  • jmholla 2y ago
    I don't think you should do that unprompted. There are reasons for using short options like the portability mentioned in other comments. It'd put an undo burden on open source maintainers. Something opt-in like dependabot though could be useful.
  • zelphirkalt 2y ago
    Then you will get to feel the unreasonable hate of lots of lazy people, caring more about typing a few characters less, than their stuff being readable, and from all the Apple users.
• johnisgood 2y ago
  What is this? Use libgit2 or use a proper language where you do not need exec(), do it in Bash, or do it in C with libgit2 or with popen() or something. Using "system()" in C is never the right thing to do, the least you can do is using popen(). Not sure in which language you use "try shell.exec", but I am not even sure that it is the right way.
  • deleted 2y ago
    [deleted]
• pixelkink 2y ago
  Prepares to launch in flaming rant... sigh You're right.
• jFriedensreich 2y ago
  The hype side of this is to always add instructions to use long options to the agent base prompt. Its much easier to spot mistakes and long options have the advantage to not do something completely different if they are wrong.
• susam 2y ago
  I prefer long options too. However, while writing programs that need to invoke POSIX commands in a portable manner, short options are the only viable choice, as POSIX doesn't specify long options. For instance, see the specification for diff at <https://pubs.opengroup.org/onlinepubs/9799919799/utilities/diff.html https://pubs.opengroup.org/onlinepubs/9799919799/utilities/d...>, or that of any POSIX utility listed at <https://pubs.opengroup.org/onlinepubs/9799919799/idx/utilities.html https://pubs.opengroup.org/onlinepubs/9799919799/idx/utiliti...>. That said, this is more of a corner case. In most scenarios, rather than relying on POSIX utilities, there are often better alternatives, such as using library bindings instead of spawning external processes. For example, instead of invoking grep, using something like libpcre could be a more efficient choice. For non-POSIX utilities like git, hg, rg, ag, etc., using long options makes perfect sense.
  • Wowfunhappy 2y ago
    > However, while writing programs that need to invoke POSIX commands in a portable manner ...probably a stupid question, but something I have earnestly been wondering about... when does this actually happen nowadays? What POSIX systems are you targeting that aren't one of the major ones (Linux, Darwin, or one of the major BSDs)? I was writing a shell script a few months ago that I wanted to be very durable, and I targeted sh instead of bash just because, well, it seemed like the correct hacker spirit thing to do... but I don't actually know what system in the past decade (or more) wouldn't have bash.
    • Gud 2y ago
      FreeBSD doesn’t come with bash though.
      • justaj 2y ago
        Nor does OpenBSD for that matter.
      • vermaden 2y ago
        But it also have drawbacks :) But being honest - you can install BASH that way: # pkg install -y bash
• amelius 2y ago
  Before invoking a command, always first check if the length of the command is not longer than ARG_MAX. For example, if this is your command: grep --ignore-case --files-with-matches -- "hello" *.c Then invoke it as follows: CMD="grep --ignore-case --files-with-matches -- \"hello\" *.c" ARG_MAX=$(getconf ARG_MAX) CMD_LEN=${#CMD} if (( CMD_LEN > ARG_MAX )); then echo "Error: Command length ($CMD_LEN) exceeds ARG_MAX ($ARG_MAX)." >&2 exit 1 fi eval "$CMD" # warning, evaluates filenames
  • mhitza 2y ago
    That might be sensible, but also obscure the script logic. Since using Linux exclusively, I don't think I've ever encountered an issue due to too many arguments/length. And it's the first time I'm actively searching online for ARG_MAX. I understand that different shells might be different, but with reasonable lengths is there any chance of it being relevant (aside from xargs, where it's generally intended, or better, to pass along each argument individually).
    • amelius 2y ago
      I started running into these issues when I started working with training examples in the context of deep learning. A folder with millions of files is then not unheard of. Also if you do things like */*/*, then you can quickly get large command lines. Or even if you do long_name/another_long_name/*.
    • wodenokoto 2y ago
      I've had problems with "cat *.csv" plenty of times processing data that is generated in many small files. It is really difficult to deal with, because on top of the arg max limit, globs are not guaranteed to be in order. The solution is not obvious and hard to get to if you don't know the foot guns in advanced and hard to read once implemented.
  • apgwoz 2y ago
    You should always type check your shell scripts as well. For example, you just: $ shelltypes script.sh # Welcome to shelltypes v 3.23.2 # type ‘help’ if you’re stuck >>> {1) # import POSIX;; Importing 73 items. >>> {2} # append loadpath “/opt/local/shelltypes/base”;; >>> {3} # import base::YOURPROJECT;; Importing 15 items. >>> {4} # check “YOURSCRIPT.sh” Parsing YOURSCRIPT.sh. Reticulating splines. Expanding aliases. Analyzing free shell environment variables. Found inconsistencies in PATH. Warning: Low battery!!! Warning: found free type for ‘shred’, ignoring. Warning: use of sudo requires password under /etc/sudoers. Warning: this utility is fake. Error: use of cat impossible in the presence of mutt. Found 15 errors. Try again. Goodbye. $ Then you can be pretty sure your script isn’t going to do unnecessary harm, and has some proper guardrails in place.
• ashu1461 2y ago
  Reminds me of our code having llm generated regular expressions which are impossible to understand and the only way you can tweak it is giving it to llm to change.
  • akovaski 2y ago
    Folks, remember record your LLM prompt in a comment so that your regex can be validated.
• malkia 2y ago
  It depends...
• sloooooooooooop 2y ago
  [flagged]
• lapsed_lisper 2y ago
  I used to think this, but now mostly (but weakly) don't. Long options buy expressiveness at the cost of density, i.e., they tend to turn "one-liners" into "N-liners". One-liners can be cryptic, but N-liners reduce how much program fits on the screen at once. I personally find it easier to look up flags than to have to page through multiple screenfuls to make sense of something. In this respect, ISTM short options are a /different/ way of helping a subsequent reader, by increasing the odds they see the forest, not just the trees.
• chasil 2y ago
  If your goal is to help your coworkers, this is correct. If not, it isn't.
• deleted 2y ago
  [deleted]
• wodenokoto 2y ago
  Detracting from the message, but what is the `try shell.exec(" ...` thing?
• m463 2y ago
  not only long options, also multiple lines: somecmd \ --option1 \ --option2 $FOO \ --option3 $BAR